There is a version of neural-network theory that sounds almost too clean:

make the network wide enough
freeze the tangent features at initialization
train a linear model in that feature space

The trained network and the trained linearization then agree.

This is the neural tangent kernel story. It is beautiful because it takes a nonlinear training process and gives it a function-space equation. It is also dangerous because a good asymptotic description can harden into a bad metaphor. Real networks are finite. Their features can move. Sometimes that movement is noise. Sometimes it is the point.

This post is a small instrument for watching that distinction.

The Linear Ghost at Initialization

Let a neural network be written as \(f_\theta(x)\). Around its initialization \(\theta_0\), the first-order Taylor expansion is

\[f_\theta(x) \approx f_{\theta_0}(x) + \nabla_\theta f_{\theta_0}(x)^\top(\theta-\theta_0).\]

The vector \(\nabla_\theta f_{\theta_0}(x)\) is a feature vector. It is not a feature learned after training. It is the network’s sensitivity to each parameter at time zero.

If those tangent features stay effectively fixed while gradient descent runs, training becomes ordinary linear regression in a high-dimensional random feature space. The corresponding kernel is

\[K_{\theta_0}(x,x') = \nabla_\theta f_{\theta_0}(x)^\top \nabla_\theta f_{\theta_0}(x').\]

Jacot, Gabriel, and Hongler introduced the neural tangent kernel and showed that, in the infinite-width limit for fully connected networks, this kernel converges to a deterministic limit and stays constant during training.1 Lee and coauthors sharpened the operational picture: wide networks trained by gradient descent can evolve like the linear model obtained from the first-order Taylor expansion around initialization.2

That is the clean room.

The question outside the clean room is not whether the theorem is elegant. It is:

did my finite network stay close enough to its tangent model?

Two Machines From One Initialization

The lab below initializes one two-layer tanh network,

\[f(x) = \frac{1}{\sqrt{m}} \sum_{j=1}^{m} a_j \tanh(w_jx+b_j),\]

then trains two predictors on the same noisy one-dimensional regression task:

  1. the actual network, updating \(a_j,w_j,b_j\) by gradient descent;
  2. the linearized model, freezing the tangent features \(\nabla_\theta f_{\theta_0}(x)\) and updating only the parameter displacement in that fixed feature space.

They start at exactly the same function. They receive the same learning rate and number of steps. If the lazy approximation is good, their curves should nearly track each other.

The lab also measures two pieces of evidence:

  • parameter movement: how far the trained network moved relative to the initialization norm;
  • NTK drift: the relative Frobenius change in the training-set tangent kernel from initialization to the end of training.

The Audit tile is generated by the same JavaScript. It checks parameter sanitation, tangent-feature dimensions, a finite-difference gradient test, training-loss improvement for both dynamics, kernel symmetry, finite output on a 72-case grid, trace construction, and the default fact ledger.

Trained network Fixed linearization Target function Training data Kernel drift

Deterministic toy regression. It trains a finite two-layer tanh network and its first-order Taylor model from the same initialization; it is not a proof of an infinite-width limit or a benchmark of modern architectures.

With the default settings, the true network finishes at train MSE 0.0559 and test MSE 0.0729. The fixed linearization finishes at train MSE 0.0962 and test MSE 0.1247. The network moved 34.03% of its initialization norm, and the training-set NTK drifted by 14.29%.

That is not a failure of the NTK idea. It is a finite-width diagnostic. The linear ghost is still useful, but the living network has begun changing its own features.

Now push Width toward 128 and lower the learning rate. The two predictors usually move closer together, and NTK drift falls. Push Width down, raise the learning rate, or increase the local bump. The true network often separates from the fixed tangent model because it can reshape the hidden units around the target.

The most honest sentence is:

the kernel approximation is a regime, not an identity card

Lazy Is a Scaling Choice

Chizat, Oyallon, and Bach argued that lazy training is not a mystical property of neural networks. It can arise from scaling: when the model’s output scale and parameterization make the function change a lot for small parameter movement, gradient descent can fit while barely moving in parameter space.3

In that regime, the model behaves like its linearization. The optimization problem may become easier to analyze, but the learned representation is mostly the one sampled at initialization.

Woodworth and coauthors framed the contrast as kernel versus rich regimes. In the kernel regime, gradient descent behaves like a kernelized linear predictor. In the rich regime, the model’s internal representation changes and the implicit bias need not be an RKHS norm from the initial tangent kernel.4

This is why the phrase “neural networks are just kernels” misses the hinge. Sometimes the model is kernel-like because the features are effectively fixed. Sometimes the model earns its accuracy by learning features. Both can fit the training set. They are not the same explanation.

What the Lab Is and Is Not Saying

The lab uses a one-dimensional regression target, a small tanh network, full batch gradient descent, and squared loss. It is designed to make the accounting visible:

same initialization
same data
same optimizer clock
moving tangent features versus frozen tangent features

It is not a theorem about transformers. It is not evidence that feature learning is always better. It is not a refutation of NTK theory. The point is more modest and more practical: if you want to use a kernel-limit story to understand a trained finite network, you should measure whether the finite network stayed in the relevant regime.

Useful diagnostics include:

  • distance from initialization, normalized by the initialization scale;
  • tangent-kernel drift on a representative data slice;
  • agreement between the trained model and its linearization;
  • whether feature-learning-sensitive slices improve while the tangent model stalls;
  • how those quantities change with width, learning rate, initialization scale, normalization, and training time.

Those are not decorative plots. They tell you which explanation you are allowed to use.

The Practical Moral

Kernel limits gave deep learning theory a powerful microscope. They turned training dynamics into something solvable enough to inspect. That matters.

But microscopes have working distances.

If the tangent kernel barely changes, the linearized model may be the right language. Then one should ask kernel questions: spectrum, effective dimension, regularization, early stopping, and which components are learned fastest. Jacot and coauthors’ NTK analysis explicitly connects training speed to kernel principal components.1

If the tangent kernel changes substantially, the model is not merely fitting in a sampled feature space. It is learning the space. Then one should ask representation questions: which features moved, which examples caused motion, which symmetries were discovered, and which implicit bias selected the moving solution.

The distinction is not philosophical. It changes what you debug.

kernel regime: why did this fixed feature map generalize?
rich regime: what feature map did training create?

A network can begin as a kernel machine. The interesting part is catching the moment it stops being only that.

  1. Arthur Jacot, Franck Gabriel, and Clément Hongler, “Neural Tangent Kernel: Convergence and Generalization in Neural Networks”, NeurIPS 2018. ↩ ↩2

  2. Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington, “Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent”, NeurIPS 2019. ↩

  3. Lénaïc Chizat, Edouard Oyallon, and Francis Bach, “On Lazy Training in Differentiable Programming”, NeurIPS 2019. ↩

  4. Blake Woodworth, Suriya Gunasekar, Jason D. Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro, “Kernel and Rich Regimes in Overparametrized Models”, COLT 2020. PMLR record: https://proceedings.mlr.press/v125/woodworth20a.html. ↩