A learning rate looks like a speed knob.

That is the first lie.

It is a stability budget.

When you choose a step size, you are making a promise about the sharpest direction of the loss surface. If the local curvature is gentle, the same step looks cautious. If the local curvature is large, the same step can turn from descent into a numerical trampoline.

The old lesson is simple enough to fit on a napkin. The new lesson is stranger: large neural networks often train near the edge where the napkin says ordinary gradient descent should be nervous.

This post is about that edge, and about why a run can wobble there without immediately falling off.

The One-Mode Cliff

Start with the smallest possible loss surface:

\[f(z)=\frac{1}{2}\lambda z^2.\]

Gradient descent with step size \(\eta\) gives

\[z_{t+1} = z_t - \eta \lambda z_t = (1-\eta\lambda) z_t.\]

Everything depends on the multiplier:

0 < eta lambda < 1     monotone decay
1 < eta lambda < 2     sign-flipping decay
eta lambda = 2         perfect two-cycle
eta lambda > 2         sign-flipping divergence

For a quadratic objective, this is not metaphor. It is the exact recurrence. The number \(2/\eta\) is the largest curvature this scalar mode can tolerate. In many dimensions, the same story appears along each Hessian eigenvector, with the largest eigenvalue setting the most restrictive local stability condition.

So a fixed learning rate secretly says:

I can survive curvature below 2 / eta.

Bottou, Curtis, and Nocedal’s review of large-scale machine-learning optimization emphasizes that ML optimization is not merely classical nonlinear optimization at larger scale; stochastic gradients, data access, and practical training dynamics change what methods are useful.1 Still, the scalar quadratic remains the right little warning light. When a training run oscillates violently, the first suspect is often not “nonconvexity.” It is a step size that has overspent its local stability budget.

An Edge Lab

The lab below compares two systems under the same step size.

The blue panel is the frozen quadratic mode above. Its curvature is fixed. If eta x lambda exceeds 2, there is nowhere to hide.

The red panels use a tiny nonlinear model:

\[\hat y = ab, \qquad L(a,b)=\frac{1}{2}(ab-y)^2.\]

It is just a two-parameter deep linear network with one scalar training example. The Hessian changes as the parameters move. That makes it a microscope for a different possibility: the optimizer can change the sharpness of the place it is standing while it trains.

At the default settings:

eta = 1.01
frozen sharpness = 2.00
eta x lambda = 2.02
edge 2 / eta = 1.98

The frozen quadratic mode diverges. The nonlinear toy does not politely converge to zero loss either. Instead, it falls quickly, becomes non-monotone, and settles into a small oscillation whose tail-average sharpness is almost exactly the 2 / eta edge.

That is the shape to notice: short-timescale loss bumps, long-timescale improvement, and sharpness living near the numerical stability boundary.

I audited the lab before rewriting. At the default controls, the frozen quadratic multiplier is (-1.02), so the scalar mode diverges exactly as the (\eta\lambda>2) rule predicts. The nonlinear toy runs all 140 steps without diverging, has 68 upward-loss steps, ends at loss 0.0044, and averages sharpness 1.002x the (2/\eta) edge over its tail. A 324-case sweep over step size, curvature, target, initialization, and step count found no phase-classification errors in the frozen quadratic, 297 finite nonlinear runs, and 27 honest divergences. I also fixed the lab to export its API through module.exports, so the audit no longer has to reach through globalThis.

Frozen quadratic Nonlinear toy Hessian sharpness Stability edge Zero-loss curve

Deterministic full-batch gradient descent toy. The right-hand model is scalar and deep-linear, not a reproduction of a full neural network. It is a small dynamical system for inspecting how loss, curvature, and step size can interact.

The source is deliberately inspectable in assets/js/edge-stability-lab.js. The top Hessian eigenvalue for the two-parameter toy is computed from the exact Hessian:

\[\nabla^2 L(a,b) = \begin{bmatrix} b^2 & 2ab-y \\ 2ab-y & a^2 \end{bmatrix}.\]

At a zero-loss point with \(ab=y\), the Hessian has eigenvalues \(0\) and \(a^2+b^2\). The flat direction is the rescaling symmetry: multiply \(a\) by a constant and divide \(b\) by the same constant. The sharp direction changes the product.

Already this tiny model contains three warnings.

First, local stability is about the update map, not just the loss value. A point can have low loss and still be numerically hostile to a large step.

Second, non-monotone loss is not automatically failure. In the default toy, many steps increase the loss relative to the previous step, but the run has still moved from roughly 0.49 loss to a small oscillation around 0.005.

Third, “sharpness” is a dynamical quantity during training. It is not merely a label stamped on the final model.

The Neural-Net Surprise

The classical story would be:

choose eta small enough that eta * lambda_max stays below 2

Then Cohen, Kaur, Li, Kolter, and Talwalkar measured full-batch gradient descent on neural networks and reported something more provocative. In their edge of stability regime, the maximum Hessian eigenvalue hovers just above \(2/\eta\), the training loss is non-monotone over short windows, and the loss still decreases over longer windows.2

This was not supposed to be the polite behavior. A textbook local quadratic approximation would suggest that once the sharpest direction crosses the edge, gradient descent should become unstable. But real neural training is not a frozen quadratic. The parameters move. The Hessian changes. The objective has many nearly flat and scale-sensitive directions. The optimizer can step into a region, kick itself away, reduce sharpness, and continue.

Lewkowycz, Bahri, Dyer, Sohl-Dickstein, and Gur-Ari studied a related large learning-rate regime they called the catapult mechanism. Their solvable models and experiments show a phase transition between small and large learning rates; in the large-rate phase, training can initially increase loss yet still converge and can favor flatter minima.3

So the learning rate is doing more than setting speed. It is changing the kind of training dynamics available.

Small step:

follow the local slope, stay inside the local quadratic story

Large step:

interact with curvature, bounce across sharp directions,
possibly select a different region of parameter space

The second behavior is risky. It is also part of why learning-rate schedules, warmup, batch size, normalization, architecture, and optimizer choice can change the result even when the model class and data are fixed.

The Edge Is Not a Generalization Certificate

It is tempting to say:

large learning rate -> flatter minimum -> better model

Sometimes that is directionally useful. As a universal law, it is too neat.

Sharpness is not an invariant property of a represented function. Dinh, Pascanu, S. Bengio, and Y. Bengio showed that in networks with rectifier symmetries, one can construct equivalent models with arbitrarily different apparent sharpness; more broadly, reparameterization can change the geometry of parameters without changing the function.4

That warning does not make Hessian measurements useless. It changes what they mean.

Sharpness during training is a statement about the local update dynamics of the parameterization you actually trained. It can explain why a step size oscillates or fails. It can diagnose when a schedule is driving the optimizer near a stability boundary. It can compare two runs when the parameterization and measurement protocol are controlled.

It is not, by itself, a moral ranking of models.

What to Log When Training Gets Weird

When a training run suddenly becomes jagged, I would rather see a stability ledger than another screenshot of the loss curve.

Log:

  • the learning-rate schedule actually applied after warmup, decay, and optimizer scaling;
  • the short-window loss increases, not only the smoothed dashboard line;
  • the gradient norm and update norm;
  • an estimate of the top Hessian eigenvalue or a cheaper curvature proxy;
  • the product \(\eta \lambda_\max\) when that estimate is available;
  • whether the same behavior appears with full-batch, large-batch, and noisy minibatch updates;
  • whether normalization layers, weight decay, clipping, or adaptive preconditioning changed the effective coordinate system.

The point is not to worship \(2/\eta\). The point is to notice that the loss curve alone hides the numerical mechanism. A run can look noisy because the data are noisy, because the optimizer is bouncing off a sharp direction, because momentum is storing old gradients, or because an adaptive method changed units. Those are different failures and different opportunities.

The Step Is Part of the Model

A trained model is not only architecture plus data.

It is also the trajectory that found the weights.

The step size decides which local quadratic approximations are trusted, which sharp regions are rejected, which oscillations are tolerated, and which parts of the loss surface become reachable. A small learning rate makes training look like careful descent. A large learning rate makes training look more like a dynamical system negotiating with its own curvature.

The old quadratic lesson remains true:

eta times sharpness near two is the cliff

The newer neural-net lesson is that the cliff is not always a wall. Sometimes it is a coastline. Training can run along it, wobble, shed sharpness, and keep making progress.

That does not make large learning rates safe. It makes them interesting. They are not merely asking “how fast can I go?”

They are asking:

what geometry am I willing to survive?

  1. Leon Bottou, Frank E. Curtis, and Jorge Nocedal, “Optimization Methods for Large-Scale Machine Learning”, SIAM Review, 2018. ↩

  2. Jeremy M. Cohen, Simran Kaur, Yuanzhi Li, J. Zico Kolter, and Ameet Talwalkar, “Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability”, ICLR 2021. ↩

  3. Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari, “The large learning rate phase of deep learning: the catapult mechanism”, 2020. ↩

  4. Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio, “Sharp Minima Can Generalize For Deep Nets”, ICML 2017. ↩