Adam is often introduced as the optimizer you reach for after SGD has made you stare at a loss curve for too long.

That description is useful in the same way “use a car when walking is slow” is useful. It tells you something about convenience and almost nothing about the machine.

Adam is not merely SGD with momentum and a nicer default learning rate.

Adam changes the coordinate system of the update.

That is why it can feel miraculous on sparse, noisy, badly scaled problems. It is also why it can behave differently from SGD even when both reduce the training loss. The optimizer is not just deciding how far to move. It is deciding what a unit of movement means in each coordinate.

Optimizer Keeps a Ruler

Stochastic gradient descent keeps a single global learning rate:

\[\theta_{t+1} = \theta_t - \alpha g_t.\]

Momentum adds a smoothed first moment. Adam adds two memories:

\[m_t = \beta_1 m_{t-1} + (1-\beta_1)g_t,\] \[v_t = \beta_2 v_{t-1} + (1-\beta_2)g_t^2.\]

After bias correction, Adam updates each coordinate roughly as

\[\theta_{t+1} = \theta_t - \alpha \frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}.\]

The denominator is the important object. If a coordinate has historically large gradients, Adam shrinks its effective step. If a coordinate has small or rare gradients, Adam gives it a larger effective step.

AdaGrad made this geometry explicit earlier: accumulate squared gradients and use them to adapt the per-coordinate step size.1 Kingma and Ba’s Adam combined adaptive second moments with momentum-like first moments, bias correction, and an exponentially decaying memory.2

The slogan I trust is:

Adam divides by a running estimate of the coordinate's gradient unit.

That is a coordinate system.

Two Coordinates, Two Rulers

The lab below minimizes a toy quadratic:

\[f(x,y)=\frac{1}{2}(r x^2 + y^2).\]

The \(x\) direction is sharp. The \(y\) direction is flat. The curvature ratio \(r\) is the condition number of this little diagonal problem.

SGD uses a stable learning rate capped by the sharp coordinate:

\[\alpha_{\text{SGD}} = \frac{\alpha}{r}.\]

That makes the sharp coordinate safe, but it makes the flat coordinate crawl. Adam uses the same base step \(\alpha\), but divides each coordinate by its running second moment. On this diagonal toy, that is almost exactly the missing unit conversion.

Deterministic two-coordinate optimizer toy. SGD uses a stable step alpha/ratio. Adam and AdamW use the same base alpha with beta1 fixed at 0.9. The point is the geometry of the update, not a deep-network benchmark.

At the default setting, the curvature ratio is 80. The stable SGD step is therefore only 0.0005. After 240 steps, SGD has killed the sharp coordinate, but the flat coordinate is still about 0.885 away from the optimum.

Adam’s effective learning rate in the flat coordinate is about 78x larger than its effective learning rate in the sharp coordinate. It is not guessing a magic global step size. It is using the gradient history to define local units.

Raise Curvature ratio. SGD slows down in the flat direction because its single stable step is chained to the sharp direction. Adam increases the unit ratio because the sharp coordinate has much larger second moments.

Raise Base step alpha. Adam can overshoot too. Adaptive scaling is not a license to ignore the learning rate; it changes which learning rate mistakes are likely.

Raise Adam beta2. The second-moment estimate changes more slowly. A large \(\beta_2\) can make the unit system smoother, but less responsive to a changing gradient scale.

Raise Gradient noise. The denominator starts measuring noise as well as curvature. That is often useful, but it is also the beginning of a different implicit bias: coordinates with noisy gradients get smaller steps.

The Decay Was Being Measured Wrong

The lab also compares Adam with coupled \(L_2\) regularization to AdamW.

In plain SGD, adding \(L_2\) regularization to the gradient and applying weight decay are essentially the same operation up to the learning rate. In an adaptive method, they are not.

If Adam receives the regularized gradient

\[g_t + \lambda \theta_t,\]

then the regularization term is also divided by

\[\sqrt{\hat v_t}+\epsilon.\]

That means the shrinkage is coordinate-dependent. A coordinate with a large second moment gets less effective decay; a coordinate with a small second moment gets more. The lab’s “L2 decay skew” metric is the ratio of those denominators. At the default setting it is about 78x.

AdamW, proposed by Loshchilov and Hutter, decouples weight decay from the adaptive gradient step.3 Instead of hiding decay inside the gradient that Adam divides by \(\sqrt{\hat v_t}\), it applies a direct multiplicative shrink:

\[\theta \leftarrow (1-\alpha \lambda)\theta\]

and then takes the adaptive gradient step.

This is a small code change with a large semantic difference:

coupled L2: regularization is measured in Adam's adaptive coordinates
AdamW: weight decay is measured in parameter coordinates

The question is not which one is universally correct. The question is which operation you meant.

The Warning Label

Adam works so often that it is easy to forget it has theory scars.

Reddi, Kale, and Kumar showed that Adam can fail to converge even in simple convex settings, and proposed AMSGrad as one repair using a long-term maximum of the second-moment accumulator.4 The lesson is not that everyone should replace Adam with AMSGrad in every model. The lesson is that the denominator is part of the algorithm’s memory, and memory can create failure modes.

Wilson, Roelofs, Stern, Srebro, and Recht later argued that adaptive methods can find different solutions from SGD and sometimes generalize worse, even when training loss improves faster.5 Again, the point is not “Adam bad, SGD good.” The point is that an optimizer is an inductive bias. It changes the path and sometimes the destination.

Once you see Adam as a coordinate system, these warnings feel less mysterious. Changing coordinates can make an optimization problem easier. It can also change which solution looks nearby.

Optimizer Review

When Adam is the default, I would still ask:

  1. Are gradients sparse or badly scaled, or are we using Adam only because it is familiar?
  2. Does the task benefit from fast optimization, or does final generalization matter more than early training loss?
  3. Is weight decay implemented as coupled \(L_2\), AdamW-style decoupled decay, or something else?
  4. Are learning rate, \(\beta_2\), and epsilon logged with the same seriousness as model architecture?
  5. Do the coordinates with large second moments correspond to noise, curvature, rare features, or genuinely important directions?
  6. Does the SGD or momentum baseline get a fair schedule, warmup, and decay search?

Adam is a brilliant piece of engineering because it lets many models start moving before we understand their geometry.

But that convenience should not hide the mechanism.

The optimizer is choosing units.

  1. John Duchi, Elad Hazan, and Yoram Singer, “Adaptive Subgradient Methods for Online Learning and Stochastic Optimization”, Journal of Machine Learning Research, 2011. ↩

  2. Diederik P. Kingma and Jimmy Ba, “Adam: A Method for Stochastic Optimization”, ICLR 2015. ↩

  3. Ilya Loshchilov and Frank Hutter, “Decoupled Weight Decay Regularization”, ICLR 2019. ↩

  4. Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar, “On the Convergence of Adam and Beyond”, ICLR 2018. ↩

  5. Ashia C. Wilson, Rebecca Roelofs, Mitchell Stern, Nathan Srebro, and Benjamin Recht, “The Marginal Value of Adaptive Gradient Methods in Machine Learning”, NeurIPS 2017. ↩