There is a small lie hidden in the phrase “take the gradient.”

It makes the gradient sound like a mathematical object waiting politely beside the function. In software, the gradient is a second computation. Someone has to build it, schedule it, store what it needs, and decide what to recompute when memory runs out.

Finite differences ask the function a clumsy question:

What happens if I wiggle parameter 17?
What happens if I wiggle parameter 18?
What happens if I wiggle parameter 19?
...

Reverse-mode automatic differentiation asks a different question:

Which earlier quantities were responsible for this final scalar?

Then it runs the program backward.

Not backward in time as poetry. Backward as accounting.

Not Symbolic, Not Numerical

Baydin, Pearlmutter, Radul, and Siskind describe automatic differentiation as a family of techniques for accurately evaluating derivatives of numeric programs, distinct from hand-coded derivatives, finite differences, and symbolic manipulation.1 That distinction matters.

Symbolic differentiation manipulates expressions. It may simplify

\[\frac{d}{dx}\sin(x^2)\]

into

\[2x\cos(x^2).\]

Finite differences evaluate the original program at nearby inputs:

\[\frac{f(x+h)-f(x-h)}{2h}.\]

Automatic differentiation runs the actual program, but augments each primitive operation with local derivative information. For

\[z = a x + b,\qquad y = \tanh(z),\]

the forward value (y) is ordinary computation. The derivative facts

\[\frac{\partial y}{\partial z}=1-\tanh^2(z),\quad \frac{\partial z}{\partial a}=x,\quad \frac{\partial z}{\partial x}=a\]

are local receipts.

Reverse mode keeps those receipts until the final output is known.

A Chain Has a Transpose

Consider a scalar computation that evolves a state through many steps:

\[x_{t+1}=\phi_t(x_t,a_t,b_t),\qquad L=\frac12(x_T-y)^2.\]

There are (2T) parameters, two per step. A finite-difference gradient with central differences evaluates the whole chain about (4T+1) times. Forward mode can propagate all (2T) parameter tangents at once, but each step carries a large derivative vector. Reverse mode does something stranger and better for a scalar output.

First run forward and remember the needed intermediate values. Then define the adjoint

\[\bar{x}_t = \frac{\partial L}{\partial x_t}.\]

At the end,

\[\bar{x}_T=x_T-y.\]

Then for (t=T-1,T-2,\ldots,0),

\[\frac{\partial L}{\partial a_t} = \bar{x}_{t+1} \frac{\partial \phi_t}{\partial a_t}, \qquad \frac{\partial L}{\partial b_t} = \bar{x}_{t+1} \frac{\partial \phi_t}{\partial b_t},\]

and

\[\bar{x}_t = \bar{x}_{t+1} \frac{\partial \phi_t}{\partial x_t}.\]

That is the whole trick. The program’s dependency graph has been transposed. The final scalar sends blame backward through every edge.

Paul Werbos’s 1974 thesis placed this kind of backward differentiation inside learning systems and network models.2 Seppo Linnainmaa’s work on algorithmic determination of Taylor coefficients makes the reverse traversal and storage burden visible from the numerical-analysis side: first-order coefficients can be computed by reading saved operation information in reverse order, and the storage needed for that information is a central concern.3

Backpropagation is not a special neural-network spell. It is reverse-mode AD applied to the particular program that evaluates a network loss.

The Tape Is the Price

The forward pass produces values. The backward pass needs many of them.

For the chain above, (\partial \phi_t/\partial a_t) depends on (x_t), and (\partial \phi_t/\partial x_t) depends on the local preactivation. If those values are gone, the backward pass cannot pay its bills. It must either store them or recompute them.

This is why deep learning systems talk about “activation memory.” The expensive object is often not just the parameters. It is the tape of intermediate values needed to transpose the computation.

Griewank and Walther’s revolve checkpointing work starts from exactly this pressure: reverse mode can compute a scalar-output gradient at a small multiple of the original evaluation cost, but the original formulation needs memory proportional to the runtime of the evaluation program.4 Checkpointing stores selected intermediate states and recomputes missing pieces later.

That trade is not an implementation detail. It changes which models fit.

store everything: fast backward, large tape
store almost nothing: tiny tape, huge recomputation
checkpoint: spend extra compute to buy memory back

A Tape Lab

The lab below differentiates a deterministic nonlinear chain:

\[x_{t+1} = \tanh(a_t x_t+b_t) + c\frac{x_t}{1+x_t^2} + 0.025\sin(0.11(t+1)x_t).\]

It computes the full gradient with:

  1. reverse mode, by saving local receipts and sweeping backward;
  2. forward mode, by carrying all parameter tangents forward;
  3. central finite differences, by perturbing every parameter.

The same code also models a simple periodic checkpoint schedule. This is not the optimal revolve schedule; it is deliberately plain so the memory/recompute tradeoff is visible.

I audited the lab before rewriting. With 40 chain steps, there are 80 parameters. Reverse mode uses the model’s 2 full-chain work units, forward mode needs 81, and central finite differences need 161. The forward-mode gradient matches reverse mode to displayed zero, the selected finite-difference RMS error is about (6\times10^{-12}), and the best epsilon in the sweep is (10^{-5}). The default checkpoint stride cuts stored states from 41 to 15 in exchange for a 2x replay model. Across 432 parameter combinations, all gradient and work checks stayed finite, forward mode matched reverse mode below (10^{-10}), and the best finite-difference epsilon was always inside the tested range rather than pinned to an endpoint.

Reverse mode Forward mode / selected point Finite differences Checkpoint schedule

Deterministic toy experiment. Work ratios count full-chain evaluations as the unit and ignore constant-factor primitive costs. The checkpoint panel uses a simple periodic policy, not the optimal revolve schedule.

Move Finite diff epsilon left and right. Too large an epsilon measures the wrong secant; too small an epsilon subtracts nearly equal floating-point numbers. The error curve bends because numerical differentiation is balancing truncation error against roundoff error.

Now raise Chain steps. The number of parameters rises with the length of the program. Reverse mode still sends one adjoint backward through the chain, while finite differences ask the entire program a separate question for every parameter.

Finally change Checkpoint stride. A stride of 1 is the full tape: fast backward, high memory. Moderate strides store fewer states and replay local segments once. Storing nothing would save memory but force repeated long recomputations.

What Frameworks Hide

PyTorch, JAX, TensorFlow, Enzyme, Tapenade, ADOL-C, and friends hide a lot of machinery, but they do not remove the tradeoff.

When you call backward(), some execution trace has to exist. In eager systems it may be a dynamic tape. In compiled systems it may be an intermediate representation transformed by compiler passes. In either case, reverse mode is not free because the backward program needs information from the forward program.

That is why small source-level changes can have huge memory effects:

  • saving an activation versus recomputing it,
  • fusing an operation versus exposing an intermediate,
  • differentiating through a loop versus stopping the gradient,
  • checkpointing a block versus storing its full tape,
  • using forward mode for a Jacobian-vector product instead of reverse mode for a vector-Jacobian product.

The gradient is a program. Optimizing gradient computation is program optimization.

The Backward Pass Is a Ledger

The most useful mental model is not “calculus happens.”

It is this:

the forward pass creates value receipts
the backward pass spends them in reverse causal order

Finite differences are valuable as a test. Symbolic derivatives are valuable when expressions are small enough to manipulate. But modern differentiable programming runs on the tape, the transpose, and the memory system.

Once you see that, backpropagation feels less like a miracle and more like a beautifully disciplined accounting trick.

The Trail of Papers

  1. Atilim Gunes Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind, “Automatic Differentiation in Machine Learning: a Survey,” Journal of Machine Learning Research, 2018. JMLR, PDF, arXiv. ↩

  2. Paul J. Werbos, “Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences,” PhD thesis, Harvard University, 1974. PDF. ↩

  3. Seppo Linnainmaa, “Taylor expansion of the accumulated rounding error,” BIT, 1976. PDF. ↩

  4. Andreas Griewank and Andrea Walther, “Algorithm 799: revolve: an implementation of checkpointing for the reverse or adjoint mode of computational differentiation,” ACM Transactions on Mathematical Software, 2000. ACM, Semantic Scholar. ↩