The first thing to notice about low precision is not the missing bits.

It is the gate.

A calculation walks up with a real number. The format has a fixed set of places where that number is allowed to stand. At 32-bit or 64-bit precision, the gate is usually so fine that we keep our attention on the optimizer.

Low-precision training changes the commute. The rounder stops looking like a minor storage detail and starts acting like a turnstile.

Here is the whole problem in one line:

stored_weight = round(stored_weight - learning_rate * gradient)

If the proposed update is smaller than half of one representable grid cell, round-to-nearest can send the new weight back to the old platform. The optimizer asked to move. The gate did not open.

Stochastic rounding runs a different gate:

if the proposal is 22% of the way to the next grid cell,
move to that cell with probability 22%

Most tiny steps still wait. Some pass through, and the average movement is the movement the optimizer requested.

That is why the rounding mode belongs in the training recipe. It decides which small updates get admitted to the stored state.

The Midpoint Gate

Take a finite grid with spacing \(\Delta\). If a real value \(x\) falls between two adjacent grid points \(a < x < b\), round-to-nearest chooses the closer point. Stochastic rounding chooses

\[b \quad \text{with probability} \quad \frac{x-a}{b-a},\]

and chooses \(a\) otherwise. Its conditional expectation is therefore

\[\mathbb{E}[\operatorname{sr}(x) \mid x] = x.\]

That is the famous property, but it is easy to say it too quickly. Local unbiasedness does not mean an entire nonlinear training run is unbiased. It means the particular act of mapping this real number onto this grid does not systematically push it up or down, assuming the stochastic rounder is implemented with enough randomness and the value is not clipped at the edge of the format.

Now put that grid inside gradient descent on the scalar quadratic

\[L(w) = \frac{1}{2}(w - w^\*)^2.\]

The exact update is

\[w_{t+1} = w_t - \eta(w_t - w^\*).\]

With round-to-nearest storage, a positive update changes the stored weight only when it crosses the midpoint to the next cell:

\[\eta |w_t - w^\*| \geq \frac{\Delta}{2}.\]

So the deterministic rounder has a dead zone:

\[|w_t - w^\*| < \frac{\Delta}{2\eta}.\]

That dead zone can be much larger than one grid cell. With \(\Delta = 0.05\) and \(\eta = 0.035\), round-to-nearest can stop once the remaining gap is below about 0.714. That is not a tiny numerical scar. It is a turnstile rule that says small gradients do not board.

Stochastic rounding removes the hard threshold. If the proposed update is only one fifth of a grid cell, the stored weight moves one grid cell about one fifth of the time. Learning becomes lumpy, but it does not get stranded at the midpoint gate.

Why This Old Gate Keeps Reopening

Stochastic rounding is not new. Higham traces early versions to work from the 1950s and 1960s, long before modern deep learning made low-precision arithmetic a daily engineering concern.1 The recent revival comes from the same pressure that keeps returning everywhere in deep learning: memory bandwidth, energy, and accelerator throughput are too important to leave every tensor in wide precision.

Gupta, Agrawal, Gopalakrishnan, and Narayanan made the neural-network version of the point very directly in 2015. In low-precision fixed-point training, they found the rounding scheme was crucial, and reported that deep networks could be trained with 16-bit fixed-point representation when stochastic rounding was used.2

Numerical analysts give a second reason to care. Connolly, Higham, and Mary analyze stochastic rounding through probabilistic backward error, and the broader literature shows why random signs can make accumulated roundoff behave less like a one-way drift and more like a controlled fluctuation.3 For algorithms dominated by summation, later work on limited-precision stochastic rounding summarizes a typical contrast: probabilistic bounds can scale like \(O(\sqrt{n}u)\) where worst-case deterministic bounds scale like \(O(nu)\), under the model assumptions.4

Modern low-precision training is not simply “use stochastic rounding and relax.” FP8 work, for example, is a whole systems design: E4M3 and E5M2 formats, scaling rules, higher-precision accumulation, optimizer-state choices, and model-class experiments.5 But stochastic rounding remains one of the places where arithmetic touches optimization rather than merely representing it.

The 2025 wave makes that explicit. Ozkara, Yu, and Park study stochastic rounding for LLM training under Adam and report BF16+SR results up to 6.7B parameters, with better perplexity in their setting and up to 1.54x higher throughput plus 30% lower memory than the mixed-precision baseline they compare against.6 Liu, Andronic, Gunduz, and Constantinides study mini-batch SGD with stochastic rounding for edge LLM training; their core message is not “rounding fixes everything,” but that batch size, weight quantization, and activation quantization interact differently with the variance introduced by lower precision.7

That is the right level of seriousness. Rounding is not garnish. It is a variance source, a bias source if implemented cheaply, and sometimes the difference between an update and no update.

A Pocket Turnstile

The lab below uses a deliberately plain model. It is not an FP8 emulator and it is not a claim about a real transformer. It stores one scalar weight on a fixed grid and optimizes the quadratic target \(w^\* = 1\).

The simplification is the point. If the mechanism is not visible in one dimension, it will be too easy to hide inside a billion parameters.

Exact arithmetic Round-to-nearest Stochastic mean Stochastic 10-90% Target

Deterministic toy simulation. The optimizer panel reuses the same gradient noise sequence for each mode so the visible spread comes from rounding. The exported JavaScript audit runs 390 assertions over local unbiasedness, threshold decisions, seeded ensembles, deterministic dead-zone settings, sub-grid accumulation, and clipping caveats.

With the default settings, the exact optimizer ends at 0.998. Round-to-nearest stops at 0.350, after step 9, because the proposed moves become too small to cross a grid midpoint. The stochastic-rounding ensemble mean ends at 0.999 over 160 runs. Same route, different gate policy.

The second panel repeats the idea without gradients. It adds a fixed micro update equal to 22% of one grid cell. Round-to-nearest keeps every passenger behind the line, so the accumulator stays at 0. Stochastic rounding recovers about 1.95 against an exact total of 1.98. The default audit reports 390 assertions: exact expectation checks for the rounding operator, deterministic branch-threshold checks, seeded ensemble reproducibility, 12 dead-zone settings where stochastic rounding keeps the optimizer closer to the target than deterministic rounding, and accumulator cases that make the clipping caveat explicit.

This is not magic. It is gate accounting.

Round-to-nearest preserves the most likely local value. Stochastic rounding preserves the local mean. When training is made of many small update requests, those are different operating promises.

The Gate Needs Real Randomness

There is a practical trap here. The clean formula for stochastic rounding assumes the rounder can compute the adjacent grid points and choose between them with probabilities proportional to the distances. Hardware usually cannot spend infinite randomness and infinite intermediate precision to round one number.

El Arar, Fasi, Filip, and Mikaitis study this exact gap. Their 2025 paper on limited-precision stochastic rounding models the number of random bits used by an implementation, and shows that the limited-randomness version can become biased; the bias disappears as the number of random bits grows.4

That caveat matters for machine learning. “Use stochastic rounding” is not a complete implementation note. A serious route map should say:

  1. which tensors are stored in low precision;
  2. which matrix products accumulate in wider precision;
  3. whether gradients, weights, activations, optimizer states, or updates are rounded stochastically;
  4. how scaling, clipping, subnormals, and overflow are handled;
  5. how random bits are generated, shared, or correlated across lanes;
  6. whether the claimed gain survives multiple seeds and batch sizes.

The last item is easy to understate. Stochastic rounding adds variance while removing a certain kind of systematic stagnation. Depending on the optimizer, loss surface, batch size, and scaling policy, that variance can help, hurt, or wash out. The right comparison is not a slogan. It is an ablation table.

The Dispatch Rule

I like to think of low-precision training as a station queue of update requests.

Round-to-nearest says:

ignore every request below the service threshold

Stochastic rounding says:

serve small requests with probability proportional to their size

That station metaphor is imperfect, but it points at the operational fact. A low-precision format does not merely compress the state of training. It changes which update requests are allowed through the gate.

This also explains why stochastic rounding is more than generic noise injection. Noise injection usually perturbs the computation and hopes the perturbation has a useful regularizing effect. Stochastic rounding is tied to a conservation law: conditional on the exact proposal, the rounded value has the right mean. Its variance is the price paid to buy that conservation on a coarse grid.

Sometimes the price is worth paying. Sometimes wider accumulators, loss scaling, master weights, or a different format are better. But once updates live near the grid spacing, “rounding mode” is no longer a numerical footnote.

It is part of the dispatch system.

Platform Notes

  1. Nick Higham, “What Is Stochastic Rounding?” 2020. Blog. ↩

  2. Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan, “Deep Learning with Limited Numerical Precision,” ICML 2015. PMLR. ↩

  3. Michael P. Connolly, Nicholas J. Higham, and Theo Mary, “Stochastic Rounding and Its Probabilistic Backward Error Analysis,” SIAM Journal on Scientific Computing, 2021. SIAM. ↩

  4. El-Mehdi El Arar, Massimiliano Fasi, Silviu-Ioan Filip, and Mantas Mikaitis, “Probabilistic Error Analysis of Limited-Precision Stochastic Rounding,” SIAM Journal on Scientific Computing, 2025. SIAM, arXiv. ↩ ↩2

  5. Paulius Micikevicius et al., “FP8 Formats for Deep Learning,” 2022. arXiv. ↩

  6. Kaan Ozkara, Tao Yu, and Youngsuk Park, “Stochastic Rounding for LLM Training: Theory and Practice,” AISTATS 2025. PMLR. ↩

  7. Taowen Liu, Marta Andronic, Deniz Gunduz, and George Anthony Constantinides, “Training with Fewer Bits: Unlocking Edge LLMs Training with Stochastic Rounding,” Findings of EMNLP 2025. ACL Anthology. ↩