The suspicious part is not that speculative decoding can be fast.

The suspicious part is that it can be fast without changing the samples.

A small draft model runs ahead with a few guesses. The large target model checks those guesses in one parallel pass. Some are accepted, the first failure is repaired, and the loop begins again. It sounds like approximation wearing a systems jacket. Surely the little model has smuggled its preferences into the output?

Not if the correction is the right one.

The important sentence is:

the draft model is a proposal distribution, not a replacement model

Leviathan, Kalman, and Matias introduced speculative decoding as a way to sample from autoregressive Transformers faster while preserving the output distribution.1 Chen, Borgeaud, Irving, Lespiau, Sifre, and Jumper called the stochastic version speculative sampling and gave the compact modified rejection-sampling proof.2 The speedups are the headline. The more delicate contract is the real story: the scout may be wrong, because the verifier prices every wrong guess.

The Coin Flip at One Token

Forget sequences for a moment. Suppose the target model has next-token distribution (p), and the draft model has next-token distribution (q).

The draft samples a token (y \sim q). The verifier accepts that token with probability

\[a(y)=\min\left(1,\frac{p(y)}{q(y)}\right).\]

If the token is accepted, emit (y). If it is rejected, do not simply sample from (p). Sample from the residual distribution

\[r(x) \propto \max(0, p(x)-q(x)).\]

That residual is the missing target mass: tokens the draft under-proposed relative to the target.

Now compute the probability that the final emitted token is (x). It can happen in two ways. The draft can propose (x) and be accepted, contributing (\min(p(x), q(x))). Or some draft token can be rejected, after which the residual sampler contributes exactly (\max(0, p(x)-q(x))). Therefore:

\[\Pr[X=x] = \min(p(x),q(x))+\max(0,p(x)-q(x)) = p(x).\]

That is the whole trick. The output distribution is the target distribution. The draft is only deciding which parts of (p) can be cheaply reused. It is not trusted; it is audited.

The total acceptance probability is

\[\alpha=\sum_x \min(p(x),q(x)).\]

This is also (1-\mathrm{TV}(p,q)), where (\mathrm{TV}) is total variation distance. A better draft increases overlap. It does not change the target distribution; it changes how often the verifier can keep the proposal.

A Rejection-Sampling Lab

The lab below builds a tiny Markov language model with eight toy tokens. The target model is the distribution we must preserve. The draft model is a noisy proposal model whose similarity to the target you can change.

The visualization separates three ideas that are often blurred together:

  1. the draft distribution can be biased,
  2. the corrected speculative distribution is algebraically identical to the target,
  3. speed depends on overlap, lookahead length, and draft cost.

The Audit tile is generated by the same JavaScript that draws the lab. Its exported runAudit() runs 20 deterministic checks: parameter sanitation, probability normalization, residual-mass algebra, the identity (\alpha=1-\mathrm{TV}(p,q)), seeded trace determinism, legacy latency-ledger balance, and a 1,296-setting grid over certainty, draft match, draft length, draft cost, sequence length, and seed. With the default controls, the one-token overlap is 92.47%, the exact corrected TV distance is 0 to displayed precision, and the deliberately wrong fallback has TV distance 2.79%. The sampled 96-token trace uses 20 target calls and 100 draft calls for a toy 3.31x speedup, close to the 3.46x accounting estimate. Across the grid, the worst exact TV distance is (1.95\times10^{-16}), all speed values are finite, and no trace emits the wrong number of tokens.

target distribution draft proposal corrected / residual rejection mass

The corrected TV distance is computed from the exact finite distribution, not from the chart. Empirical sequence speed uses this toy Markov model and a draft-cost knob; it is an inference accounting model, not a GPU benchmark.

Why the Residual Matters

The tempting wrong algorithm is:

propose from draft
accept with min(1, p/q)
if rejected, sample from the target p

That sounds reasonable. It is not exact.

The accepted part already contains (\min(p,q)). If the fallback samples from the whole target distribution again, it gives probability mass to tokens whose target mass was already fully covered by the draft. The fallback must sample only the target mass that is still missing.

This is the role of ((p-q)^+). It is not an implementation detail. It is the conservation law.

The lab reports “wrong fallback TV” because this mistake is easy to make when explaining the method informally. The wrong fallback may look close when the draft is good. But close is not the same as exact, and speculative decoding’s strongest claim is exactness up to numerical effects.

When One Guess Becomes a Block

For a sequence, apply the same one-token machine repeatedly from left to right. The draft proposes (\gamma) tokens. The target scores the proposed continuation in parallel. The algorithm checks the draft tokens in order:

accept token 1 or correct it
if accepted, accept token 2 or correct it
if accepted, accept token 3 or correct it
...

The first rejection stops the block. If every drafted token is accepted, the target logits computed at the end of the block can also produce one more token. That is why the idealized expected yield is

\[1+\alpha+\alpha^2+\cdots+\alpha^\gamma = \frac{1-\alpha^{\gamma+1}}{1-\alpha},\]

with the obvious (\gamma+1) limit when (\alpha=1).

This equation is not a law of nature. It assumes a constant independent acceptance probability. Real language-model acceptance varies by context, temperature, tokenization, prompt domain, and draft architecture. But it is a good first accounting identity. A draft length of 8 does not mean 8 free tokens. It means you bought a geometric tail whose value depends on overlap.

The 2025 “Decoding Speculative Decoding” study is useful because it pushes against a naive story: draft-model language-modeling accuracy does not by itself determine speculative-decoding throughput, and draft latency can become the bottleneck.3 A slightly worse but much faster draft can be the better systems object.

What Is Actually Being Verified

Speculative decoding turns target decoding into a verifier-shaped operation. The target model still evaluates the proposed continuation. This is why the method can preserve the distribution. The expensive model has not been replaced; its serial interaction pattern has been changed.

That distinction matters for product claims.

If you skip verification, you are no longer doing exact speculative decoding. If you use approximate acceptance rules, typical acceptance, tree pruning, or multi-head predictors, you may get a useful engineering tradeoff, but the contract has changed. Medusa, for example, avoids a separate draft model by adding multiple decoding heads and verifying candidate continuations with tree-based attention; its paper distinguishes lossless and quality-preserving variants rather than pretending every acceleration has the same guarantee.4

The clean mental model is borrowed from rejection sampling:

proposal quality controls cost
acceptance correction controls correctness

A bad proposal distribution is not invalid. It is expensive. A proposal that never overlaps with the target will be rejected often, and the system will spend draft work to get little speed. But as long as the correction is implemented correctly, the target distribution is still recovered.

What Makes a Useful Scout?

The draft model wants to be close to the target where the target has high mass, but closeness is not the only objective. It also wants to be cheap, easy to serve, cache-friendly, and compatible with the target’s tokenizer and sampling policy.

This creates strange design pressure. A draft model can be too large to help. A small model can have worse benchmark perplexity but better speculative utility if it produces high-overlap tokens cheaply. N-gram or prompt-copying drafts can be useful in repetitive regimes even though they are not impressive language models. The original speculative decoding paper explicitly notes that negligible-cost approximation models such as n-grams can still have nonzero acceptance and therefore nonzero speedup in some tasks.1

So “draft quality” should not mean one scalar. For serving, the useful question is closer to:

how much accepted target mass do we buy per unit draft latency?

That ratio is contextual. Code completion, structured JSON, boilerplate agent traces, and repeated editing tasks may have long stretches where the future is easy to guess. Open-ended high-temperature writing may not. A static speculation depth is therefore a compromise across contexts.

The Theorem Is Small; the System Is Not

The proof fits in three lines. The deployment does not.

The verifier pass must be arranged so that the target scores the candidate continuation efficiently. Sampling code must apply exactly the same temperature, top-k, top-p, or other probability transformation to the target and draft distributions before the accept/reject rule. Floating-point differences can matter. Batch scheduling can turn a latency win into a throughput loss. KV-cache pressure can erase the expected gain. Multi-user serving changes the objective from “one request got faster” to “the tail of the queue got better.”

That is not just theoretical caution. The current vLLM documentation describes speculative decoding as a way to reduce inter-token latency under medium-to-low-QPS, memory-bound workloads, says real gains depend on model family, traffic pattern, hardware, and sampling settings, and separates theoretical losslessness from numerical and batch-size caveats.5 Hugging Face’s assisted-decoding docs draw another practical boundary: classic assistant-model speculative decoding works best with a much smaller assistant and the same tokenizer, while universal assisted decoding adds re-encoding and alignment machinery for different tokenizers.6 TensorRT-LLM states the systems assumption even more bluntly: speculation needs cheap token proposals and more than one validated draft token per target pass to become a latency win.7

That is why I like the small rejection-sampling view. It refuses two bad stories at once.

The first bad story is that speculative decoding is just approximation. No: the exact algorithm reconstructs the target distribution.

The second bad story is that exactness makes speed automatic. Also no: exactness only says the samples are right. Speed comes from the economics of accepted draft mass, verifier parallelism, and hardware utilization.

The small model is allowed to be wrong.

It is not allowed to be unpriced.

  1. Yaniv Leviathan, Matan Kalman, and Yossi Matias, “Fast Inference from Transformers via Speculative Decoding”, ICML 2023. ↩ ↩2

  2. Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper, “Accelerating Large Language Model Decoding with Speculative Sampling”, 2023. ↩

  3. Minghao Yan et al., “Decoding Speculative Decoding”, NAACL 2025. ↩

  4. Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao, “Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads”, 2024. ↩

  5. vLLM documentation, “Speculative Decoding”, accessed June 15, 2026. ↩

  6. Hugging Face Transformers documentation, “Assisted decoding”, accessed June 15, 2026. ↩

  7. NVIDIA TensorRT-LLM documentation, “Speculative Sampling”, accessed June 15, 2026. ↩