Speculative Decoding: Two Models, One Output

Decode is memory-bound: the hardware spends its time streaming weights, not computing. Speculative decoding exploits that idle arithmetic to check several guessed tokens in the time one token would have taken — and, remarkably, does it without changing the output distribution at all.

The idea, and the surprising part

A small, cheap draft model proposes the next k tokens. The large target model then evaluates all k proposals in a single forward pass — which costs it barely more than evaluating one, because it was bandwidth-limited anyway — and a rejection-sampling rule decides how many of the proposals to keep.

The surprising part, and the reason this is a serving technique rather than a quality trade, is that the acceptance rule is exact. Leviathan, Kalman and Matias (ICML 2023) and Chen et al. (DeepMind, 2023) independently showed that with the right accept/reject test plus an adjusted resample on rejection, the sequence of tokens emitted is distributed identically to sampling from the target model directly. Not approximately: identically. The draft model influences speed only.

The algorithm

# q = draft model distribution, p = target model distribution
draft k tokens x_1..x_k from q, recording q(x_i)
run the target ONCE over the prefix + all k drafts -> p(x_i) for each i

for i in 1..k:
    if p(x_i) >= q(x_i):
        accept x_i                       # target likes it at least as much
    else:
        accept x_i with probability p(x_i) / q(x_i)
        on reject:
            resample from normalised max(0, p - q)
            stop                          # discard x_{i+1}..x_k

if all k accepted:
    emit one bonus token sampled from the target's k+1-th distribution

Two details carry the whole result. The rejection branch does not resample from p — it samples from the normalised positive part of p − q, which is exactly the mass the draft under-weighted, and that correction is what makes the composite exact. And a fully accepted block yields a free extra token, because the target’s forward pass already produced a distribution at the last position. So a block of k drafts can emit up to k+1 tokens.

What the speedup depends on

Let α be the probability that any single drafted token is accepted, and assume acceptances are independent — an approximation, since a rejection tends to make later drafts worse, but the standard one. The expected number of tokens emitted per verification step is a truncated geometric series:

E[tokens per step] = (1 - a^(k+1)) / (1 - a)

a = 0.8, k = 4  ->  (1 - 0.328) / 0.2   = 3.36 tokens per target pass
a = 0.6, k = 4  ->  (1 - 0.078) / 0.4   = 2.31
a = 0.4, k = 4  ->  (1 - 0.010) / 0.6   = 1.65
a = 0.8, k = 8  ->  (1 - 0.134) / 0.2   = 4.33   (diminishing: k=4 gave 3.36)

That is the numerator of the speedup. The denominator is the cost: each step now runs the draft model k times plus the target once. If the draft costs a fraction c of the target per token, the wall clock per step is roughly 1 + k·c target-passes, so:

speedup ~= (1 - a^(k+1)) / ((1 - a) * (1 + k*c))

a = 0.8, k = 4, c = 0.05  ->  3.36 / 1.20  = 2.8x
a = 0.4, k = 4, c = 0.05  ->  1.65 / 1.20  = 1.4x
a = 0.4, k = 8, c = 0.15  ->  1.67 / 2.20  = 0.76x   <- slower than not doing it

Three readings. Acceptance rate is the dominant term, and it is a property of how similar the draft is to the target on your text — high on boilerplate, code and structured output, much lower on open-ended prose. Longer drafts have diminishing returns because the series saturates while the cost stays linear. And with a draft that is not cheap enough, speculation is a net loss, which is why serving stacks size draft models at a small percentage of the target and tune k per workload.

Note also what speculation does not buy: throughput at high batch sizes. It converts spare arithmetic into latency, and at large batch the arithmetic is no longer spare. This is why it helps most on low-concurrency, latency-sensitive serving and can be counterproductive on a saturated server.

What the papers report

Reported speedups belong to the papers that ran them, on their models and their tasks, and should be read as such rather than as a number to expect:

• Leviathan, Kalman & Matias, ICML 2023 — “Fast Inference from Transformers via Speculative Decoding”. Reports roughly 2–3× wall-clock speedups on T5-XXL for translation and summarisation, with the acceptance-rate analysis the maths above follows.
• Chen et al., 2023 — “Accelerating Large Language Model Decoding with Speculative Sampling”. Reports roughly 2–2.5× on Chinchilla-scale models, and states explicitly that sample quality is unchanged.
• Medusa (Cai et al., 2024) — replaces the separate draft model with extra prediction heads on the target itself, avoiding the second model entirely; reports speedups in a similar band on its own evaluation.
• EAGLE and successors — draft at the feature level rather than the token level to raise acceptance; the published numbers are higher again, on their own benchmarks.

The honest summary of the literature is that the technique reliably helps at low batch size, that the size of the help is workload-dependent through α, and that anybody quoting one universal multiplier has stopped reading at the abstract.

What it looks like from the API

Because output is distributionally identical, you cannot detect speculation from the text. You can sometimes see it in the timing: with block verification, tokens tend to arrive in bursts of several followed by a pause, rather than at an even cadence. If you compute inter-token latency, a bimodal distribution — many near-zero gaps and periodic larger ones — is the signature.

It also means a provider can turn speculation on or off without announcing it, and your average tokens-per-second can move without any change on your side. That is one more reason to watch the distribution rather than the mean.

Two variants are worth knowing because they change where the technique applies. Prompt lookup, sometimes called n-gram speculation, dispenses with the draft model entirely: it searches the existing context for a matching n-gram and proposes the continuation that followed it last time. It costs essentially nothing, and its acceptance rate is high precisely when the output copies the input — editing a document, rewriting a code file, answering from a retrieved passage. For those workloads it is close to free speedup; for open-ended generation it does nothing at all.

The other is sampling temperature. Acceptance depends on the two distributions agreeing, and temperature reshapes both. At temperature 0 the target is a point mass, so a draft is accepted only when it names the same argmax; at higher temperatures the target flattens, which generally makes agreement easier but also makes the corrected resample on rejection more consequential. If you change temperature and your tokens-per-second changes, this is why — and it is the one case where a sampling parameter has a direct, mechanical latency effect.