Speculative Decoding Demo: Why Longer Drafts Stop Paying
The draft-and-verify speedup identity on a draft-length slider, with sampled cycles at your acceptance rate and the optimal k marked.
■ accepted draft tokens · ■ the rejection · ■ discarded, never used · ■ the target model’s own token, produced free by the same verification pass
2.77 tokens per cycle for 1.72 target-step-equivalents of work. At this acceptance rate the best draft length is k = 3, worth 1.64×.
- Expected accepted draft tokens
- 1.77
- …plus the target's own token
- 1.00
- Expected tokens per cycle
- 2.77
- Cost of a cycle, in target steps
- 1.72 (4 × 0.18 + 1)
- Speedup
- 1.612×
- Baseline generation rate
- 45.5 tok/s
- With speculation
- 73.3 tok/s
- Time for 500 tokens, baseline
- 11.0 s
- …with speculation
- 6823 ms
- Best draft length at this acceptance rate
- k = 3
(1 − α^(k+1)) / (1 − α), and the cycle costs k·c + 1 target-step-equivalents. That assumes acceptance is independent and identically distributed across positions within a draft, which is a simplification — in practice a draft that has gone wrong tends to keep going wrong, so real long-k performance is slightly below this. Verification is one target forward pass over all k+1 positions, which is why it is priced as a single step. Output is distributionally identical to unspeculated decoding; speculation is a latency optimisation, not a quality one.Why a longer draft eventually loses
Speculative decoding exists because generating one token at a time wastes a GPU. A decode step reads the entire model from memory to produce a single token, so the arithmetic units sit mostly idle. If a small draft model proposes several tokens and the large model checks all of them in one pass, that same memory read yields several tokens instead of one — and because the check is exact, the output distribution is unchanged. It is free speed, when it works.
Whether it works is one number, and the curve is where it becomes intuitive. Each extra draft token is only reached if every token before it was accepted, so its contribution is discounted by α raised to its position — the fifth draft token at 70% acceptance arrives about a sixth of the time. Its cost, meanwhile, is charged every single cycle whether or not it survives. Expected benefit decays geometrically, cost grows linearly, and somewhere they cross. Drag α down and watch the optimum walk left; push it above 90% and the curve keeps climbing well past k = 8.
The acceptance rate is the input you cannot fake, and it is far more variable than people expect. A draft model that agrees with the target 80% of the time on conversational English can fall below 40% on code or on a domain vocabulary it never saw, and the same setup that gave a 2× win in a demo gives nothing in production. Measure it on your own traffic: run the draft, run the target, count the matches.
One trade this page does not model: speculation costs extra compute for the same output. If you are latency-bound at low batch it is close to free, and if you are already throughput-bound at high batch it can make total capacity worse.