A small, fast draft model proposes the next few tokens; the big target model verifies them all in a single parallel pass — several tokens for the price of one expensive step, with output identical to plain decoding.
Decode's expensive step reads the whole model just to make one token — wasteful. So a small fast draft model proposes the next k tokens cheaply, guessing where the big model is probably going.
The big target model then checks all k in a single parallel pass — the same parallelism as prefill, so verifying k tokens costs about one normal decode step. It keeps the longest prefix it agrees with and rejects from the first disagreement, always emitting one correct token at the break.
When the draft is right you get several tokens per expensive step — and the text is exactly what the target would have written alone. Higher acceptance rate = more free tokens.
TRY → Drag acceptance rate a and watch E and net speedup recompute from the formula (no replay). Switch draft length k or flip aligned ⟷ weak draft to re-run the accept/reject animation — the aligned toggle sets a high a so the animation and the numbers agree.
Expected tokens per verify pass, drafting k tokens with independent per-token acceptance probability a:
E = (1 − a^(k+1)) / (1 − a). Net speedup folds in the draft's own cost — the draft runs the small model k times per pass, at cost fraction c = 0.1 of a target step each: speedup = E / (1 + k·c).
| a, k | a^(k+1) | E (tokens/pass) | net speedup |
|---|---|---|---|
| a = 0.8, k = 4 | 0.328 | 3.36 | 2.40× |
| a = 0.9, k = 4 | 0.590 | 4.10 | 2.93× |
| a = 0.5, k = 4 | 0.031 | 1.94 | 1.38× |
| a = 0.8, k = 2 | 0.512 | 2.44 | 2.03× |
At the default a = 0.8, k = 4: (1 − 0.8⁵) / 0.2 = 0.6723 / 0.2 = 3.36 tokens per verify pass, and 3.36 / (1 + 4·0.1) = 3.36 / 1.4 = 2.40× net.
Why the target pass is cheap: verifying k proposed tokens is one parallel forward pass over k positions — the same parallelism as prefill (lesson 1), so it costs about one decode step, not k of them. Longer k helps only while acceptance stays high — a wrong early guess wastes the whole tail after it. And the output is identical to plain decoding: every committed token is target-verified, and rejected drafts are discarded, so the text is exactly what the target alone would have produced.
A cheap draft proposes k tokens; the target verifies them all in one parallel pass and keeps the run it agrees with. You get more tokens per expensive step — identical output, higher acceptance = more speed.