Every earlier token's Key and Value are computed once and stored. Without that cache, each decode step re-derives the K and V of every token before it — and the wasted work piles up as a triangular number. Count it live.
In attention (Part 1, lesson 6) a new token's Query is scored against every earlier token's Key, then blends every earlier token's Value. Here's the thing: those earlier Keys and Values never change as you generate the next word, and the next, and the next.
So compute each token's K and V once and store them — per token, per layer, per attention head — in the KV cache. Each new step then computes only its own K and V and reads the rest straight out of memory. The Query is not cached: you only ever use the current token's Query, and a past token's Query is never needed again.
Without the cache, step t has to recompute the K/V of all t−1 tokens before it. Do that for a whole generation and the redundant work is 1 + 2 + … + n = n(n+1)/2 — the triangular blow-up. With the cache it's just n.
TRY → Flip to no cache · recompute and watch the grid re-light the whole prefix every step while the "work this step" bar climbs — then drag n to 2048 and read the ratio hit ×1024.5. The two big numbers are computed straight from recomputeWork(n), not from the animation.
Generating n tokens, counting each token's K/V computation once:
| n tokens | with cache (n) | no cache · n(n+1)/2 | ratio (n+1)/2 |
|---|---|---|---|
| 8 | 8 | 36 | ×4.5 |
| 64 | 64 | 2,080 | ×32.5 |
| 256 | 256 | 32,896 | ×128.5 |
| 1024 | 1,024 | 524,800 | ×512.5 |
| 2048 | 2,048 | 2,098,176 | ×1024.5 |
At n = 2048: without a cache you'd perform 2048·2049/2 = 2,098,176 token-K/V computations; with the cache, exactly 2,048 — the ratio is (2048+1)/2 = 1024.5×. The waste grows with n, because each new step drags the entire history along with it.
What's actually stored is the computed Key and Value vectors for every past token — one K and one V per token, per layer, per attention head — never the Query (a past Query is never reused) and never the raw text. The cache trades compute for memory: you stop recomputing K/V, but now you have to store them, and that storage grows linearly with context — which is exactly the next page's problem.
Past Keys and Values never change, so cache them once and reuse them. That turns each decode step's attention work from the whole prefix into a single token — collapsing the triangular n(n+1)/2 down to n.
n(n+1)/2 redundant computes into n. The weights (the next lessons' bottleneck) still stream every single step.