The KV cache buys speed by storing every past token's Keys and Values. That storage is a clean formula that scales linearly with context length and concurrency — until, at long context, the cache outweighs the model itself.
Last page the cache saved us from recomputing the past — but that stored Key/Value has to live somewhere, and that somewhere is GPU memory. The bill is a clean formula:
kv_bytes = 2 (K and V) × layers × n_kv_heads × head_dim × seq_len × batch × bytes.
For our 7B preset, n_kv_heads × head_dim = d_model = 4096, so that's 512 KiB per token across all 32 layers — and it grows linearly with the sequence length and with the number of concurrent requests you're serving.
At long context the cache outweighs the model: 128k tokens is roughly 69 GB of cache against just 13 GB of fp16 weights. That is why long chats and big batches get expensive — and what paging (next lesson) and KV quantization are built to tame.
TRY → Drag sequence length to 128k — the cyan cache overruns the amber 80 GB line and passes the rose weights (the won't fit flag lights). Flip KV precision to int8 to halve it, or raise batch to stack concurrent requests. Every number recomputes from the formula — nothing is stored.
Per token, all layers: 2 × 32 layers × 32 heads × 128 head_dim × 2 B = 524,288 bytes = 512 KiB. Because n_kv_heads × head_dim = d_model = 4096, that's just 2 × 4096 × 32 layers × 2 B — two 4096-vectors (one K, one V) per layer, per token.
| sequence | batch | KV (fp16) | vs 13.2 GB weights |
|---|---|---|---|
| 4,096 | 1 | 2.0 GiB (2.15 GB) | 0.16× |
| 32,768 | 1 | 16.0 GiB (17.2 GB) | 1.30× |
| 131,072 (128k) | 1 | 64.0 GiB (68.7 GB) | 5.2× — bigger than the model |
| 4,096 | 32 | 64.0 GiB (68.7 GB) | 5.2× — batch does the same |
Switching KV to int8 halves every row (1 byte instead of 2). At 128k the cache is 68.7 GB — larger than the entire fp16 model — so weights + cache blow past an 80 GB card long before you hit the context limit.
GQA is the standard lever. Grouped-query attention cuts n_kv_heads from 32 down to, say, 8 — Keys and Values are then shared across query heads — and the cache shrinks 4× with no change to the query side. It's the single biggest structural saving on this formula.
Tie-back to Part 1 (Q15): the residual-stream vector flowing through the model is recomputed every step and thrown away; the K/V are the stored part that piles up here. That's the trade the cache makes — compute for capacity.
The KV cache costs a fixed number of bytes per token, per layer — 512 KiB/token for our 7B model. Total memory scales linearly with sequence length and batch, so long context and big batches are memory problems, not compute problems.