Running a model splits into two phases with wildly different economics. Prefill reads your whole prompt in one parallel pass; decode writes the answer one token at a time, dragging every weight out of memory for each one. Slide the lengths and watch decode swallow the wall-clock — every millisecond computed live, nothing replayed.
Running a model has two phases. Prefill reads your entire prompt in one parallel pass — every prompt token is processed at once — so the GPU's math units are the limit. This is compute-bound: lots of arithmetic per byte of weight fetched.
Decode then writes the answer one token at a time. To produce each new token the GPU must stream all the model's weights (the frozen Part-1 E / Wq / Wk / Wv / W1 / W2) out of memory, do a tiny amount of math, and emit one token — then do it all again. The math is trivial; the memory traffic is enormous. This is memory-bandwidth-bound.
Same total token count, wildly different cost: 512 prompt tokens prefill in about 22 ms; generating 512 tokens takes about 3.4 seconds. Decode is the wall-clock — and the rest of Part 2 is tricks that attack that decode number.
TRY → Push G up and watch the decode segment swallow the timeline — 8 extra output tokens cost more wall-clock than a thousand extra prompt tokens. Flip to int4: decode roughly quarters (fewer bytes to move), previewing lesson 4. The 156× ratio is per-token; it holds whenever P and G match.
N = 6.6B parameters (paramCount() = 6,607,077,376). One multiply-add is 2 FLOP, so a forward pass over one token is 2·N = 1.32×10¹⁰ FLOP, and the weights weigh N·2 B = 13.2 GB at fp16.
| phase | per token | bottleneck resource | cost |
|---|---|---|---|
| prefill | 2·N / peak = 1.32e10 / 312e12 | math units (FLOP/s) | 0.042 ms |
| decode | 13.2 GB / 2.0 TB/s | memory bandwidth | 6.61 ms |
Per-token ratio 6.61 / 0.042 = 156. Prefill shares that one pass across all P prompt tokens at once (they ride the same weight-read in parallel), so prefill ≈ 2·N·P / peak; at P=512 that's 21.7 ms. Decode gets no such sharing at batch 1 — every one of the G tokens pays the full weight-read again — so decode ≈ G · (weight_bytes / bandwidth); at G=512 that's 512 · 6.61 ms = 3.38 s.
Prefill does lots of math per byte fetched → compute-bound. Decode fetches all 13.2 GB to do only 13 GFLOP of useful work → bandwidth-bound: the GPU sits idle waiting on memory. Every trick in the rest of Part 2 — KV cache, quantization, batching — is a move against that one decode number.
Prefill reads the whole prompt in one parallel pass (compute-bound, cheap); decode writes one token at a time, re-streaming every weight each step (bandwidth-bound, expensive). Decode owns the wall-clock.