You know how an LLM works — tokens, attention, next-token prediction, prefill vs decode. This part is the other half: how a 140-gigabyte model gets squeezed onto a single GPU and made to answer in milliseconds. Every chapter is one trick, and every trick beats one of these bottlenecks.
Running a model splits in two. Prefill reads your whole prompt in one parallel pass — lots of math at once, so the GPU's math units are the limit (compute-bound). Decode writes the answer one token at a time, and for every token the GPU must stream all the model's weights out of memory just to take one step. The math is tiny; the memory traffic is huge (memory-bandwidth-bound). That one re-read is what the whole page is about.
THE TELL → During decode the math bar sits nearly empty while the bandwidth bar is pinned at 100%. The GPU is mostly waiting on memory, not computing — so faster memory helps generation more than faster math.
A weight is a floating-point number in three parts: a sign (±), an exponent (roughly how big), and a mantissa (the precise digits). fp32 spends 32 bits per weight, fp16 16, bf16 also 16 but trades mantissa precision for more range. Fewer bits = less to store and less to move — and since decode is bottlenecked on moving weights, fewer bits is fewer milliseconds.
RANGE vs PRECISION → fp16 and bf16 are both 16 bits, but bf16 keeps a wide exponent (won't overflow) at the cost of a shorter mantissa (coarser steps). Same size, different trade — that's why training loves bf16.
Weights don't need full precision — the model shrugs off small rounding. Quantization maps a range of real values onto a small set of evenly spaced levels: pick a scale, then round each weight to the nearest level. 8-bit gives 256 levels, 4-bit just 16 — and it still works. A 70-billion-parameter model is ~140 GB in fp16, but ~35 GB in 4-bit: the difference between four GPUs and one.
THE HARD PART → A few giant outlier weights stretch the range so everyone else crams into a handful of levels. Per-group scales (a separate scale per small block) and protecting outliers (keeping the few big ones in high precision — the idea behind GPTQ/AWQ) rescue the precision. Modern low-bit formats: int4 · nf4 · fp8 · mxfp4 · nvfp4.
In attention, each new token needs the keys and values of every earlier token — and those never change as you generate. So compute each token's K and V once and cache them; each new token then only computes its own and reads the rest. Decode goes from redoing all past work every step to doing just one token's worth. The catch: the cache grows with every token, and at long context it can outweigh the model itself.
THE NEW PROBLEM → Push context and concurrency up and the cache balloons — soon rivalling the weights. That's what the next chapter's paging and quantized KV cache exist to tame.
Two problems with the growing cache, two fixes. Reserve each request's maximum cache up front and most of it sits empty — paged attention (the idea behind vLLM) chops the cache into fixed pages placed wherever there's room, like an OS's virtual memory, so far more requests fit. And when many requests share a prefix — a system prompt, a document, earlier chat turns — prefix caching computes it once and reuses it.
MULTI-TURN → Each new turn of a chat reuses the cache of all previous turns — the shared prefix (system prompt + history) is computed once and simply extended. Paging never changes the answer; it's pure memory management.
Decode's expensive step reads the whole model to make one token — wasteful. So a small fast draft model proposes several tokens cheaply; the big target model then checks them all in a single parallel pass and keeps the longest run it agrees with, rejecting from the first disagreement. When the draft is right, you get several tokens for the price of one expensive step — and the output is exactly what the target would have written alone.
One decode step reads the entire model no matter what — so push many requests' tokens through that single read at once, and the expensive weight-read is amortized across everyone. Static batching wastes it by waiting for the whole batch to finish; continuous batching swaps a finished request out and a new one in every step, keeping the GPU near 100% busy.
THROUGHPUT ≠ LATENCY → Batching multiplies total tokens/second across users; it doesn't make any single reply arrive sooner. It's the reason a served model can answer thousands of people at once for the same weight-reads.
A quick sweep of the other big levers — each with the bottleneck it beats.
Compute attention in small tiles that stay in the GPU's fast on-chip memory, so the huge N×N attention matrix is never written out to slow memory.
Split the feed-forward layer into many experts and route each token to just one or two — huge total capacity, but only a slice of the weights runs per token.
Train a small "student" model to imitate a big "teacher" — much of the quality at a fraction of the size and cost. A genuinely smaller model to run.
Answer easy tokens or queries with cheap computation — fewer layers, or a small model first — and only escalate the hard ones to the full model.
The payoff. ● compute-bound · ● memory-bandwidth · ● memory-capacity. A dashed ◇ marks a cost the trick adds.
| Technique | Compute | Bandwidth | Capacity |
|---|