One picture holds the whole part. Arithmetic intensity is FLOPs per byte moved; a GPU's ridge point is where math and memory balance. A batch-1 decode step lands 156× below it — bandwidth-bound. Drag the batch slider and watch the operating point climb toward the ridge.
Arithmetic intensity = FLOPs performed per byte moved from memory. A GPU has a ridge point = peak FLOP/s ÷ memory bandwidth (≈156 FLOP/byte on an A100). Below the ridge you're bandwidth-bound — starved for data; above it you're compute-bound — the math units are the limit.
A batch-1 decode step is a matrix-vector product: each weight is read once and used for about 2 FLOPs, so its intensity is ≈ 1 FLOP/byte at fp16 — 156× below the ridge. The GPU spends the step waiting on memory, not doing math. That single fact is the why under everything in Part 2.
Quantization shrinks the bytes moved. The KV cache avoids re-moving past work. Batching raises intensity toward the ridge (AI ≈ 2·batch / bytes). Three levers, one plot — and none of them change a number the model outputs.
TRY → Start at batch = 1, fp16: the dot sits far left at 1 FLOP/byte, deep in the bandwidth zone. Drag batch up to 156 and the dot climbs the diagonal and touches the ridge — now compute-bound. Switch to int4: the same batch jumps 4× to the right (fewer bytes moved), so the ridge arrives at batch 39. Every number recomputes from AI = 2·batch / bytes.
The ridge. An A100 does 312 TFLOP/s of dense fp16 math and moves 2.0 TB/s from HBM. The ridge is their ratio: ridge = 312e12 / 2.0e12 = 156 FLOP/byte. To keep the math units fed, you must do at least 156 FLOPs for every byte you fetch. Fewer than that, and the units idle while data arrives — bandwidth-bound.
Batch-1 decode. Each weight is streamed once and used in one multiply-add ≈ 2 FLOPs. At fp16 a weight is 2 bytes, so AI = arithmeticIntensity(1, 2) = 2·1/2 = 1 FLOP/byte — a factor of 156 under the ridge. At int4 (0.5 byte/weight) AI = 2·1/0.5 = 4: four times higher, still far below 156. A single decode step simply doesn't do enough math per byte to be anything but bandwidth-bound.
Reaching the ridge. Compute-bound needs AI ≥ 156. Since AI = 2·batch / bytes, the crossover batch is ridge · bytes / 2:
| precision | bytes / param | AI at batch 1 | batch to reach ridge 156 |
|---|---|---|---|
| fp16 | 2 | 1 | 156 |
| int8 | 1 | 2 | 78 |
| int4 | 0.5 | 4 | 39 |
Above the ridge, adding batch gives you no more throughput — you're already compute-bound and every extra request just queues. Below it, adding batch is nearly free tokens, which is exactly why serving stacks batch aggressively.
Every Part-2 trick is one move on this plot:
bytes_per_param — raises AI (dot slides right) and cuts the bytes actually moved, so decode gets faster even without crossing the ridge.
batchingraises batch — walks the dot up the diagonal toward the ridge, amortizing one weight-read across many requests.
KV cacheremoves redundant K/V movement — so the bytes you do move each step are just the weights, keeping the intensity honest instead of drowning in recomputed past work.
None of them touch a single number the model outputs (Part-1 Q18) — they change only how the computation is scheduled and stored. The roofline is the map they're all navigating.
Arithmetic intensity ≈ 2·batch / bytes; the A100 ridge is 156. Batch-1 fp16 decode does ~1 FLOP/byte — 156× below the ridge, so it's bandwidth-bound. Quantization, KV cache, and batching all attack that one bottleneck: moving weights, not doing math.