One decode step streams the entire model no matter how many requests ride along. So push many requests through that single weight-read and the cost is shared — total tokens per second (throughput) multiplies, while any one reply arrives no sooner.
Decode is bandwidth-bound: every step streams all the model's weights out of memory to make progress — 13.2 GB moved for the 7B preset, whether one request is running or thirty-two. So push many requests' tokens through that single weight-read at once and the expensive read is amortized across everyone. More total tokens per second — throughput — for the same bytes moved.
Static batching launches a fixed batch and waits for the longest member to finish. A request that only needs 6 tokens sits idle for the other 14 steps while a 20-token neighbour keeps going — its slot burns capacity doing nothing until the whole batch resets.
Continuous batching swaps a finished sequence out and a waiting one in every step — interleaving fresh prefills with in-flight decodes — so no slot ever idles. It multiplies throughput; it does not make any single reply arrive sooner. Throughput is not latency.
TRY → Flip to static and watch the short rows go hatched-idle while the longest slot runs out the clock — utilization drops. Flip back to continuous and those idle cells refill with a new-coloured request; utilization jumps to ~100%. Then edit the lengths: make them all equal and static catches up; make one huge and static's waste (and continuous's win) grows.
Four slots, decode lengths [20, 8, 12, 6] → Σ = 46 useful tokens, max = 20, B = 4.
| mode | steps | useful slot-steps | total slot-steps | utilization |
|---|---|---|---|---|
| static | 20 | 46 | 4 × 20 = 80 | 46 / 80 = 57.5% |
| continuous | ⌈46/4⌉ = 12 | 46 | ≈ 46 (slots refill) | ≈ 100% |
Static runs 20 steps because it must wait for the longest member; only 46 of the 80 slot-steps do real work, so the GPU sits 1 − 0.575 = 42.5% idle. Continuous keeps all four slots full by swapping finished sequences out and queued ones in, clearing the same 46 tokens in 46 / 4 = 11.5 steps.
Throughput speedup = static steps / continuous steps = 20 / 11.5 = 1.74× — which is exactly 1 / 0.575, the reciprocal of static utilization. The single weight-read per step is the fixed cost; filling every slot amortizes it across 1.74× more tokens.
Throughput ≠ latency. Batching multiplies total tokens/second across all users, but a single request still advances exactly one token per step — its own reply arrives no sooner. This is why one served model answers thousands of people at once for the same weight-reads. (Paging, the previous page, is what makes room in VRAM for all those concurrent KV caches.)
One decode step reads the whole model regardless of batch size, so packing more requests into it is nearly free throughput. Continuous batching keeps every slot busy by swapping finished sequences out mid-flight — utilization climbs from ~57% to ~100%, throughput ~1.74× — while each reply's latency stays the same.