A GPU is a few hundred tiny processors bolted to a big, slow pool of memory — with two tiny-but-fast tiers wedged between them. This one picture is the physical stage every trick in Part 2 was playing on: the weights live far away in VRAM, and dragging them up is the whole cost.
A GPU is a few hundred tiny processors — SMs (Streaming Multiprocessors, ~108 on an A100) — wired to a big, slow pool of memory: VRAM / HBM, 80 GB at 2 TB/s. Between the SMs and that pool sit two tiny-but-fast tiers: on-chip SRAM (~192 KB per SM) and registers.
Everything the model is lives in VRAM: the frozen weights (13.2 GB), the KV cache (Part-2 lesson 3), the activations. The SMs can only compute on what's already been dragged up into SRAM and registers first.
Here's the wall. SRAM is ~10× faster than VRAM but holds only kilobytes, so a decode step must stream all 13.2 GB of weights up from VRAM just to do a little math — VRAM bandwidth is the bottleneck, exactly the "bandwidth-bound" of Part-2 lesson 1. Every move in Part 2, and FlashAttention next page, is a play against this hierarchy.
TRY → You start with the 13.2 GB weights — they blow past the SRAM line and land in slow VRAM. Now click Flash tile · 64×64: at 56 KiB it drops under the 192 KiB line and lands on-chip (SRAM lights cyan). Then pick Score matrix and drag seq up — the bar grows from 512 KiB to tens of MiB, always deep in VRAM. Only the Flash tile ever fits on-chip; everything recomputes from fits SRAM ⇔ bytes ≤ 192 KiB.
The A100, tier by tier. 108 SMs do the multiply-adds. Each SM has its own 192 KB of SRAM (~19 TB/s) and a 256 KB register file. All 80 GB of HBM VRAM sit off to the side at 2.0 TB/s — huge, but ~10× slower to read than SRAM (19 / 2.0 = 9.5×). That gap is the whole story.
What each thing costs, and where it lands (fp16, from the shared helpers):
| item | formula | bytes | fits SRAM? | tier |
|---|---|---|---|---|
| weights fp16 | paramCount × 2 | 13.2 GB | no | VRAM |
| KV cache · 4k | 2·32·32·128·4096·2 | 2.15 GB | no | VRAM |
| score matrix · n=4096 | 4096² × 2 | 32 MiB | no | VRAM |
| Flash tile · 64×64 | (3·64·128 + 64²)×2 | 56 KiB | yes ✓ | SRAM |
The 6.6 ms. A batch-1 decode step reads all 13.2 GB of weights up from VRAM to do ~13 GFLOP of math. At 2.0 TB/s the fetch alone takes 13.2e9 / 2.0e12 ≈ 6.6 ms — the exact number from Part-2 lesson 1. The math is trivial; the fetch is the cost. Only the 56 KiB Flash tile is small enough to sit on-chip while it's worked on — which is precisely the door FlashAttention walks through next page.
This is the roofline (Part-2 lesson 8) rendered in hardware: below the ridge point you're waiting on memory, and the memory you're waiting on is this VRAM, two tiers down from the SMs.
A GPU is a memory hierarchy: registers → 192 KB SRAM → 80 GB VRAM, fast-and-tiny to slow-and-huge. The SMs compute, but only on what's dragged up into SRAM first — and VRAM bandwidth is the wall that makes batch-1 decode bandwidth-bound. Only a sliver (a 56 KiB Flash tile) ever fits on-chip; the 13.2 GB of weights never do.