The frozen weights from Part 1 don't need full precision. Round them onto a small set of levels and a 7B model drops from 13.2 GB to 3.3 GB — 4× less to store and to move.
Weights don't need full precision — the model shrugs off small rounding. Quantization maps a range of real values onto evenly spaced levels: pick a scale, then round each weight to the nearest level. int8 gives 256 levels, int4 just 16 — and it still works.
Fewer bits shrinks memory and bandwidth. Our 7B-class model is 13.2 GB in fp16 but 3.3 GB in int4 — 4× smaller. And since decode is bandwidth-bound (lesson 1), 4× fewer bytes to stream is up to 4× fewer milliseconds per token.
The hard part is outliers: a few giant weights stretch the range so everyone else crams into a handful of levels. Per-group scales and protecting outliers (the GPTQ/AWQ idea) rescue the precision.
TRY → Drop the bits from fp16 → int4 → int2 and watch the dots collapse onto fewer red gridlines, the memory tank shrink, and the rounding-error bar grow. Then at int3/int2 flip on per-group scales and protect outliers — the error bar visibly shrinks as the two planted outliers stop crushing everyone else.
Symmetric absmax quantization of a tiny weight vector, computed live by quantizeSym(QUANT_W, 4) from the shared helpers:
absmax 0.42, scale = 0.42 / 7 = 0.06, codes [2,-7,6,-2,1,5,-6,4], mean-abs-error 0.015 (~3.6% of the 0.42 range), 16 levels.
Whole-model memory, from weightBytes(MODEL, BYTES[…]) on the 6.6B-param preset: fp16 13.2 GB → int8 6.6 GB → int4 3.30 GB, exactly 4× smaller than fp16. int3 2.48 GB, int2 1.65 GB.
Per-tensor forces every weight into the same 16 levels; per-group gives each small block its own scale, so small weights aren't crushed by one big outlier. Modern low-bit formats: int4 · nf4 · fp8 · mxfp4 · nvfp4.
Quantization rounds each weight onto a handful of levels. It cuts weight memory and bandwidth ~4× (fp16 → int4) for a small, non-linear quality cost — and per-group scales plus outlier protection buy back most of that cost.