Part 1 · Lesson 12/15

One block, stacked N times

You've now met every piece. Bundle them in order — attention, add back, FFN, add back, with a layer-norm to keep the numbers tidy — and you have one transformer block. A model is that same block repeated. Drag the layer slider and watch a token's vector drift, layer by layer, toward its contextual meaning.

⏱ ~5 min 🎚 drift, computed live math: light

Every step in the last few lessons was a piece of the same machine. Stack them in order and the shape appears: attention (tokens look at each other) → add back → FFN (each token thinks alone) → add back. That whole unit is one transformer block.

One honest extra line: between sub-steps a layer-norm rescales the vector — roughly recenter to mean 0, rescale to a standard size — so that after many add-backs the values don't blow up or vanish. That's its whole job here: keep the running total numerically tidy.

A real model is this same block repeated N times (dozens to ~100), each copy with its own learned weights. The vector that flows through gets refined block by block. Below, watch it's vector drift from raw embedding toward its contextual, cat-flavored meaning as the layers stack up.

The block, and a token drifting through it · computed live

v_in → attention + resid → FFN + resid → layer-norm → v_out ↻ × N

One block = attention → +resid → FFN → +resid → (layer-norm). Each layer nudges the vector; early layers move it fast, later layers fine-tune.

layer depth L (how many blocks the vector has passed through) L = 0 / 6

it's vector at layer 0

animacy action state function

block: v → v + attn(v) → v + ffn(v) → layer-norm
(toy step used here): v_L = v_L−1 + 0.4·(target − v_L−1)

This is a toy. One real block runs a full attention pass plus an 8-neuron FFN, each with its own learned weights and a real layer-norm. To keep the drift visible and traceable, this widget stands in one compact step — nudge the vector a fixed fraction (0.4) toward a single fixed cat-flavored target — for the whole attention + FFN + residual stack of one block. The shape of the story (fast early, fine-tuning late; a running total refined layer by layer) is real; the exact arithmetic of a block is not this one line.

TRY → Drag the layer slider from 0 to 6 and watch the animacy bar climb (0.80 → 1.37) while function fades (0.30 → 0.11) — the vector sheds its raw it-ness and gains context. The marker on the line-plot moves with it. Hit Play to sweep all six layers; every number is recomputed by driftTo(L), not read from a table.

Show me the actual numbers (the full trajectory)

Each step applies v_L = v_{L−1} + 0.4·(attractor − v_{L−1}) with attractor [1.4, 0.3, 0.25, 0.1], starting from it's raw embedding v_0 = [0.8, 0.1, 0.2, 0.3]. Six blocks later:

layer	animacy	action	state	function
L0	0.80	0.10	0.20	0.30
L1	1.04	0.18	0.22	0.22
L2	1.18	0.23	0.23	0.17
L3	1.27	0.26	0.24	0.14
L4	1.32	0.27	0.24	0.13
L5	1.35	0.28	0.25	0.12
L6	1.37	0.29	0.25	0.11

Animacy climbs 0.80 → 1.37 (toward cat's 1.0-and-then-some, pushed up by context), function drops 0.30 → 0.11 — it stops looking like a bare pronoun and starts looking like the animate thing it refers to. Notice the step sizes shrink: L0→L1 moves animacy +0.24, but L5→L6 only +0.02. That's the fixed-fraction pull closing in on the target — fast early, fine-tuning late.

This is a toy. Real blocks each have their own learned attention + FFN weights and a real layer-norm; we use one fixed attractor and one fixed rate so the drift is legible and every number is reproducible. Layer-norm (skipped in the toy arithmetic) would rescale each step's numbers back to a standard range so the running total never blows up or vanishes between blocks — the piece that keeps a 100-layer stack numerically stable.

MythEach layer is a different kind of operation — deeper means fancier, more exotic math.

RealityIt's the same block — attention, FFN, two add-backs, a layer-norm — repeated with different weights. Depth just refines the same running vector over and over; the machinery of layer 1 and layer 90 is identical in shape.

Takeaway

One transformer block bundles the cross-token step (attention) and the per-token step (FFN), each added back onto the residual stream, with a layer-norm to stay tidy. Stack that block N times and you have a model; the vector drifts toward its contextual meaning block by block.

What is one transformer block?

Why: a block bundles the cross-token step (attention) and the per-token step (FFN), each added back onto the residual stream, with a layer-norm to keep the numbers in range. Stacking that same block is what makes a model deep.