One attention computation is a head. A real layer runs several at once — same tokens, different Wq/Wk/Wv each — so the same word gets routed several different ways at the same time.
On the last page you ran one full attention pipeline: a token's Query dotted against every Key, softmax'd into weights, then used to blend the Values. That whole computation is one head.
A real attention layer runs many heads in parallel. Each has its own Wq/Wk/Wv, so each builds a different Query and different Keys — and different tokens win the dot-product race. One head learns to track reference (it → cat), another tracks state (it → tired), another tracks action (it → sat). Their outputs are combined, so each token ends up enriched from several angles at once.
Below, each active head runs the exact same page-6 pipeline — no answer is stored. Toggle heads on and off and watch the fattest arc from it jump to a different word.
TRY → With it selected, the violet head points hardest at cat. Flip on Head 2 (cyan) — a second fan appears and its fattest arc lands on tired. Flip on Head 3 (amber) — its fattest arc lands on sat. Same token, three routings, all computed live.
Same token it, same embedding [0.8, 0.1, 0.2, 0.3]. Only the matrices differ, so the Query differs, so a different Key wins.
Head 1 (reference) emphasizes animacy. q_it = Wq·e = [1.9, 0.2, 0.2, 0.3]. Dotting against every Key, cat scores highest (1.96) → softmax → cat ≈ 36%, the fattest violet arc.
Head 2 (state) multiplies the state dimension by 3. q2_it = Wq2·e = [0.24, 0.1, 0.6, 0.09]. Now the winning dot product is with tired:
q2_it · k2_tired = 0.24·0.09 + 0.1·0.1 + 0.6·3.0 + 0.09·0 = 1.83 — the biggest score in this head → softmax → tired ≈ 41%. The 0.6·3.0 term is the whole story: tired is the only token with a large state value, and this head cranks the state axis up.
Head 3 (action) multiplies the action dimension by 3 instead. q3_it = [0.24, 0.3, 0.06, 0.09], and now sat (the one high-action token) wins: q3_it · k3_sat = 0.24·0.06 + 0.3·3.0 + 0.06·0.03 = 0.916 → softmax top is sat. Its lead is gentler than the other heads, so sat's share sits around 25% — still the winner, just a softer fan.
Nothing here is looked up: each head calls attend(qi, Wq_h, Wk_h), which recomputes q, all seven q·k logits, and the softmax from scratch. Change a matrix and the winner can change.
Simplification: real heads each have their own Wv too; ours share one, because this page only shows the Q·K routing (which token wins), not the per-head Value blend.
A head is one Q·K·V attention computation. A layer runs many, each with different matrices, so the same token routes to different words — reference, state, action — all at the same time.