Attention just handed it a fresh 4-number blend. The model does not throw away it's old vector and swap the blend in — it adds the blend back on. The vector becomes a running total that every layer nudges, never overwrites.
Last page left a loose end: attention produced a blended output vector for it — [0.62, 0.24, 0.25, 0.10] — and then… what? A tempting guess is that this blend becomes it's new vector, replacing the old one. It doesn't.
Instead the model does new = old + Δattn, element by element: it keeps everything it already was and adds the refinement on top. That running total has a name — the residual stream — and it's the same vector that flows through the whole network, getting nudged layer after layer.
Why add instead of replace? Replacing would erase the token's own identity every single layer. Adding lets each layer contribute without destroying what came before — the original meaning is always still in there. Flip the toggle below and watch the difference.
TRY → Toggle Add back ⟷ Replace. In Add mode each dimension shows its sum written out (0.8 + 0.62 = 1.42) and it's original 0.8 animacy survives. Flip to Replace and watch the old column strike through — it's 0.8 animacy and 0.3 function vanish. The token forgot itself.
The blend Δattn is recomputed live — it's attend(indexOf('it'), Wq, Wk).out, the exact lesson-6 pipeline (q·k → softmax → weighted blend of Values), not a stored constant.
| dim | old e_it | Δattn | add: old + Δ | replace: Δ only |
|---|---|---|---|---|
| animacy | 0.80 | 0.62 | 1.42 | 0.62 |
| action | 0.10 | 0.24 | 0.34 | 0.24 |
| state | 0.20 | 0.25 | 0.45 | 0.25 |
| function | 0.30 | 0.10 | 0.40 | 0.10 |
e_it = [0.8, 0.1, 0.2, 0.3]; Δattn = [0.62, 0.24, 0.25, 0.10]; residual new = e_it + Δattn = [1.42, 0.34, 0.45, 0.40]. In Replace mode you'd instead get [0.62, 0.24, 0.25, 0.10] — it loses its own animacy 0.8 and function 0.3 entirely.
Every layer does exactly this add. The vector is a running total, and the original embedding is never overwritten — which is a big part of why very deep models stay trainable (gradients have a clean path straight down the sum). This same [1.42, 0.34, 0.45, 0.40] is the input to the FFN on the next page.
new = old + Δattn. The token keeps its identity and gains context; that ever-growing running total is the residual stream.Attention's output is added back onto the token's own vector — refine, don't replace. The vector is a running total the whole network keeps nudging, so the original meaning is always still in there.
it's vector becomes…