Part 1 · Lesson 9/15

The feed-forward step: each token thinks alone

Attention was the step where tokens looked at each other. The FFN is the opposite: each token's vector runs through a little neural network on its own — it's lesson 4's neuron, finally eating a real post-attention vector.

⏱ ~5 min 🎚 the net, computed live math: some

Attention mixed information across tokens: it's vector became a blend of cat, tired, and the rest. The feed-forward network does the opposite job — it transforms each token's vector by itself, with no other token involved.

That little network is just lesson 4's neuron, stacked into layers: a hidden layer of neurons hidden = tanh(W1·v + b1), then an output layer out = W2·hidden + b2. Same multiply-add-squish — now chewing on the real residual vector [1.42, 0.34, 0.45, 0.40] that it carried out of lesson 8.

This is where the model transforms what attention gathered — sharpening some features, trimming noise — one token at a time. And its output is added back too (residual, lesson 8): v + FFN(v).

The FFN · 4 → 8 → 4, computed live

hidden = tanh( W1·v + b1 ) → FFN out = W2·hidden + b2 → residual: v + FFN out

Input vector — it's post-attention residual (lesson 8)

v — the token vector (cyan)

Nudge a slider and the entire network below recomputes — every hidden and output number moves. Reset to return to the documented [1.42, 0.34, 0.45, 0.40].

fixed learned weights (rose)

W1 is 8 rows × 4 (one hidden neuron per row) with biases b1; W2 is 4 rows × 8 (one output neuron per row), biases b2 = 0. Sparse on purpose so each worked sum is short and readable.

Hidden layer — 8 neurons, each tanh(Σ v·w + b)

Output layer — 4 linear neuronsno activation on output

FFN output — Δffn, then added back onto v

TRY → Watch hidden unit h0 ("animate?") — it fires hardest at 0.73 because it's animacy is high. Drag the animacy slider down and h0's tanh (and the animacy output) drop with it. Every output neuron is linear — no squish — so its number can climb past ±1 unlike the hidden units.

Show me the actual numbers (input = [1.42, 0.34, 0.45, 0.40])

Pre-activations Σ v·w + b for the 8 hidden units:

unit	worked	pre	tanh → h
h0 animate?	1.42·1 − 0.5	0.92	0.73
h1 action?	0.34·1 − 0.3	0.04	0.04
h2 state?	0.45·1 − 0.3	0.15	0.15
h3 function?	0.40·1 − 0.3	0.10	0.10
h4 entity, not verb	1.42·1 + 0.34·(−0.5) − 0.4	0.85	0.69
h5 subject-doing	1.42·0.5 + 0.34·0.5 − 0.5	0.38	0.36
h6 animate + state	1.42·0.5 + 0.45·0.5 − 0.5	0.435	0.41
h7 strong-state only	0.45·1 − 0.8	−0.35	−0.34

So h ≈ [0.73, 0.04, 0.15, 0.10, 0.69, 0.36, 0.41, −0.34] — h0 fires hardest.

Output neurons (linear, no tanh): out0 = 0.6·0.73 + 0.4·0.69 = 0.71, out1 = 0.5·0.04 + 0.5·0.36 = 0.20, out2 = 0.5·0.15 + 0.5·(−0.34) = −0.09, out3 = 0.5·0.10 + 0.4·0.41 = 0.21. So Δffn ≈ [0.71, 0.20, −0.09, 0.21].

Added back (residual, lesson 8): the token's running vector becomes [1.42+0.71, 0.34+0.20, 0.45−0.09, 0.40+0.21] = [2.13, 0.54, 0.36, 0.61]. The FFN pushed animacy up further (more clearly cat-flavored) and trimmed the stray state it had picked up from tired.

Every token in the sentence runs through this same FFN, independently. Attention shared information between tokens; the FFN never lets them talk. Change the input vector above and only that token's numbers move — the network has no memory of the others.

MythAfter attention there's more attention — it's attention all the way down.

RealityHalf of every block is the FFN: each token pushed through the same little neuron network alone. Attention mixes across tokens; the FFN transforms each token by itself.

Takeaway

The FFN is lesson 4's neuron, stacked — tanh(W1·v + b1) then a linear W2·h + b2 — run on one token's vector at a time, then added back onto the residual stream.

How is the FFN different from attention?

Why: attention is the cross-token step (Q·K over all tokens); the FFN runs each token's vector through the lesson-4 neuron independently — every token gets the same network, alone.