Click a query token and watch the whole pipeline run on real numbers: dot products become raw scores, softmax turns them into weights that sum to 1, and the Values blend into a brand-new vector. Nothing here is looked up — every number falls out of arithmetic you can trace.
Last page we built three vectors from every token's embedding: a Query (what it's looking for), a Key (what it advertises), and a Value (the content it hands over). Now they get used.
To decide how much token A should listen to token B, take A's Query and B's Key and dot them together: q·k. A bigger dot product means the two vectors point the same way — a better match, a higher raw score. Those raw scores are the logits of the attention softmax. (These are attention-logits — one score per token in the sentence. Later, next-token prediction produces vocab-logits — one score per word in the whole vocabulary. Same softmax, different lists; lesson 10 builds the second.)
Run all the scores through softmax and you get positive fractions that add up to 1 — the attention weights. This is the exact same softmax that turns next-token logits into probabilities — the one you'll dial with temperature on lesson 11. Finally, A's new vector is the weighted blend of every token's Value: a token literally becomes a mix of what it paid attention to.
q·k → softmax → blend for this tokene^z / Σe^z fractions the sampling page (lesson 11) reshapes with temperature — here they weight attention instead of picking a word.TRY → Click a different token (try The or tired) — all four stages recompute from scratch. Hit Run to animate the pipeline one stage at a time, or Step to walk it yourself. Watch the cat arc: for it it's always the fattest outward one.
e_it = [0.8, 0.1, 0.2, 0.3]. Its Query is q = Wq·e_it = [2·0.8+1·0.3, 2·0.1, 1·0.2, 1·0.3] = [1.9, 0.2, 0.2, 0.3].
The seven raw scores are q·kⱼ for each token's Key (Keys have their function dim damped by ×0.3):
| token | q · k written out | score | e^score | weight |
|---|---|---|---|---|
| cat | 1.9·1.0 + 0.2·0.2 + 0.2·0.1 + 0.3·0 | 1.96 | 7.10 | 35.8% |
| it | 1.9·0.8 + 0.2·0.1 + 0.2·0.2 + 0.3·0.09 | 1.61 | 4.99 | 25.2% |
| tired | 1.9·0.3 + 0.2·0.1 + 0.2·1.0 + 0.3·0 | 0.79 | 2.20 | 11.1% |
| sat | 1.9·0.2 + 0.2·1.0 + 0.2·0.1 + 0.3·0 | 0.60 | 1.82 | 9.2% |
| was | 1.9·0.1 + 0.2·0.5 + 0.2·0.4 + 0.3·0.06 | 0.39 | 1.47 | 7.4% |
| because | 1.9·0 + 0.2·0.1 + 0.2·0.2 + 0.3·0.27 | 0.14 | 1.15 | 5.8% |
| The | 1.9·0 + 0.2·0 + 0.2·0 + 0.3·0.3 | 0.09 | 1.09 | 5.5% |
The e^score column sums to 19.83; each weight is that row's e^score divided by 19.83 (e.g. 7.10 / 19.83 = 0.358 = 35.8%). The seven weights add to 1.00.
Blending the Values (each token's v = Wv·e, dim 3 halved) with those weights gives out ≈ [0.62, 0.24, 0.25, 0.10] — animacy-dominant (from cat) with a state bump (from tired).
The √d note: real transformers divide each score by √(key-dimension) before softmax to keep the numbers from getting too large — with d = 4 that's ÷2. It changes the sharpness of the weights but not which token wins. We skip it here so the arithmetic stays clean.
Attention is three ops in a row: Q·K dot products → softmax → weighted blend of Values. The output vector for a token is literally a mixture of the tokens it scored highest.
it to cat means…