We finally have a final vector for the predicting position. To turn it into a word, score it against every token in the vocabulary with the very same dot product from lesson 2 — one score per word — then softmax. That's the whole trick behind "next token: cat, 61%."
After attention and the feed-forward step, the last position of "The cat sat because it was ___" holds one 4-number vector. It doesn't spell a word — it's a point in meaning-space. So we ask a simple question: which vocabulary token does this vector point at most?
One switch to notice: the last few pages followed it to learn the moves (residual, then FFN → its running vector [2.13, 0.54, 0.36, 0.61]). But the next word is predicted from the last token, was — every position carries its own running vector; this is just was's, after the same attention + FFN. That's the one we turn into a word.
Score it against every token by dotting it with that token's embedding row. Reuse the embedding table E backwards: one dot product per word gives one raw score — a vocab-logit — and softmax turns the whole list into next-token probabilities. (Real models often reuse the very same E, transposed — "tied weights.") This is the exact a·b from lesson 2, now scoring the output vector against the whole dictionary instead of scoring two words against each other.
Two different logit lists — name the difference now. Lessons 6–7 made attention-logits: one score per token in the context (~7 here). This page makes vocab-logits: one score per word in the whole vocabulary (7 here, ~50,000 in a real model). Same softmax formula, two completely different lists — never call one "the logits" without saying which.
was, after attention + FFN) — a point in meaning-space, not yet a word:TRY → Drag state down toward 0 and watch tired lose the lead to cat — the vector stops pointing at the state axis tired lives on. Every logit is final_vector · E[word], recomputed on each move — nothing here is replayed.
Each vocab-logit is the final vector dotted against one embedding row (logit(t) = dot(UNEMBED_VEC, E[t])), then softmax over all seven. Sum of e^(z − 1.70) is 3.305:
| token | final · E[token], written out | logit | e^(z−1.70) | softmax |
|---|---|---|---|---|
| tired | 0.9·0.3 + 0.3·0.1 + 1.4·1.0 + 0.1·0 | 1.70 | 1.000 | 30.3% |
| cat | 0.9·1.0 + 0.3·0.2 + 1.4·0.1 + 0.1·0 | 1.10 | 0.549 | 16.6% |
| it | 0.9·0.8 + 0.3·0.1 + 1.4·0.2 + 0.1·0.3 | 1.06 | 0.527 | 16.0% |
| was | 0.9·0.1 + 0.3·0.5 + 1.4·0.4 + 0.1·0.2 | 0.82 | 0.415 | 12.6% |
| sat | 0.9·0.2 + 0.3·1.0 + 1.4·0.1 + 0.1·0 | 0.62 | 0.340 | 10.3% |
| because | 0.9·0 + 0.3·0.1 + 1.4·0.2 + 0.1·0.9 | 0.40 | 0.273 | 8.2% |
| The | 0.9·0 + 0.3·0 + 1.4·0 + 0.1·1.0 | 0.10 | 0.202 | 6.1% |
Predicted next token = tired at 30.3% (Σ = 100.0%). So "cat 61%" in any explainer is exactly this: the final vector dotted against every embedding row, softmaxed — nothing more exotic. The runners-up cat and it are the sentence's subject, which is a sensible next-guess for "…it was ___".
Attention-logits vs vocab-logits. Lesson 6's seven scores were one-per-token in the context — q·k, softmaxed into attention weights (how much to listen to each token). These seven are one-per-word in the vocabulary — the final vector dotted against every row of E, softmaxed into next-token probabilities. Same softmax formula, different lists. Here both lists happen to be length 7; in a real model the first is a few thousand and the second is ~50,000.
To turn a vector into a word: dot it against every row of the embedding table → one vocab-logit per word → softmax → next-token probabilities. That's where "next token: cat, 61%" comes from.
cat, 61%" come from?