Attention has no built-in sense of order. Position is stitched in by literal, element-wise addition — a small position vector summed onto the embedding, in its own directions, so a word keeps its meaning coordinates exactly and only picks up a position tag in the spare slots.
Back on lesson 1 the embedding lookup handed each token a meaning vector and promised that its position would be folded in later. This is later. On its own, attention is order-blind — it would score "dog bites man" and "man bites dog" identically, because it only ever sees an unordered bag of vectors. Something has to tell it where each token sits.
That something is a position vector POS[pos], one per slot in the sentence, arithmetically summed onto the embedding: vector = meaning + position, element by element. Not a metaphor, not a lookup key — real addition, the same + you'd do by hand.
Heads-up: this page uses its own tiny demo words (dog · bites · man) and a simplified convention — meaning in dims 0–1, position in dims 2–3 — chosen to make the "own directions" point visible. It's a separate demo from the cat / sat / it sentence table, whose four dims meant something else.
"Doesn't that clobber the meaning?" No. The position vectors live in their own directions — spare dimensions the meaning barely uses (here, dims 2 and 3). So a word keeps its meaning coordinates exactly and only gains a position tag in the leftover slots. The model learns to read meaning from the meaning directions and order from the position directions — two channels, one added vector.
meaning only
TRY → Flip position ON and watch only the two amber cells move — the cyan meaning cells never budge. Then switch the word order: with position ON the slots genuinely differ ("dog" vs "man" in slot 0); with position OFF, flipping the order just shuffles the same three vectors and nothing can tell subject from object.
Meaning lives in dims 0–1 (animacy, action); position lives in dims 2–3 (the spare directions). Each slot is meaning + POS[pos], added element-wise:
| ordering | slot | dim0 anim | dim1 act | dim2 pos | dim3 pos |
|---|---|---|---|---|---|
| dog bites man | 0 · dog | 1.0 | 0.2 | 0.3 | 0.1 |
| 1 · bites | 0.1 | 1.0 | 0.0 | 0.4 | |
| 2 · man | 0.9 | 0.1 | −0.3 | 0.7 | |
| man bites dog | 0 · man | 0.9 | 0.1 | 0.3 | 0.1 |
| 1 · bites | 0.1 | 1.0 | 0.0 | 0.4 | |
| 2 · dog | 1.0 | 0.2 | −0.3 | 0.7 |
Meaning dims identical, only position dims change. dog's meaning dims are 1.0, 0.2 in both orderings and at both positions it lands in — adding POS never touched them. The entire change is in dims 2–3, the position channel. That is precisely why position doesn't clobber meaning: it's written into directions the meaning leaves free.
Orderings diverge — with position. Slot 0 is dog-flavored (animacy 1.0) under "dog bites man" but man-flavored (animacy 0.9) under "man bites dog"; slot 2 flips the other way. The two sequences of vectors are now different, so a model can tell subject from object. Without position, the two orderings are the same multiset of vectors — {dog, bites, man} either way — and order is invisible.
A token's input is meaning + position, literally summed. Because position lives in spare directions, it tags where a word sits without disturbing what it means — which is how a transformer knows "dog bites man" ≠ "man bites dog".