The transformer only ever ate a list of embedding vectors. Cut an image into patches, project each into a vector the same size as a text embedding, and drop it in the same stream — attention can't tell them apart.
Back in lesson 1, a token became an embedding vector — a short list of numbers — and everything after that (attention, the whole stream) only ever saw vectors. Nothing about that list is text-specific. So to feed an image in, you just need to turn the image into embedding vectors too.
Cut the image into fixed patches — a grid of small squares. Flatten each patch's pixels into a row and project them through one weight matrix — a single matvec, exactly like anywhere else — into a vector the same size as a text embedding. Then add a position encoding for where the patch sits.
Now append those patch-vectors to the token stream. Attention blends image-patches and text-tokens with the same math. Audio becomes frames the same way. Tokens are whatever you can turn into a vector.
Attention (lessons 5–7) blends this whole row identically — after projection there is no "image" and no "text", only vectors.
TRY → Click a patch to project it, or click any pixel to brighten it (it wraps 0.1 → 0.9 → 0.1). Watch the flattened row, the matvec, and the 4-number embedding recompute — then toggle add position to nudge the last dim. The two bright diagonal patches light up brightness + diagonal; the dark ones stay near zero.
Our toy: a 4×4 image, cut into 2×2-pixel patches → 4 patches, each flattened row-major. Patch (0,0) has pixels [0.9, 0.1, 0.1, 0.9] (a bright diagonal), so W_patch · pixels = [0.25·2.0, 0, 0, 0.8] = [0.5, 0, 0, 0.8] — it fires brightness and diagonal. The off-diagonal patches are all 0.1: [0.1, 0.1, 0.1, 0.1] → [0.1, 0, 0, 0], dark and featureless. Each 4-number vector then gets PATCH_POS[i] added (here just a tiny bump on the last dim) and joins the stream — the same shape as a text embedding like cat = [1.0, 0.2, 0.1, 0.0].
Real anchor: a Vision Transformer cuts a 224×224 image into 16×16 patches → (224/16)² = 14² = 196 patches → 196 image-tokens, each projected up to d_model (e.g. 4096) and position-encoded, then concatenated with the text tokens.
The projection matrix W_patch is learned, just like the embedding table from lesson 1 — it's the image's tokenizer. After that step there is no "image" and no "text", only vectors, and the attention you built in Part 1 runs unchanged.
An image becomes tokens: cut into patches, each projected into an embedding in the same space as text and fed into the same stream. Tokens aren't only text — they're whatever you can turn into a vector.