ChatGPT, Claude, Gemini — under the hood they're all the same surprisingly simple idea, stacked up billions of times. This is a scroll-through, click-around tour that builds the whole picture from scratch: tokens, embeddings, attention, the transformer, and the different shapes these models come in. One animation at a time.
Strip away all the hype and a language model does one humble thing: given some text, it guesses what comes next. Not one guess — a ranked list of possibilities, each with a probability. That's the entire engine. Everything else in this course is machinery that makes that one guess shockingly good.
HOW IT WRITES → To produce a whole sentence, it picks one word, sticks it on the end, and asks the very same question again — "now what comes next?" — over and over. A reply is just this guess, looped hundreds of times.
A model can't read letters or words the way you do — it works with numbers. So step one is a tokenizer that splits text into small chunks called tokens (a short word, or a piece of a longer one) and swaps each for an ID number. Common words become one token; rarer words get broken into parts.
"The cat sat on the mat, contentedly."
NOTICE → "contentedly" splits into pieces while "cat" stays whole. From here on, every length, speed and limit you hear about a model is counted in tokens, not words.
An ID number like 2543 carries no meaning. So the model turns each token into a long list of numbers — a vector, or embedding — that places it as a point in a vast space. The trick: words with similar meanings land close together, and directions in that space capture relationships.
Time to open the box. A neural network is just numbers flowing through a web of connections. Values enter on the left; along every connection they're multiplied by a weight; at each node they're summed, nudged by a bias, and gently squished into a tidy range; then handed to the next layer. Do that a few times and an answer pops out on the right. Here's a tiny one you can actually run.
Change an input and every node downstream recomputes. Line thickness = weight size; rose = positive, grey = negative.
LEARNING → Nobody sets these weights by hand. Training shows the network millions of examples and nudges every weight a hair whenever the output is wrong — until the whole web produces good answers. The two outputs here stand in for a real model's next-token scores (which has one output per possible token — tens of thousands of them).
Same idea at every scale — just more knobs. The knobs are the model; they're what gets shipped as "the weights."
In a moment, attention will let every word look at all the others at once — which means, on its own, it has no sense of order. But "dog bites man" and "man bites dog" are very different! So first, each token's embedding gets a position tag added in, telling the model where in the line it sits.
THE FIX → Each token embedding = what the word is + where it sits. With positions added, attention can tell subject from object, first from last, near from far.
Here's the breakthrough that made modern models work. To understand a word, the model lets it look at every other word in the sentence and pull in what's relevant. The word "it" figures out what it refers to by paying attention to the right earlier word.
Click it to see what it's really pointing at.
The current word asks a question: "what here matters to me?"
Every other word offers a label — "here's what I'm about."
Words that match get their meaning pulled in, weighted by the match.
MANY AT ONCE → A real model runs dozens of these attention "lenses" (called heads) in parallel — one might track grammar, another who-did-what, another tone — then blends their findings.
Attention is one ingredient. Wrap it with a small "thinking" step (a feed-forward network), add two safety rails that keep the numbers stable (residual shortcuts and layer-norm), and you have one transformer block. Then stack that same block dozens of times. Each layer refines the meaning a little more.
Early layers catch grammar and simple patterns; deeper layers capture abstract meaning, logic and style. Big models stack many dozens.
Back to that ranked list of next tokens. How does the model choose? A dial called temperature reshapes the odds. Low temperature makes it play safe and pick the top choice almost every time; high temperature flattens the odds so it takes more risks — more creative, but more likely to wander.
TWO PHASES → Reading your prompt (all tokens at once) is fast; writing the reply (one token, then the next, then the next) is the slow part. That's why long answers stream out gradually.
A model isn't programmed with facts — it's grown in stages. First it reads a huge slice of the internet to learn language itself. Then it's taught to follow instructions. Finally, it's polished using human preferences so its answers are helpful and safe.
Predict the next token across trillions of words of text. This is where raw knowledge of language, facts and patterns is soaked up. Slow, enormous, expensive.
the whole internet · months of computeShow it thousands of example question → good answer pairs so it learns to respond to requests, not just autocomplete text.
curated examples · learns to be usefulHumans (and other models) rank answers; the model is nudged toward the ones people prefer. This is the RLHF step that shapes tone, honesty and safety.
human feedback · alignmentTRAIN vs RUN → All of that is training — done once, ahead of time, to set the weights. When you chat, the weights are frozen: that's inference, just running text through the finished machine.
The same building block gets wired up in three classic ways, depending on the job. The difference comes down to who's allowed to look at whom — the attention pattern. Pick one below to watch data flow through it, and see the attention grid where a lit cell means "this token can look at that one."
Progress has largely come from turning up three dials together. More parameters, more training data, and a longer context window — how much text the model can hold in mind at once when it answers you.
CONTEXT WINDOW → This is why a model can "forget" the start of a very long chat: everything it's considering — your prompt, the documents, its own reply so far — has to fit inside that token budget.
You've now seen each piece move. Here's the quick glossary to keep — the jargon, decoded.