Softmax hands you one honest distribution over the next word. Before picking, you can reshape it with three dials — temperature, top-k, and top-p — without ever touching the logits underneath. Every bar below is recomputed live from the same eight scores. (Last page produced vocab-logits over the 7 sentence tokens; here we swap to a richer 8-candidate set for "…it was ___" so the dials have something visible to reshape — same idea, a distribution over next words.)
Last page turned the final vector into vocab-logits and softmaxed them into probabilities. That's one fixed distribution — but it's not the end of the story. Between "here are the probabilities" and "here's the word," there's a reshaping step, and it's where creativity versus focus gets dialed.
Temperature divides every logit by T before softmax: below 1 sharpens toward the leader (safer, more deterministic), above 1 flattens toward the field (riskier, more surprising). Top-k keeps only the k best candidates and zeroes the rest. Top-p keeps the smallest set whose probabilities add up to at least p.
The division of labor: top-k and top-p decide which tokens are eligible; temperature decides how peaky the choice among them is. Same softmax underneath — the model's scores never move.
p = softmax(logits / T) → keep top-k or the top-p nucleus → renormalize survivors
TRY → Pull temperature to 0.5 and watch tired shoot toward 79%; push it to 2.0 and the field flattens. Set top-k = 3 — five bars go dark and the survivors regrow. Now set top-p = 0.9 instead: it keeps four tokens, not three, because it counts probability mass, not candidates.
The eight raw logits (fixed): tired 3.2 · sleepy 2.4 · hungry 1.6 · resting 1.1 · scared 0.7 · purring 0.3 · gone -0.2 · blue -1.0. At T = 1, softmax gives:
| token | logit z | p (T=1) | cumulative |
|---|---|---|---|
| tired | 3.2 | 51.0% | 0.510 |
| sleepy | 2.4 | 22.9% | 0.740 |
| hungry | 1.6 | 10.3% | 0.843 |
| resting | 1.1 | 6.3% | 0.906 |
| scared | 0.7 | 4.2% | 0.947 |
| purring | 0.3 | 2.8% | 0.975 |
| gone | -0.2 | 1.7% | 0.992 |
| blue | -1.0 | 0.8% | 1.000 |
top-k = 3 keeps {tired, sleepy, hungry} and renormalizes them to 60.6% / 27.2% / 12.2% — a hard cap on the count of survivors, always exactly three.
top-p = 0.9 walks the cumulative column: 0.510 → 0.740 → 0.843 → 0.906. It stops the moment the running sum crosses 0.9 — which happens after adding resting. So it keeps {tired, sleepy, hungry, resting} — one more token than k=3. That extra token is the whole point: top-p caps probability mass, so the number of candidates floats with how confident the model is. Here the model isn't confident enough for three tokens to cover 90%, so a fourth is pulled in.
Temperature reshapes without cutting: T = 0.5 sharpens tired to ~79%; T = 2 flattens it to ~30%. Every one of these — temperature, top-k, top-p — changes the distribution you sample from. None of them changes the logits. The model's scores are fixed the instant the forward pass ends; the dials only reshape how you draw a word from them. (Illustrative candidates — the mechanism is the real thing.)
Three dials, one distribution: temperature sets how peaky, top-k caps the candidate count, top-p caps the probability mass. The logits never move — you only reshape how you sample from them.
top-p = 0.9. What does that do?