A dense model runs every weight for every token. An MoE keeps N separate expert FFNs and a tiny router that sends each token to only its top-k — so the model stores a huge parameter count but only computes a small slice of it per token.
Back in Part-1 lesson 9 every token ran through one feed-forward network (FFN). A mixture-of-experts replaces that single FFN with N separate expert FFNs plus a small router — a gate that scores the token's vector against each expert (a dot product then softmax, the lesson-6 move again) and keeps the top-k.
Only those k experts run — sparse activation. So you store N experts (total params grow with N) but only compute k of them per token. FLOPs, and therefore the roofline of Part 2, track the active params, not the total.
With 8 experts and top-2, a model can hold 47B parameters yet route each token through only ~13.6B of them — the memory of a big model at the speed of a small one. Below, pick a token and watch a different pair of experts fire, computed live from the same matvec and softmax you already know.
TRY → With it selected (k=2), the router fires e0 · animacy and e5 · animate+state — 13.6B active of 47.4B stored, ~29%. Pick cat and a different pair lights up (e0 + e4); pick tired and it jumps to e2 + e5. Slide k to 1 or 3 and watch the active-param bar move — total never changes.
The router. The gate is a tiny 8×4 matrix; routing is logits = matvec(gate, E[token]), then softmax over the top-k. For cat = [1.0, 0.2, 0.1, 0]:
logits = [3.0, 0.6, 0.3, 0.0, 1.8, 1.65, 0.65, −1.3] → top-2 are e0 (animacy, 3.0) and e4 (animate+action, 1.8) → gate softmax([3.0, 1.8]) = [0.769, 0.231] → e0 76.9%, e4 23.1%. Different tokens rank the experts differently — sat→e1+e4, tired→e2+e5, it→e0+e5 — so four tokens fire four different pairs. Sparse routing is real, not staged.
The parameter arithmetic (8 experts, top-2, Mixtral-8×7B-shaped: d_model 4096, d_ff 14336, 32 layers, vocab 32000):
| piece | formula | params | when |
|---|---|---|---|
| embed / unembed | 32000 · 4096 | 131.1M | always |
| attention (per layer) | 4 · 4096² | 67.1M | always |
| router (per layer) | 8 · 4096 | 0.03M | always |
| one expert FFN | 3 · 4096 · 14336 | 176.2M | top-k only |
total = 131M + 32·(67.1M + 0.03M + 8·176.2M) = 47.38B
active = 131M + 32·(67.1M + 0.03M + 2·176.2M) = 13.55B
→ activeFrac = 13.55 / 47.38 = 0.286 — only ~29% of the parameters run per token. (The raw expert fraction is k/N = 2/8 = 25%; the overall figure is a bit higher because attention and embeddings are always on.)
The router is a learned matrix — it decides routing, and a load-balancing loss during training keeps tokens spread so every expert earns its keep. FLOPs/token ≈ 2·active params (Part-2 lesson 8), so the roofline sees a 13.6B model even though 47.4B sit in VRAM.
A router sends each token to its top-k of N expert FFNs. You store all N (huge total params) but only run k per token (small active params, small FLOPs) — a 47B model that computes like a 13.6B one.