Llama 4's Mixture-of-Experts Routing and What It Changes About Output

Llama 4 is the first Llama family built on a mixture of experts, and it is why the headline parameter count stopped predicting anything useful. A model quoted at 400B parameters that runs 17B of them per token is two different models depending on which question you are asking.

The published numbers

Meta announced Llama 4 in April 2025. The figures below are the ones stated in that announcement and the accompanying model cards:

model              active params   experts   total params   context (documented)
-----------------  --------------  --------  -------------  --------------------
Llama 4 Scout      17B             16        109B           10M tokens
Llama 4 Maverick   17B             128       400B           1M tokens

Both models share an active parameter count of 17B and differ in how many experts they choose from. Meta also previewed a much larger teacher model, Behemoth, with a far higher active count; it was described as still in training at announcement, so treat any figure for it as provisional.

Two other things changed in the same release, and they are worth separating from the sparsity because they are frequently bundled together in summaries. Llama 4 is natively multimodal — image understanding is trained in from the start rather than attached to a text model afterwards, as it was for the Llama 3.2 vision models. And the context windows moved by orders of magnitude rather than incrementally. Neither of those follows from the mixture of experts; they are independent decisions that arrived at the same time.

What “active parameters” means

In a dense transformer, every parameter takes part in every forward pass. A mixture-of-experts layer replaces the single feed-forward block with many parallel copies — the experts — plus a small router that scores them per token and selects a few. Everything else in the layer, attention included, still runs for every token.

So “17B active” is the sum of: the shared parts that always run, plus the parameters of only the experts selected for that token. Maverick’s 17B out of 400B is:

17B / 400B = 4.25% of the network per token

The selection is per token and per layer, not per request. Token 1 of your prompt may route to a different set of experts than token 2, and both may differ from what token 3 uses at the next layer. There is no persistent “expert for coding” that gets attached to your conversation — that reading of the name is the most common misunderstanding of the architecture. Meta’s design for Maverick also includes a shared expert that runs for every token alongside the routed ones, and alternates dense and MoE blocks rather than making every layer sparse.

The general mechanism, independent of Llama, is covered in the mixture-of-experts page.

Sparse in compute, dense in memory

The trade at the centre of the architecture: arithmetic per token follows the active count, but memory follows the total count. Every expert must be resident, because the router might pick any of them for the next token, and there is no way to know in advance.

Maverick weights, bf16:  400B params x 2 bytes  ≈ 800 GB
Compute per token:       as though it were a 17B model

That is the whole commercial logic. You pay 17B-model latency and 17B-model cost per token, on hardware sized for a 400B model. It explains two things that otherwise look inconsistent: why a very large Llama 4 model can be priced near a much smaller dense one, and why far fewer providers host it than host an 8B — the price follows the small number and the hosting requirement follows the large one.

Meta also released quantised builds, and the community produced more, precisely because the memory figure is the binding constraint rather than the compute one.

What it changes about output

Three consequences you can observe, and one you cannot.

• Reproducibility gets harder, not impossible. Routing is deterministic given the same inputs — but on a batched server, a token’s expert assignment can be affected by capacity limits, which depend on what else is in the batch. Two identical requests at temperature 0 can therefore diverge under different load, which is a failure mode dense models do not have. See determinism with open weights.
• Latency is less uniform. Batching is harder when different tokens in a batch need different experts, so per-request latency varies more under load than for a dense model of the same active size.
• Quality does not follow either number cleanly. The active count predicts cost well. It predicts capability only loosely, because the total count buys capacity that the routing can exploit. Two models with the same 17B active count and 16 versus 128 experts are not the same model.
• What it does not change: the interface. Prompt format, tokenizer, stop tokens and streaming shape are architectural details you never see. Nothing about your request changes because the feed-forward block became sparse.

How the router picks

The router is a small learned layer — in the simplest formulation, a linear projection from the token’s hidden state to one score per expert, followed by a top-k selection and a softmax over the chosen scores to weight their outputs. It is tiny relative to the experts it selects, which is what makes the whole scheme worth doing: choosing costs almost nothing and skipping costs a great deal.

Three properties of that arrangement have consequences you can feel:

• Routing is learned, not designed. Nobody assigns experts to topics. Which expert handles what emerges from training, and the resulting specialisations generally do not correspond to anything a human would name. Explanations that describe “a maths expert” and “a code expert” are a useful fiction and not what the model contains.
• Balance has to be trained for. Left alone, routers collapse onto a few popular experts and the rest of the capacity is wasted, so MoE training adds an auxiliary objective encouraging even use. This is why the number of experts is a design choice with a cost rather than a free dial.
• Serving introduces capacity limits. An implementation must decide in advance how many tokens each expert can take in a batch. If more tokens route to one expert than its capacity allows, the excess is handled differently — dropped through the layer’s residual path, or rerouted — and which tokens are affected depends on the rest of the batch. That is the concrete mechanism behind the reproducibility point above.

It also explains a deployment characteristic: MoE models are usually served with expert parallelism, spreading experts across devices, so each token’s route implies communication between them. Sparse models are therefore more sensitive to interconnect quality than dense models of the same active size, which is another reason the hosting landscape for them is narrower.

Reading a Llama 4 spec sheet

Three habits keep the two numbers from misleading you:

1. Match the number to the question. Cost and latency: active. Whether you can host it, and on what: total. Never compare a sparse model’s total against a dense model’s parameter count — they are not the same kind of number.
2. Treat the documented context as an upper bound. Ask the endpoint what it serves rather than reading the model card, for the reason in the note above.
3. Benchmark on your own task before believing a ranking. Neither parameter count is a quality score, and a sparse model that wins on one workload can lose on another with no change in either figure.