Mixture of Experts: Why a 400B Model Can Cost Like a 40B One

A dense model uses all of itself for every token. A mixture-of-experts model uses a fraction. That single change is why headline parameter counts stopped predicting price.

The idea

In a dense transformer, every parameter participates in every forward pass. Doubling the parameters roughly doubles the arithmetic per token, so size and cost move together.

A mixture-of-experts layer replaces one feed-forward block with many — the experts — plus a small router that picks a few of them per token. If a layer has 128 experts and the router activates 8, that layer does roughly a sixteenth of the work its total size suggests. The model is large in memory and small in compute per token.

Two parameter counts, not one

This is why model cards now quote two figures, and why comparing on the first one alone is misleading:

• Total parameters — everything in the checkpoint. This decides how much memory is needed to serve it, so it drives whether a provider can host it at all.
• Active parameters — what actually runs for one token. This drives arithmetic, latency, and therefore price.

A model advertised at 400B total with 40B active is, to a first approximation, priced and served like a 40B model that happens to need a great deal of memory.

Why the price follows the smaller number

Per-token price tracks the work per token, and the work per token is set by the active count. That is the whole reason sparse models exist commercially: they buy the quality that comes with capacity without paying dense inference cost for it.

It also explains a thing that otherwise looks like a mistake in a catalogue — a model with a far larger headline size priced near a much smaller one. Nothing is wrong; they have similar active counts.

What it costs you

• Memory does not get cheaper. All the experts must be resident even though most are idle, so the hosting requirement follows the total count. This is why sparse models appear on fewer providers.
• Throughput can be less predictable. Which experts a token needs depends on the token, so batching is harder than for a dense model and per-request latency can vary more under load.
• Quality is not a straight line in either number. Active parameters predict cost well and capability only loosely. Treat both counts as inputs to a decision you verify on your own task, not as a ranking.