DeepSeek-V3's Mixture-of-Experts Architecture and Active Parameters

DeepSeek-V3 is published as a 671B-parameter model that activates 37B parameters per token. Both figures are correct and they answer different questions; using the first one to reason about cost is the mistake the architecture exists to make available.

The two numbers

The DeepSeek-V3 Technical Report, published by DeepSeek-AI in December 2024, states a total of 671 billion parameters with 37 billion activated for each token. The same two figures appear on the model repository. The architecture combines Multi-head Latent Attention with DeepSeekMoE, both introduced in the earlier DeepSeek-V2 work and carried forward.

Written as a ratio, roughly 5.5% of the model participates in producing any given token. That figure is not a compression trick or a quantisation claim. Every one of the 671 billion parameters is a real, trained parameter that must be loaded to serve the model; they are simply not all consulted at once.

What 'active' means for one token

In a dense transformer every parameter participates in every forward pass, so doubling the parameter count roughly doubles the arithmetic per token. A mixture-of-experts layer breaks that link. It replaces the single feed-forward block with a large set of smaller ones — the experts — plus a small router that scores them and selects a few per token. Only the selected experts run.

The selection is per token and per layer, not per request. A single sentence passing through the model touches a different subset of experts at every position and at every layer, and across a long response very nearly all of the model gets used — just never simultaneously. That is why the correct phrasing is “activated per token” and why “DeepSeek-V3 only uses 37B of itself” is a misreading.

Attention is the other half of the design and it is not sparse. Multi-head Latent Attention instead compresses the key/value cache into a lower-dimensional latent representation, which reduces the memory the cache consumes per token rather than the arithmetic per token. The two mechanisms attack different bottlenecks: MoE cuts the feed-forward compute, MLA cuts the cache footprint that otherwise limits how long a context and how large a batch you can serve.

Shared experts and routed experts

DeepSeekMoE’s distinguishing feature is that not all of the active parameters are chosen. The design splits experts into two groups: a small number of shared experts that run for every token, and a much larger pool of routed experts from which the router selects a handful.

The reasoning behind the shared group is worth understanding because it explains a class of MoE failure it was designed to avoid. In a purely routed design, knowledge that every token needs — basic syntax, common patterns — has to be learned redundantly inside many experts, because any given expert might be the one selected. Dedicating a few always-on experts to that common load frees the routed pool to specialise, which is what makes fine-grained routing worthwhile in the first place.

The report also describes an auxiliary-loss-free strategy for keeping expert load balanced. The problem it addresses is real and structural: routers left alone tend to collapse onto a favourite subset of experts, which wastes capacity and makes throughput unpredictable, and the usual remedy — an auxiliary loss penalising imbalance — trades model quality for balance. Removing that tradeoff is one of the report’s claimed contributions and it is the part to read if you care about serving behaviour rather than benchmark scores.

What each number predicts

• Active parameters predict compute, latency and price. Arithmetic per token scales with what actually runs. This is why a 671B model can be priced in the neighbourhood of far smaller dense models rather than in proportion to its headline size.
• Total parameters predict memory and who can host it. Every expert must be resident because the router may select it for the next token. At any sensible precision, 671B parameters is a multi-node deployment. This is the number that decides whether you can run it at all.
• Neither predicts quality. Capability is a function of architecture, data and training budget together. Two models with the same active count can differ enormously, and the only reliable comparison is on your own task.

The general form of this argument, independent of DeepSeek, is on the mixture-of-experts page. What is specific to V3 is the ratio: at roughly eighteen to one, total to active, the gap between the two numbers is wider than in most published MoE models, and so is the gap between what the headline suggests and what a token costs.

Memory does not shrink

Sparsity buys arithmetic, not storage. All 671 billion parameters sit in memory throughout, because the router’s choice is made per token and cannot be anticipated. That is the cost side of the design and it has consequences you can observe from outside.

• Fewer providers host it. A model requiring many accelerators just to hold its weights is served by fewer people than one that fits on a single node, whatever its per-token cost looks like.
• Batching is harder. Tokens in the same batch want different experts, so the work is less uniform than a dense model’s and per-request latency varies more under load. This is part of why a capacity-related interruption has its own finish_reason value in the DeepSeek API.
• Quantisation matters more, not less. With memory rather than compute as the binding constraint, reducing the precision of stored weights is the lever that changes what hardware can serve the model.

Reading the two numbers on a model card

Once sparse models are in the catalogue, parameter counts stop being comparable in the way they used to be, and a few habits keep the comparison honest.

1. Find out whether it is sparse at all. A single parameter count with no mention of experts is a dense model, and for a dense model the one number is both figures. Comparing a dense model’s count against a sparse model’s total is comparing two different quantities.
2. Use the active count for cost and latency questions. “Will this be fast enough” and “why is this priced here” are answered by what runs per token, not by the headline.
3. Use the total for deployment questions. Memory, hardware and which providers offer it at all follow the total. If you are self-hosting, this is the number that decides feasibility, and precision is the lever that moves it.
4. Do not infer quality from either. The ratio between them is an architectural choice, not a score. A wider gap means more capacity per unit of compute, and whether that capacity was well-trained is a separate question that only evaluation answers.
5. Check the context window separately. It is unrelated to sparsity and is set by entirely different parts of the design — where the window comes from is its own page.

If you only remember one thing from this page, make it the sentence that the headline number describes what has to be in memory and the smaller number describes what a token costs. Almost every confused comparison between modern models resolves once those two are kept apart.