Qwen3's Mixture-of-Experts Sizes and Active Parameter Counts

Qwen3 ships two parameter counts per sparse checkpoint and they differ by an order of magnitude. Which one you need depends on whether you are buying memory or buying throughput, and the model card gives you both because neither answers the other question.

What the model cards publish

The Qwen3 release of April 2025 included two mixture-of-experts checkpoints alongside six dense ones. Both MoE cards, published by the Qwen team on Hugging Face, list total parameters, non-embedding parameters, layer count, expert count and the number of experts activated per token:

Qwen3-30B-A3B
  total parameters      30.5B
  activated per token    3.3B
  layers                    48
  experts                  128
  experts activated          8
  native context        32,768 tokens (131,072 with YaRN)

Qwen3-235B-A22B
  total parameters       235B
  activated per token     22B
  layers                    94
  experts                  128
  experts activated          8
  native context        32,768 tokens (131,072 with YaRN)

The naming convention encodes the same two numbers: Qwen3-30B-A3B is thirty billion parameters in the checkpoint with roughly three billion active, and Qwen3-235B-A22B is likewise. A Qwen3 name without an A segment is a dense model, where the two counts are the same number. That convention is the subject of a separate page on reading Qwen3 model names.

Where the active number comes from

In a dense transformer every parameter participates in every forward pass. In a Qwen3 MoE layer, the feed-forward block is replaced by 128 smaller feed-forward experts plus a router. For each token, at each layer, the router scores all 128 experts and the top 8 are run. The other 120 sit in memory doing nothing for that token.

So the active count is the model minus the experts that were skipped. Attention weights, embeddings, layer norms and the router itself are used every time and count in full; the expert weights count at eight out of 128, which is one sixteenth. That is why 30.5B collapses to 3.3B rather than to 30.5/16 — the dense remainder does not shrink.

Splitting the checkpoint into its two halves makes the shape of that arithmetic clear. Almost all of a Qwen3 MoE checkpoint is expert weights; a small remainder is attention projections, embeddings and norms. The sparse half is divided by sixteen and the dense half is not, so the active figure is the dense remainder plus a sixteenth of the experts. Applied to Qwen3-30B-A3B, roughly 30 billion parameters of experts contribute about two billion active, and the rest of the active count is the dense machinery that runs regardless. That is also why the ratio is not identical between the two checkpoints — 30.5B to 3.3B is about 9×, 235B to 22B is about 10.7× — even though both route 8 of 128. The proportion of the model that is expert weights differs.

One more property is easy to miss and matters for capacity planning: the routing decision is per token and per layer. Qwen3-235B-A22B makes 94 routing decisions for every token it reads or writes, each selecting 8 experts out of 128 independently. Over a few hundred tokens of a single request, the union of experts touched approaches the whole pool. Sparsity is a claim about work per token, not about how much of the model a request uses.

Two details of the Qwen3 design are worth knowing because they differ from earlier Qwen MoE generations. The experts are fine-grained, which means each individual expert is much narrower than the dense feed-forward block it replaces, so activating eight of them is not eight times the work of one dense block. And Qwen3 does not use a shared always-on expert, which some earlier sparse models did. The routing choice is made per token and per layer, not once per request, so the set of experts touched by a long generation is close to all of them even though each individual token touches eight.

The four config.json fields

You do not have to trust a card. Every Qwen3 MoE repository has a config.json with the routing configuration in it, and four fields carry all of it:

• num_experts — how many experts exist per MoE layer. 128 on both April 2025 checkpoints.
• num_experts_per_tok — how many the router activates. 8.
• moe_intermediate_size — the width of one expert’s feed-forward block. This is the number that makes the experts “fine-grained”; compare it to intermediate_size on a dense model of similar depth.
• num_hidden_layers — how many layers, and therefore how many times per token the routing decision is repeated.

from huggingface_hub import hf_hub_download
import json

path = hf_hub_download("Qwen/Qwen3-30B-A3B", "config.json")
cfg = json.load(open(path))

print(cfg["num_experts"], cfg["num_experts_per_tok"])
print(cfg["moe_intermediate_size"], cfg["num_hidden_layers"])

A checkpoint whose config.json has no num_experts field is dense, whatever the name suggests. That is the cheapest possible check and it is the one to run when a quantised or community re-upload claims to be something it is not.

What the active count predicts

Active parameters predict arithmetic per token, and therefore generation speed and per-token price. A hosted Qwen3-235B-A22B is priced and served much more like a 22B model than like a 235B one, which is the entire commercial reason sparse models exist.

Two things it does not predict, and both cost real money:

• Memory. Every expert must be resident, because the router can ask for any of them at any layer for any token. Weights for Qwen3-235B-A22B at BF16 are on the order of 470 GB before you allocate a single token of KV cache. The 22B in the name buys you nothing here. This is why sparse checkpoints appear on far fewer self-hosting setups than their active count would suggest.
• Batched throughput. Which experts a token needs depends on the token, so a batch of unrelated requests can touch most of the 128 experts at once. The per-token saving is real at batch size one and erodes as concurrency rises, which is why hosted pricing for a sparse model is not simply its active count scaled.

The KV cache is a fourth thing the active count says nothing about, and on long-context work it is often the binding constraint rather than the weights. Cache size is a function of layer count, key-value head count and sequence length — none of which is affected by routing, because attention in a Qwen3 MoE layer is dense. A 94-layer model caches 94 layers of keys and values per token whether it activated 8 experts or 128. This is why a sparse model can be cheap per token at short context and stop fitting entirely at long context, and why the memory sums on a self-hosting plan have to be done at the sequence length you actually intend to serve.

Capability is a fifth thing neither number predicts. Total parameters are a capacity budget and active parameters are a compute budget; benchmark results are a function of both plus the training run, and the only honest way to rank two Qwen3 sizes for your workload is to run your workload. Where the two counts genuinely help is in eliminating options before you test them — a model whose total will not fit your hardware is not a candidate regardless of how good it is, and a model whose active count implies a per-token price above your budget is not one either.

Later Qwen3 MoE checkpoints

The Qwen3 line kept growing after April 2025, and every addition follows the same naming rule, so the two counts are readable from the name before you open anything: Qwen3-Coder-480B-A35B is 480 billion total with about 35 billion active, and Qwen3-Next-80B-A3B is 80 billion total with about three billion active — an unusually sparse ratio achieved with a much larger expert pool than 128.