Why a Mixture-of-Experts Model's VRAM Need Isn't Its Total Parameter Count

Mistral describes Mixtral 8x7B as having 46.7 billion total parameters and using 12.9 billion per token. The second number tells you how fast it runs. The first tells you whether it runs at all, and it is the one your card cares about.

Two parameter counts

In its Mixtral of experts announcement, published in December 2023, Mistral AI states that Mixtral has 46.7B total parameters and only uses 12.9B per token, and that it processes input and generates output at the same speed and cost as a 12.9B model. Both halves of that sentence are true and they describe different resources.

The name “8x7B” is misleading in a specific way: it is not eight 7B models, and 8×7 is not 46.7. The experts replace only the feed-forward block in each layer. Attention projections, embeddings and norms are shared across all eight, so the total is well below eight copies and the active fraction is well above one eighth.

Memory follows the total

Weights are parameter count times bits per weight over eight, exactly as for a dense model. Using llama.cpp’s published 4.8944 bits/weight for Q4_K_M:

total  46.7e9 * 4.8944 / 8 = 2.857e10 bytes = 26.6 GiB
active 12.9e9 * 4.8944 / 8 = 7.892e9  bytes =  7.4 GiB

the number you must fit:   26.6 GiB
the number that sets speed: 7.4 GiB of weights read per token

That is a 3.6x gap, and it is the whole of the sizing problem. A card that could run a dense 13B at Q4 comfortably cannot load Mixtral at all, even though Mixtral moves roughly the same amount of data per token. Sparse models are cheap in bandwidth and expensive in capacity — the opposite of the trade most local setups are provisioned for, since consumer cards are far more limited in VRAM than in bandwidth.

Generalise it: for a model advertised as T total and A active, memory scales with T and decode speed scales with A. A model with a 10:1 ratio is a small model that needs a large card. That is the trade, and it is why sparse models appear on fewer local setups than their active count would suggest. The routing side of it is covered under how a modern MoE architecture routes tokens.

Why every expert has to be resident

The obvious optimisation is to keep only the experts you need. It does not work, and the reason is in the shape of the routing decision.

• The router runs per layer, not per request. Every MoE layer has its own gate, so a model with 32 sparse layers makes 32 independent routing decisions for a single token. There is no “expert for this conversation” to load once.
• The decision depends on the hidden state. The gate reads the activations arriving at that layer, which depend on everything before it, including the token just generated. It cannot be computed ahead of the forward pass that needs it.
• The decision is per token, and tokens differ. Over a few hundred generated tokens a session will touch essentially every expert in every layer. Caching the hot ones does not help when the working set is the whole model.

So the runtime holds all of them. The saving is in arithmetic and in bytes read per token, not in bytes held.

It is also worth being exact about what “active” counts. The 12.9 billion figure is not one eighth of 46.7 plus rounding: it is the shared parameters, which are read for every token no matter what the router decides, plus the experts the router selected in each layer. Attention projections, embeddings, norms and the routers themselves are all in the shared group. So the active count has a floor that sparsity cannot go below, and a model advertising a very large total with a very small active count is one where the feed-forward blocks dominate the total and the shared part is comparatively tiny. That is the design the sparsity ratio is really describing.

The cache does not get sparser

Sparsity applies to the feed-forward blocks. Attention is dense in every MoE design in common use, so the key/value cache is sized exactly as it would be for a dense model of the same depth and head configuration:

kv_per_token = 2 * L * H_kv * D * b

Mixtral 8x7B: 32 layers, 8 kv heads, head dim 128, fp16
  2 * 32 * 8 * 128 * 2 = 131,072 bytes = 128 KiB per token

  32,768 tokens -> 4.00 GiB   (identical to a dense 8B of the same shape)

Which means the total budget for a sparse model is a large fixed weights term plus an ordinary context term. On a 24 GiB card, Mixtral at Q4_K_M is 26.6 GiB before any context — it does not fit, and no amount of shortening the context changes that. The relevant lever is the quant level, or a smaller sparse model. Context sizes for this family are covered under Mixtral’s context window.

Expert offloading, and what it costs

Runtimes can keep the shared parameters — attention, embeddings, norms — on the GPU and the expert feed-forward tensors in system RAM, fetching over PCIe as the router selects them. This is a real technique and it does let a sparse model run on a card that cannot hold it. The cost is derivable from the bus:

bytes fetched per token, if all selected experts miss:
  active_expert_params * bpw / 8

PCIe 4.0 x16 is 32 GB/s theoretical, less in practice.
Fetching even 4 GiB of expert weights per token over that bus:
  4.29e9 / 32e9 = 0.134 s  ->  about 7 tokens/s ceiling, before any compute

A ceiling, not a measurement, and one that assumes every fetch misses. Real implementations cache recently-used experts and overlap transfer with compute, so the observed rate is somewhere between that ceiling and the all-resident rate. The point of the arithmetic is the shape: a bus an order of magnitude slower than VRAM sets the pace whenever the working set does not fit, which is the same mechanism as ordinary layer offloading.

Expert offloading is nonetheless a better bet than ordinary layer offloading for the same shortfall, and the reason is in the numbers above. When you offload layers of a dense model, every offloaded byte is read every token. When you offload experts, only the selected fraction is needed — so a model that keeps the shared parameters resident is moving the active expert bytes rather than the whole offloaded set. That is the one place where sparsity pays back some of the capacity it costs, and it is why a sparse model can be a reasonable choice on a card that cannot hold it while a dense model of the same file size is not.

Two practical consequences for sizing a machine. First, when comparing a sparse model against a dense one, compare the totals for capacity and the actives for speed, and never one number against the other. Second, prefer system RAM over another GPU if the shortfall is in the experts and prefer another GPU if it is in the shared parameters — the shared group is read every token and belongs on the fast side of the bus, whatever else has to move.