Mixtral's Context Window

Mixtral 8x7B documents 32k tokens and Mixtral 8x22B documents 64k. Both models were retired from Mistral’s API on 30 March 2025, which changes what those numbers are useful for but does not make them wrong.

The two figures

From Mistral’s own model cards:

• Mixtral 8x7B — a documented context window of 32,000 tokens, API name open-mixtral-8x7b, 47B total parameters with 13B active, Apache 2.0, released 11 December 2023. Sourced from Mistral’s Mixtral 8x7B model card.
• Mixtral 8x22B — a documented context window of 64k tokens, API name open-mixtral-8x22b, 141B total parameters with 39B active, Apache 2.0, announced 17 April 2024. Sourced from Mistral’s Mixtral 8x22B model card and its release announcement of 17 April 2024, which describes a 64K tokens context window.

The doubling between the two releases is the whole story of that four-month period: same architecture family, larger experts, twice the window.

Both are retired on Mistral’s API

Both model cards carry a retirement date of 30 March 2025, and Mistral’s models overview lists them among deprecated models with Mistral Small 4 named as the replacement. If you send open-mixtral-8x7b to the API today you are not going to get a 32k window; you are going to get an error.

This is the reason the question is still asked. Most people searching for Mixtral’s context window are holding something written while the model was current — a benchmark table, a blog post, a repository README, a config file in a service nobody has touched in a year — and need to know whether the number in front of them still describes anything. It describes the weights. It no longer describes an endpoint.

The weights still work

Both Mixtral models were released under Apache 2.0 in base and instruct forms, and an Apache 2.0 release cannot be withdrawn from people who already have it. Retirement is a statement about Mistral’s hosted service, not about the artifact. If you are running Mixtral on your own hardware or through a third-party host, the 32k and 64k figures continue to be the numbers that matter — see what Mistral’s Apache 2.0 release actually permits.

One caveat that catches people running the weights themselves: the window you get is the window your serving stack is configured for, not the window the model card documents. Inference servers take a maximum sequence length as a parameter, and it is commonly defaulted lower than the model supports to fit the key-value cache into available memory. A Mixtral 8x22B deployment that rejects a 40,000-token prompt is almost certainly configured at 32k, not broken.

Why the window has nothing to do with the expert count

“8x7B” invites the assumption that the eight experts have something to do with capacity for input. They do not, and separating the two ideas makes both model cards easier to read.

The mixture-of-experts structure replaces the feed-forward block in each layer with several alternatives plus a router that picks a couple per token. That is a statement about how much arithmetic each token costs — 13B active out of 47B total for 8x7B, 39B out of 141B for 8x22B — and it says nothing about how long a sequence the attention layers can address. Context length is set by the positional encoding scheme and by what the model was trained and fine-tuned on, and it would be unchanged if the experts were collapsed into one dense block.

Practically: the sparse structure is why these models were cheap for their size, and the window is why they could or could not hold your document. Two independent numbers on the same card.

The naming convention adds to the confusion because it is not arithmetic either. “8x7B” suggests eight seven-billion parameter models, which would be 56B; the card says 47B total. The difference is that only the feed-forward blocks are replicated across experts — attention layers, embeddings and the rest are shared by all of them, counted once. That sharing is also why the active figure is 13B rather than a clean fraction of the total: every token pays for the whole shared trunk plus its two selected experts. The same reading applies to 8x22B at 141B total and 39B active. Neither number is a typo, and neither can be derived from the name.

What the window costs you when you host it

The model cards give GPU memory requirements for the weights alone — roughly 94GB down to 13GB for 8x7B and 283GB down to 71GB for 8x22B across bf16 to fp4 precision. Those figures are for parameters. They do not include the key-value cache, which is what actually grows with your context length and which is allocated per concurrent request.

This is the trap in provisioning a long-context deployment. You size the hardware from the weights table, it fits, and then the service falls over at four concurrent users with 30,000-token prompts because the cache is competing with the weights for the same memory. On a mixture-of-experts model the squeeze is worse than on a dense model of equivalent speed, because all the experts must be resident even though most are idle for any given token — the memory footprint follows the total parameter count while the compute follows the active one.

If you are choosing between the two Mixtrals on window alone, the honest framing is that 64k costs you roughly three times the weight memory of 32k before you have served a single request. For most workloads that reached for Mixtral, the current Mistral Small line is the better trade — 256k documented, covered separately — and the reason to stay on Mixtral is a licence or a reproducibility requirement rather than a capability one.

Reading an old benchmark that quotes these numbers

Most people arrive at this question through a document written in 2024, so it is worth saying what those documents are and are not evidence of now.

A context-window figure is a durable property of a released checkpoint. If a paper says Mixtral 8x22B was evaluated at 64k, that remains true of those weights forever, and it is a fair number to reproduce. What is not durable is any comparison built on top of it. A 2024 table showing Mixtral’s 64k against a competitor’s 32k was describing a moment when tens of thousands of tokens was the frontier; both sides of that comparison have since moved by an order of magnitude, so the ranking carries no information about the current field even though neither number was ever wrong.

Long-context evaluation results from that era need reading with the same care for a different reason. A model documented at 64k has been trained or adapted to address sequences that long; how well it uses the far end of them is a separate question that needle-in-a-haystack style evaluations were built to probe, and results on those varied a great deal between models with identical documented windows. “Supports 64k” is a statement about what the API or the config will accept, not a guarantee that retrieval accuracy at token 60,000 matches accuracy at token 2,000. That distinction was true then and is still true of the 256k models today.

The practical advice, if you have inherited a system built on these numbers: treat the window as a hard input constraint you must respect and the quality-at-length behaviour as something to verify on your own documents. The first is documented; the second never has been, for any model, by any vendor.