Mistral Small's Context Window

The current Mistral Small line documents a 128k-token context window. An earlier generation carrying the same product name documented 32k, and code written against one and pointed at the other is the reason this page exists.

The number

Mistral documents the Small 3 generation — the 24B-parameter releases, including the 3.1 and later revisions — with a 128k-token context window, on the model pages in Mistral’s model overview documentation and on the corresponding cards under the mistralai organisation on Hugging Face. 128k is the rounded figure; the configuration value is a power of two and the API reports it exactly, which is the number your budgeting code should use.

Which Small you mean matters

“Mistral Small” has named more than one thing. The product name has been carried across generations with different parameter counts, different licences and — the part that breaks code — different context lengths. An earlier 22B-class Small documented a 32k window; the 24B Small 3 generation documents 128k. That is a factor of four, and it moves in the direction that hides the bug: code written for the shorter window works perfectly against the longer one, so nobody notices until someone points a service back at the older snapshot, or runs the older open weights locally, and every long request fails at once.

The lesson is not to memorise the mapping. It is that mistral-small-latest is not one model and its window is not one number over time. If your chunking, your retrieval budget or your history-truncation logic has a constant in it, that constant belongs next to a pinned model id — which is what pinning a dated snapshot is for.

Getting the exact integer

Marketing figures are rounded and configuration values are not. The models endpoint returns the real number for every model your key can reach:

curl -s https://api.mistral.ai/v1/models \
  -H "Authorization: Bearer $MISTRAL_API_KEY" \
  | jq -r '.data[]
           | select(.id | test("small"))
           | [.id, .max_context_length, ((.aliases // []) | join(","))]
           | @tsv'

It is worth translating that integer into something you can hold in your head, with the caveat that the conversion is approximate and content-dependent. For ordinary English prose, a rough working figure is three quarters of a word per token, so a 128k window is on the order of ninety to a hundred thousand words — a long novel. Source code, heavily punctuated text, and languages that do not use the Latin script tokenise considerably less efficiently, sometimes by a factor of two or more, so the same window holds far less of them. Never size a budget on a character or word estimate when you are anywhere near the limit; count tokens with the model’s own tokeniser.

If you are running the open weights rather than calling the API, the equivalent value is max_position_embeddings in the checkpoint’s config.json. Read it from the repository you actually pull — that is the file the serving stack reads, and it cannot be out of date with respect to the weights sitting beside it.

The window is shared with the output

128k is not 128k of input. Prompt and completion occupy one sequence, so every token you generate reduces the room left. A request with 127k of prompt has about a thousand tokens of answer available regardless of what max_tokens says. Budget as:

system + tools + history + user_message + completion  ≤  max_context_length

Tool schemas are the term people forget. They are serialised into the prompt and a rich toolset can cost several thousand tokens on every single request, before any conversation happens. The full arithmetic, and the finish reason that tells you when you got it wrong, is in Mistral’s max_tokens behaviour.

What happens when you exceed it

The API rejects the request rather than silently truncating it. You get an HTTP error with a message naming the limit and the size of what you sent — which is the behaviour you want, because a silent truncation that drops your system prompt produces an answer that looks fine and is not.

The error is also the right place to notice a class of bug that has nothing to do with document size: unbounded conversation history. A chat service that appends every turn and never trims works fine for weeks and then fails for the handful of users whose conversations grew past the window — which means the failure arrives as a small number of confused reports rather than as an outage, and correlates with nothing you deployed. If you accumulate history, the trimming logic is not an optimisation you add later; it is part of the feature.

Handle it by counting before you send rather than by catching after. Two useful habits:

• Keep a safety margin. Budget to about 90% of the window. Tokenisation of user-supplied text is not perfectly predictable, and the chat template adds tokens you are not counting.
• Truncate history from the middle, not the end. The system prompt and the most recent turns carry the most signal. Dropping the oldest user turns while keeping the system message preserves the instructions that shape the answer.
• Count with the right tokeniser. An estimate from a different model’s tokeniser can be off by a substantial margin on the same text, and the direction of the error is not predictable. If the count is load-bearing, it has to come from the vocabulary the model actually uses.

Self-hosting adds a constraint the API hides from you entirely. The documented window is what the model architecture supports; what your hardware supports may be less, because the attention key-value cache grows linearly with sequence length and has to sit in the same memory as the weights. A 24B model at half precision is roughly 48GB of weights before a single token of context, and filling a 128k window on top of that is several gigabytes more per concurrent request. This is why a local runtime will often default to a context length far below the model’s maximum, and why raising the setting on a machine that cannot back it produces an out-of-memory failure partway through a long request rather than a clean rejection at the start. The window is a ceiling, not an allocation.

One last caution on the headline number: a documented window is a capacity, not a promise of uniform quality across it. Retrieval accuracy at the far end of a long context is a property of the model and of where in the prompt the relevant text sits, and it is worth measuring on your own data before designing a system that depends on filling the window.