Mistral's Tokenizer: v1 to v3 and Why the Vocabulary Grew

Mistral has shipped four tokenizer versions and the vocabulary quadrupled along the way. That number is not trivia: it decides how many tokens your prompt costs, and a count taken with the wrong version will be wrong in a predictable direction.

Four versions, and what each added

Mistral’s mistral-common repository and the tokenization deep-dive in Mistral’s cookbook describe the lineage as follows:

• v1 — the tokenizer behind the first Mistral models. SentencePiece, no control tokens beyond the basics.
• v2 — introduced control tokens and function calling. This is the version at which the tokenizer stopped being purely a text encoder and started encoding conversation structure.
• v3 — a better function calling implementation. Same family as v2, refined tool-call encoding.
• v3-Tekken — a different build of v3 based on tiktoken rather than SentencePiece, with a far larger vocabulary. It powers Mistral NeMo 12B and Pixtral 12B.

The thing to take from that list is that versions two and three are about structure, not about text. Adding control tokens means the tokenizer gained dedicated single-token symbols for things like the start of an instruction or the boundary of a tool call. Before that, those boundaries had to be spelled out in ordinary text and cost several tokens each, and — worse — were indistinguishable from a user typing the same characters.

The documented vocabulary sizes

The figures Mistral publishes for each version:

version      backend         vocab_size
─────────────────────────────────────────
v1           sentencepiece       32,000
v2           sentencepiece       32,768
v3           sentencepiece       32,768
v3-tekken    tiktoken           128,000  (built out to 131,072 in
                                          recent implementations)

The step from 32,000 to 32,768 between v1 and v2 is not a capability change in any meaningful sense; 32,768 is 2¹⁵, and the extra 768 slots are headroom used for the control tokens that version introduced. The step from 32,768 to 128,000 at Tekken is the real change, and it is a change of kind rather than degree.

Why a bigger vocabulary means fewer tokens

A subword tokenizer works from a fixed inventory of pieces and encodes your text as the shortest sequence of pieces it can. With 32,000 entries, that inventory has to cover English, every other language the model handles, source code, punctuation, whitespace runs and byte fallbacks. Something has to give, and what gives is everything that is not English: a common English word is one token, while a common German compound or a Cyrillic word or a run of four-space indentation fragments into several.

Quadruple the inventory and you can afford to give whole tokens to things that previously fragmented. The consequence is arithmetic: the same string encoded with a larger vocabulary produces the same number of tokens or fewer, never more, because the smaller inventory is effectively a subset of the choices available to the larger one. The effect is small for plain English — that was already well covered — and large for code, for non-Latin scripts, and for anything with heavy whitespace or markup.

That matters in three places at once. Your context window is counted in tokens, so a better tokenizer is a bigger effective window for the same documented figure. Your bill is counted in tokens. And generation speed is per token, so fewer tokens for the same output is a real latency win, not an accounting one.

It also means token counts are not portable. A count computed with a v3 SentencePiece tokenizer is not a count for a Tekken model, and neither is a count from another vendor’s tokenizer. Estimating Mistral usage with a rule of thumb borrowed from a different model family will be wrong by a margin that grows with how unlike English your input is.

Control tokens are the other half

The vocabulary size gets the attention, but the change at v2 — introducing control tokens — is the one that made the modern API possible, and it is worth understanding because it explains a security property people assume without knowing where it comes from.

A control token is an entry in the vocabulary that has no textual spelling. It exists as an id, it is emitted by the template renderer and recognised by the model, and — this is the point — there is no sequence of characters a user can type that encodes to it. When your request is rendered, the boundaries between the system content, the user turn and the assistant turn are marked with these tokens.

Without them, turn boundaries have to be spelled in ordinary text: something like a literal [INST] marker made of the characters [, INST and ]. And if a boundary is made of ordinary characters, a user can type those characters. A message containing the marker verbatim would, once concatenated, be indistinguishable from a genuine turn boundary — the model would read the user’s text as though part of it had arrived from the operator. Control tokens close that specific hole by construction, because the encoder never produces them from user input no matter what the user writes.

Two caveats keep this from being a general defence. It only holds if the encoding path actually refuses to emit control tokens from text — some tokenizer APIs offer a flag that permits it, and enabling that flag on user-supplied strings reopens the hole exactly. And it says nothing about instructions arriving as ordinary content in a retrieved document, which are correctly attributed to the right turn and still read as instructions. Control tokens guarantee turn attribution, not instruction provenance.

The practical version: never build a prompt by string concatenation when a chat template exists. Rendering through the template is what puts the control tokens in the right places, and it is also why the token count of a conversation is larger than the sum of its message contents.

Tekken is a different kind of tokenizer

The backend swap is not cosmetic. SentencePiece and tiktoken are different algorithms with different handling of whitespace and byte fallback, so Tekken does not merely have more entries — it segments differently. Mistral’s own documentation describes it as dealing with encoding differently rather than as a scaled-up v3.

The operational consequence is that you cannot treat “v3” as one thing. If you are loading a tokenizer by version string to count tokens locally, loading v3 when the model uses v3-Tekken gives you a number that is confidently wrong, and it will be wrong in the direction of over-estimating, which is the safer direction but still means you are truncating context you did not need to truncate.

Checking which one your text is hitting

Rather than trusting any published count — including the ones on this page — the reliable move is to tokenize your own representative text with the tokenizer the model actually ships. Mistral publishes mistral-common for exactly this.

1. Install the library: pip install mistral-common.
2. Load the tokenizer by model name rather than by version number, so the version follows the model:

   from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
   
   tok = MistralTokenizer.from_model("mistral-large-latest")

3. Encode a request, not a bare string. Token counts that ignore the chat template understate the total, because the control tokens wrapping each turn are real tokens you are billed for.
4. Repeat with a second tokenizer version and compare the counts on your corpus. The direction of the difference is predictable from the mechanism above; the size of it depends entirely on what your text looks like, which is why a number quoted from someone else’s corpus is not useful to you.
5. Record the count you get next to the model version you got it from. When you change models, re-run it rather than carrying the number over.

If your budgeting depends on token counts, this is the one measurement worth having your own numbers for, because it is cheap to take and every published estimate is an estimate of somebody else’s text.