Phi-3's Tokenizer: Where It Was Reused From

Phi-3-mini does not have its own vocabulary. It reuses Llama-2’s, extended by 64 slots. That decision explains its vocabulary size, its special-token ids, and why the same paragraph costs more tokens on Phi-3-mini than on Phi-4.

Which tokenizer each size uses

Microsoft’s model cards state this per checkpoint, and the family is not uniform:

checkpoint            tokenizer base                 vocabulary size
Phi-3-mini            Llama-2 (SentencePiece BPE)    32,064
Phi-3-medium          Llama-2 (SentencePiece BPE)    32,064
Phi-3.5-mini          Llama-2 (SentencePiece BPE)    32,064
Phi-3.5-MoE           Llama-2 (SentencePiece BPE)    32,064
Phi-3-small           tiktoken-based                 100,352
Phi-4                 tiktoken-based                 100,352

So “the Phi-3 tokenizer” is two tokenizers. A token count computed for Phi-3-mini does not transfer to Phi-3-small, and a context-budget calculation carried from a Phi-3 deployment to a Phi-4 one will be wrong in the direction that makes you truncate more than you need to.

Why 32,064 and not 32,000

Llama-2’s vocabulary is 32,000 entries. Phi-3-mini reports 32,064, and the extra 64 are not new words. They are the special tokens the instruct format needs — <|endoftext|>, <|system|>, <|user|>, <|assistant|>, <|end|> — plus a run of placeholder entries reserved for later use.

The round number is deliberate. The embedding matrix and the output projection both have the vocabulary as one dimension, and matrix multiplication on tensor cores is fastest when that dimension is a multiple of 64 or 128. Padding to 32,064 costs a few megabytes and buys aligned kernels. It also means the padded rows exist in the weights and are never produced by the tokenizer — which is why a sampler that has been told to consider every logit can, in rare malfunctioning configurations, emit an id that decodes to nothing.

The reserved placeholders are why Phi-3.5-vision could add image tokens without changing the embedding shape or invalidating anything trained against the earlier layout. It is cheap forward compatibility, and it is the reason the id numbers have gaps in them; see the id table in Phi-3’s chat template and special tokens.

What reuse buys

Training a tokenizer is cheap; changing one after the fact is not, because the vocabulary is baked into every embedding row. Adopting an existing one has concrete downstream effects:

• Tooling works immediately. Anything that already handled Llama-2’s SentencePiece model — dataset pipelines, token counters, GGUF conversion, speculative-decoding setups — handles Phi-3-mini without modification.
• Draft models line up. Speculative decoding requires the draft and target models to share a vocabulary. A shared base vocabulary makes a Llama-family small model a candidate drafter, which is not possible across a 32K/100K split.
• Pre-tokenized corpora are reusable. Training data already encoded for one model does not need re-encoding for the other, which is a real cost at pretraining scale.

The move to a tiktoken-based 100K vocabulary for Phi-3-small and Phi-4 gives up all three in exchange for the compression discussed next.

What a 32K vocabulary costs

A tokenizer’s job is to represent text in as few tokens as possible. A larger vocabulary can afford to keep longer, rarer pieces as single entries, so it represents the same text in fewer tokens. The gap is small for ordinary English and large for two things Phi is often pointed at:

• Code. Indentation runs, punctuation clusters and identifiers in camelCase or snake_case are exactly what a 32K vocabulary has no room to keep whole. Four spaces may be four tokens.
• Non-English text. A vocabulary built mostly from English text falls back to byte-level pieces for other scripts. Cyrillic, Greek, Arabic and CJK text can cost several tokens per character.

Combine that with Phi-3-mini-4k’s 4,096-token window and the effect compounds: the smallest window in the family is paired with the less efficient of the two vocabularies. A source file that fits comfortably in Phi-4’s 16K may not fit in Phi-3-mini’s 4K even though the ratio of the windows suggests it should. See the Phi-3 context window table for the lengths.

It cuts the other way for latency, though, and this is the part people forget. Fewer tokens for the same text means fewer forward passes to produce the same answer, and generation cost is linear in output tokens. A model with a more efficient vocabulary is doing less work per unit of prose, which partly offsets whatever it costs per token. Comparing two models on tokens is therefore comparing them on different units; compare on the text you actually send.

Adding your own special tokens

The reserved placeholder slots make this tempting, and it is a reasonable thing to do when fine-tuning — a marker for a retrieved document boundary, or for a domain-specific field, that the model can learn to treat as structural rather than as prose. Two mechanics matter.

First, a token added to the tokenizer does not exist in the model until the embedding matrix is resized, and a resized row is random-initialised. An untrained embedding produces noise, so this is only useful in combination with fine-tuning that teaches the model what the new token means:

tok.add_special_tokens({"additional_special_tokens": ["<|doc|>", "<|/doc|>"]})
model.resize_token_embeddings(len(tok))   # new rows are randomly initialised
# ...then fine-tune. Without training, these embeddings mean nothing.

Second, resizing changes the vocabulary dimension, and if the new size is not a multiple of 64 you lose the alignment the 32,064 padding bought. resize_token_embeddings accepts a pad_to_multiple_of argument for exactly this. Using the existing placeholder slots instead — renaming a reserved entry rather than appending a new one — avoids the resize entirely, which is precisely what the slots are there for.

Counting tokens correctly

There is no shortcut and no ratio worth memorising. Count with the tokenizer belonging to the checkpoint you will actually call:

from transformers import AutoTokenizer

mini = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")
phi4 = AutoTokenizer.from_pretrained("microsoft/phi-4")

text = open("handler.py").read()
print(len(mini(text).input_ids), len(phi4(text).input_ids))

# And count the rendered conversation, not the raw content:
messages = [{"role": "user", "content": text}]
print(len(mini.apply_chat_template(messages, add_generation_prompt=True)))

That last line is the one that matters in production. The role tags, the newlines and the generation prompt are all tokens, and a budget computed on the message content alone under-counts every request by a fixed amount that grows with the number of turns.

Two habits follow from the split vocabulary. First, do not cache token counts across models: a length computed once and reused when somebody swaps Phi-3-mini for Phi-4 will be wrong in the direction that makes you truncate content you did not need to. Store the tokenizer identity alongside any count you persist. Second, be sceptical of every published characters-per-token or words-per-token ratio, including ones stated confidently for “Phi”. Such a ratio is a property of a tokenizer and a corpus together, and both halves of this family have a different tokenizer from the other.

The tokenizer is also, quietly, a versioned dependency. It ships in the checkpoint as tokenizer.json and tokenizer_config.json, and those files have been patched after release on Phi repositories — which is enough to change a token count, a special-token id, or where generation stops. If token budgets are load-bearing for you, pin the revision; pinning a Phi checkpoint covers the mechanics.