Phi-3's Chat Template and Special Tokens

An instruction-tuned model is a base model plus a habit: given this exact arrangement of marker tokens, produce an answer. Get the arrangement wrong and Phi-3 still answers, worse, with no error anywhere.

The shape of a rendered turn

The chat template shipped in Phi-3’s tokenizer_config.json renders a conversation like this. Every turn opens with a role tag on its own line and closes with <|end|>:

<|system|>
You are a terse assistant. Answer in one sentence.<|end|>
<|user|>
Why does prefill dominate time to first token?<|end|>
<|assistant|>
Because the whole prompt is processed in one parallel pass before any output token is produced.<|end|>
<|user|>
And after that?<|end|>
<|assistant|>

Read the annotations off that literally. The role tag is followed by a newline, then the content, then <|end|> with no space before it, then a newline. The final line is the model’s cue: the string ends immediately after <|assistant|> and its newline, and the very next thing generated is the answer.

Three roles are recognised: system, user and assistant. Phi-3’s template places the system turn first if one is supplied and omits the block entirely if not — it does not fabricate a default system message, so an unprompted Phi-3 has no persona beyond whatever its instruction tuning left it with.

The tags are single tokens

This is the part that makes hand-rolling the format quietly wrong. <|user|> is not seven characters that the tokenizer breaks into pieces. It is one entry in the vocabulary, added above the inherited 32,000-token base vocabulary, and Phi-3 was tuned on that single id appearing at a turn boundary. Microsoft’s Phi-3-mini repository lists these in added_tokens.json, with <|endoftext|> at 32000, <|assistant|> at 32001, <|system|> at 32006, <|end|> at 32007 and <|user|> at 32010, with placeholder slots filling the gaps.

If you build the prompt as a Python f-string and tokenize it with add_special_tokens=False on a tokenizer that has not been told these are special, you can get the literal characters split into ordinary subword pieces. The model then sees something that resembles a turn boundary typographically and not at all numerically. The output degrades in the way that is hardest to debug: it is grammatical, on topic, and ignores the system turn.

The generation prompt

The trailing <|assistant|> with nothing after it is called the generation prompt, and forgetting it is the single most common template bug. Without it, the last thing the model sees is <|end|> after a user turn, and the highest-probability continuation is to open a new turn itself — so you get <|assistant|> printed as text, or worse, the model inventing another user message and answering that.

The mirror image of this is prefilling. Because the template is just text, you can put the beginning of the answer after the generation prompt and the model will continue it. Ending the prompt with <|assistant|>\n{"name": is a cheap way to force JSON out of a model that has no structured-output mode, which Phi-3 does not.

Render it rather than typing it

The template lives in the checkpoint, so let the tokenizer apply it. This is the only version that stays correct when you change size or revision:

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")

messages = [
    {"role": "system", "content": "You are a terse assistant."},
    {"role": "user", "content": "Why does prefill dominate TTFT?"},
]

text = tok.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,   # appends <|assistant|>\n
)
print(repr(text))

ids = tok.apply_chat_template(messages, add_generation_prompt=True)
print(ids[-3:])   # ends with the <|assistant|> id, not a split string

Printing with repr rather than print matters here: the difference between a correct render and a broken one is usually a newline, and a bare print hides it.

If you are serving through an OpenAI-compatible endpoint — vLLM, llama.cpp’s server, Ollama — the server applies the template for you when you post to /v1/chat/completions and does not when you post to /v1/completions. Mixing those up gives you a base-model continuation from an instruct checkpoint, which reads like the model got worse overnight.

Where the template is stored

The template is not code in the transformers library. It is a Jinja string in the checkpoint’s tokenizer_config.json, under the key chat_template, and it is worth reading once because everything above is derived from it:

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")
print(tok.chat_template)

# an abbreviated shape of what you will see:
# {% for message in messages %}
#   {% if message['role'] == 'system' %}{{ '<|system|>\n' + message['content'] + '<|end|>\n' }}
#   {% elif message['role'] == 'user' %}{{ '<|user|>\n' + message['content'] + '<|end|>\n' }}
#   {% elif message['role'] == 'assistant' %}{{ '<|assistant|>\n' + message['content'] + '<|end|>\n' }}
#   {% endif %}
# {% endfor %}
# {% if add_generation_prompt %}{{ '<|assistant|>\n' }}{% endif %}

Two consequences follow from the template living in the checkpoint rather than in your code. The first is that it can change when the checkpoint changes: a commit to tokenizer_config.json alters every prompt you render, with no change on your side and no version number to notice. That is an argument for pinning the revision of the tokenizer specifically, and not only of the weights.

The second is that serving frameworks keep their own copy. Ollama stores a TEMPLATE block in the Modelfile; vLLM accepts a --chat-template file that overrides whatever the tokenizer carries; llama.cpp infers one from GGUF metadata written at conversion time. Each of those is a chance for the served template to drift from the checkpoint’s. When a model behaves differently through your server than it does in a local script with the same weights, compare the two rendered prompts byte for byte before looking anywhere else — it is nearly always a missing newline or a dropped generation prompt.

Phi-4 uses different tags

Do not carry this template forward. Phi-4 moved to a ChatML-derived format built from <|im_start|>, a <|im_sep|> separator between the role name and the content, and <|im_end|>:

<|im_start|>system<|im_sep|>
You are a terse assistant.<|im_end|>
<|im_start|>user<|im_sep|>
Why does prefill dominate TTFT?<|im_end|>
<|im_start|>assistant<|im_sep|>

Note what changed beyond the spelling. The role name is now content between two markers rather than being the marker itself, which means a new role can be introduced without adding a vocabulary entry. That is a more extensible design, and it is why the format is shared with several other model families rather than being Microsoft’s alone.

The Phi-4-mini line reverted to <|system|>-style tags and added tool-calling tokens on top, so “the Phi template” is not one thing across the family. This is a large part of why the end-of-turn token goes out of sync with serving frameworks — see why a local Phi-3 server misses the stop token. The rule that survives every release is: read the template out of the checkpoint you loaded, and never out of a blog post.