Qwen's ChatML Chat Template

Qwen uses ChatML, the same turn format OpenAI published for its early chat models. It is three special tokens and a strict newline convention, and getting the newlines wrong is a real failure with a quiet symptom.

A rendered prompt, in full

Here is what tokenizer.apply_chat_template produces for a two-message conversation on a Qwen2.5 instruct checkpoint, with add_generation_prompt=True. Newlines are significant, so this is reproduced literally:

<|im_start|>system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>
<|im_start|>user
What is the capital of Portugal?<|im_end|>
<|im_start|>assistant

Read that as a repeating unit. Each turn is <|im_start|>, then the role name, then a newline, then the content, then <|im_end|>, then a newline. No space after the start token, no newline before the end token. The final block is the same unit truncated: the start token, the role assistant, a newline, and then nothing — that is the model’s cue that it is the one speaking now.

You can produce this yourself rather than taking it on trust, which is the right habit for any checkpoint you have not used before:

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
messages = [{"role": "user", "content": "What is the capital of Portugal?"}]

print(tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True))

Note that the system message appears in the output even though the message list did not contain one. That injection is the template’s doing and it is specific to the Qwen2.5 generation; the default system prompt has its own page because the behaviour changed in Qwen3.

The three special tokens

ChatML’s delimiters are single tokens in Qwen’s vocabulary, not multi-character strings the model has to parse. In the added_tokens.json of the Qwen2 and Qwen2.5 repositories they carry fixed ids:

• <|endoftext|> — id 151643. The pretraining document separator, and the EOS token of the base checkpoints.
• <|im_start|> — id 151644. Opens a turn. Never generated by a well-behaved instruct model in the middle of an answer.
• <|im_end|> — id 151645. Closes a turn, and is the EOS token of the instruct checkpoints.

That the base and instruct checkpoints have different EOS tokens is the root of the most common Qwen serving bug, and it has its own page on why generation runs past the stop token. Do not memorise the ids — resolve them, because a fine-tune can extend the vocabulary and shift them:

tok.convert_tokens_to_ids(["<|endoftext|>", "<|im_start|>", "<|im_end|>"])

add_generation_prompt and the trailing newline

add_generation_prompt is the argument that decides whether the rendered string ends with an open assistant turn. It matters in two opposite directions.

When you are asking for a completion, it must be True. Without it the string ends after the user’s <|im_end|>, and the model is being asked to continue a transcript at a turn boundary with no indication of whose turn it is. Models handle this by guessing, which usually means emitting <|im_start|>assistant themselves — so the answer arrives, wrapped in a control token your parser was not expecting, and everything downstream that strips special tokens now silently eats the first line.

When you are building training data or computing the log-probability of a known assistant response, it must be False, because the assistant turn is already in the message list and adding a second opener corrupts the sequence.

The trailing newline after assistant is part of the generation prompt and not decorative. The model has only ever seen content begin on the line after the role name. Omit it and the first token is being sampled in a context the model never saw during post-training; the usual symptom is a leading space or a stray newline at the start of every reply, which people then strip in application code without ever finding the cause.

Which roles the template accepts

Qwen2.5’s template handles four roles, and only the first three render the way you would guess:

• system — rendered as a turn like any other. There is no separate mechanism for it, which is why a system message costs exactly the tokens it contains and why multiple system messages render as multiple turns rather than being merged or rejected.
• user and assistant — the conversational alternation.
• tool — not rendered as a <|im_start|>tool turn. The template folds tool results into a user turn wrapped in <tool_response> tags, which is the Hermes convention Qwen adopted. This surprises people often enough that it is covered in the page on Qwen’s tool-call format.

Consecutive messages with the same role are rendered as separate turns, not merged. Qwen does not raise the alternation error that some other families raise, so a bug that produces two consecutive user messages shows up as slightly odd output rather than as a 400.

Four ways hand-built prompts break

If you are formatting prompts by hand — because you are calling a raw completions endpoint, or building a dataset, or working in a language with no Jinja templating — these are the four failures in rough order of frequency.

• A space after the role name. <|im_start|>assistant with a trailing space instead of a newline. Tokenises differently from what the model trained on, and the effect is a small persistent quality loss with no error anywhere.
• A newline before <|im_end|>. The template puts the end token immediately after the content. An extra newline is an extra token in a position the model has not seen.
• Passing the special tokens as text. If you concatenate the string yourself and then tokenise with add_special_tokens semantics that split them, you get the literal characters rather than ids 151644/151645. Verify by decoding: the rendered prompt should tokenise to a sequence containing 151644, not a run of a dozen punctuation tokens.
• Reusing a Llama or Mistral template. They are not interchangeable in either direction. Llama 3 uses <|start_header_id|> and <|eot_id|>, which are simply absent from Qwen’s vocabulary and will tokenise as gibberish. See the Llama 3 template tokens for the contrast.

The general rule that avoids all four: never hand-write the template if a tokeniser is available. apply_chat_template reads the Jinja template shipped in tokenizer_config.json alongside the weights, so it is correct by construction for the exact checkpoint you have, including any fine-tune that changed it.