Special Tokens, Chat Templates and Silent Formatting Bugs

You send a list of role-and-content objects. The model receives a flat string full of markers it was trained to expect. Everything in between is the chat template, and when it is wrong the model does not error — it just gets worse, which is a far harder thing to debug.

The gap between messages and text

A base model has no concept of a conversation. It continues text. Chat behaviour is manufactured during instruction tuning by training on a specific serialisation — particular delimiters, in a particular order, with particular whitespace — and at inference time you must reproduce that serialisation exactly. Deviate and you are prompting the model slightly outside the distribution it was tuned on.

For hosted APIs the provider does this for you and the whole subject is invisible. The moment you run open weights yourself, or fine-tune, or use a completions endpoint instead of a chat endpoint, it becomes your problem — and it is a problem that manifests as “this model is worse than the benchmarks suggested”.

It is worth being clear about how much of the model’s behaviour rides on these markers. The end-of-turn token is what stops generation; if the template you applied uses a different one, the model will keep writing past the point it meant to stop and invent the user’s next message. The role headers are what distinguish an instruction you wrote from text a user pasted, which makes them the only structural defence the model has against prompt injection. And the tool-call blocks, in families that have them, are a template concern too — a runtime that renders tool definitions in a shape the checkpoint was not tuned on gets malformed calls and blames the model.

What the scaffolding looks like

Two widely-copied shapes. ChatML, which originated with OpenAI and has been adopted by many open models:

<|im_start|>system
You are a terse assistant.<|im_end|>
<|im_start|>user
What is a token?<|im_end|>
<|im_start|>assistant

And the Llama 3 header format, which uses a different set entirely:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a terse assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>

What is a token?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Note the trailing fragment in both: the string ends having opened an assistant turn but written nothing. That is the generation prompt, and it is the thing that tells the model it is its turn to speak. Note also the blank lines in the Llama 3 form — the double newline after each header is part of the format, not formatting, and it tokenizes differently from a single newline.

You should almost never write these by hand. Every model repository ships a Jinja template in the chat_template field of tokenizer_config.json, and the tokenizer will apply it:

prompt = tok.apply_chat_template(
    messages,
    tokenize=False,             # give me the string so I can look at it
    add_generation_prompt=True, # append the opening of the assistant turn
)

Five bugs, in order of how often they happen

• Missing generation prompt. Omit add_generation_prompt=True and the string ends with a closed user turn. The model, quite reasonably, continues the user’s message — you get another question instead of an answer, or a rambling continuation. Extremely common and instantly recognisable once you have seen it.
• Double BOS. apply_chat_template already emits the beginning-of-text token. Feeding its output into tok(prompt) with default settings adds a second one, because add_special_tokens defaults to true. Two BOS tokens is an input the model never saw in training and quality drops measurably. Pass add_special_tokens=False when tokenizing an already-templated string, or use tokenize=True and skip the second step.
• The wrong template entirely. Applying Llama 2’s [INST] … [/INST] wrapper to a Llama 3 checkpoint produces output that is coherent, on-topic and noticeably stupider. Nothing errors. Always take the template from the checkpoint you are loading, never from a blog post about the previous generation.
• Unsupported system role. Some templates have no system slot and silently fold system content into the first user message; others raise. Either way your carefully-tuned system prompt is not occupying the position you think it is.
• Stray whitespace. A trailing space after the generation prompt changes the first token the model must produce, because in byte-level BPE the leading space is part of the following token. It is a real and reproducible quality difference from an invisible character.

Special tokens as an injection surface

Special tokens are entries in the vocabulary, and if user-supplied text is tokenized in a mode that recognises them, a user who types <|im_end|> can close the turn and open a new one with a role of their choosing. That is prompt injection at the tokenizer level, and it bypasses every instruction you wrote in the system prompt because it operates below them.

The mitigation is to encode untrusted content with special-token parsing disabled, so those strings become ordinary text. In tiktoken this is the default and enabling it is explicit — encode raises on disallowed special tokens unless you opt in, which is a good default worth preserving. In the Hugging Face tokenizers the equivalent control is split_special_tokens. Either way: never concatenate user text into a template string yourself.

The one-line diagnostic

Every bug on this page is visible in the same place, and almost nobody looks:

print(repr(tok.apply_chat_template(messages, tokenize=False,
                                   add_generation_prompt=True)))

# then, to see how it actually tokenizes:
ids = tok.apply_chat_template(messages, add_generation_prompt=True)
print(tok.convert_ids_to_tokens(ids)[:12], "...", tok.convert_ids_to_tokens(ids)[-6:])

Use repr, not print, so the newlines and trailing spaces are visible. Then read the first dozen and last half-dozen tokens: one BOS, the expected header sequence, and an open assistant turn at the end. Ten seconds, once, and this entire class of bug stops being mysterious. It is also where you will find the scaffolding tokens that Make it a test, not a habit. A single assertion that the templated string starts with exactly one beginning-of-text token and ends with an open assistant turn will catch four of the five bugs above, runs in milliseconds, and fails loudly on the day somebody upgrades the tokenizer library or swaps the checkpoint. It is the cheapest test in an inference codebase and almost nobody has it. The same scaffolding it prints is what makes your count differ from the provider’s, which is the subject of the count mismatch page.