Gemma's Chat Template: the start_of_turn Format

Gemma instruction-tuned checkpoints were fine-tuned on one specific text format. Feed them anything else and they still answer, usually well enough that you do not notice the problem until quality is oddly poor or the model will not stop. This is that format, in full.

The shape of one turn

A turn is a marker, a role word, a newline, the content, and a terminator:

<start_of_turn>ROLE
CONTENT<end_of_turn>

There are exactly two role words: user and model. Not assistant. That is the first thing people get wrong when porting a prompt from an OpenAI-shaped template, and it is a silent error rather than a loud one, because the model has seen enough text to guess what you meant.

There is no third role. Gemma’s reference template has no system turn at all, which is a large enough problem on its own that it has its own page.

A rendered two-turn conversation

This is what the model actually receives for a two-exchange conversation, with the beginning-of-sequence token shown and newlines significant:

<bos><start_of_turn>user
What is the boiling point of water at 3000 m?<end_of_turn>
<start_of_turn>model
About 90 degrees Celsius, because atmospheric pressure falls with altitude.<end_of_turn>
<start_of_turn>user
And at 5000 m?<end_of_turn>
<start_of_turn>model

Note the last two lines. The final <start_of_turn>model and its newline are present but the content is not; that trailing fragment is the cue that it is the model’s turn to speak. It is what add_generation_prompt=True appends.

The whitespace is not decorative. There is a newline after the role word and before the content, and a newline after <end_of_turn> and before the next <start_of_turn>. There is no blank line between turns and no space between the marker and the role. Every one of those is a token-level difference from what the model was tuned on, and while a single extra newline will not break a Gemma checkpoint outright, a template that differs in several places at once produces exactly the kind of mild, unattributable quality loss that people spend a week blaming on the model.

What each token is doing

• <bos> — beginning of sequence, added once at the very front and never again. Gemma’s tokenizer adds it automatically when it encodes text, which is the source of a common double-BOS bug covered on the duplicated-token page.
• <start_of_turn> — opens a turn. It is a single token in the vocabulary, not five characters the tokenizer splits up, which is why writing the literal string into your prompt text does not reliably produce the same token.
• user / model — the role, as ordinary text on the same line as the opening marker, followed by a newline before content begins.
• <end_of_turn> — closes a turn. This is the token the instruction-tuned models were trained to emit when they are finished, and it is the one your stop condition needs to watch, not <eos>.

The special tokens are real vocabulary entries with fixed ids in the released tokenizers, and you can confirm the ids for your checkpoint rather than trusting a number in an article:

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("google/gemma-2-9b-it")
for t in ["<bos>", "<eos>", "<start_of_turn>", "<end_of_turn>"]:
    print(t, tok.convert_tokens_to_ids(t))

Two of those four have a habit of being confused with each other. <eos> is the tokenizer’s nominal end-of-sequence token and it is not what an instruction-tuned Gemma emits at the end of a reply; <end_of_turn> is. A serving stack that stops only on the first will run past the end of the answer and begin writing the user’s next question. That is common enough to have a page of its own, and it is the single most consequential thing to take away from the token list above.

The base checkpoints, the ones without -it in the name, were not tuned on this format at all. They have the special tokens in their vocabulary because the tokenizer is shared, and they have no trained behaviour attached to them: a base model given a <start_of_turn>model cue will continue the document rather than answer, and will not reliably emit <end_of_turn> to stop. If you are applying a chat template to a base checkpoint, the template is not the problem.

The generation prompt

The distinction between a conversation and a prompt is the trailing generation cue, and getting it wrong produces two distinct symptoms. Omit it and the model, having just seen a completed <end_of_turn>, is as likely to open a new <start_of_turn>user and write your next question for you as it is to answer. Include it twice and you have an empty turn in the transcript.

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("google/gemma-2-9b-it")
messages = [
    {"role": "user", "content": "What is the boiling point of water at 3000 m?"},
    {"role": "model", "content": "About 90 degrees Celsius."},
    {"role": "user", "content": "And at 5000 m?"},
]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(repr(text))

Printing with repr rather than plain print is the point of that snippet. Template bugs in Gemma are almost always whitespace bugs, and a bare print hides the difference between one newline and two.

The same flag has a second use that is easy to overlook. Setting add_generation_prompt=False is what you want when you are building training data rather than a request, because the target text the model should learn is the completed model turn including its terminator. Rendering training examples with a generation prompt attached is a quiet way to teach a fine-tune the wrong thing, and the symptom appears much later as a model that emits an extra turn header before answering.

Building it by hand, and when not to

Most of the time you should not build it by hand. The template ships inside tokenizer_config.json as a Jinja string under chat_template, which means it travels with the checkpoint and gets corrected when Google corrects it. String-concatenating the format yourself freezes a copy of it in your codebase, and copies go stale silently.

There are two situations where you have no choice. One is a serving stack that takes raw text and does its own tokenization, where you need the exact string. The other is a framework whose bundled template disagrees with the checkpoint’s, which happens after a template revision. In both cases, render with apply_chat_template at build time and diff the output against what your server is actually sending; the mismatch is usually one newline or one stray <bos>.

One more constraint applies if you are constructing turns yourself: the roles must alternate. The trained format is user, model, user, model, and Gemma’s bundled template enforces it, raising rather than rendering two consecutive turns with the same role. Message lists assembled by an agent loop violate this constantly — two tool results in a row, or a retry that appends a second user message — so the fix belongs in your message assembly rather than in the template. Merge consecutive same-role messages into one turn, separated by a blank line, before you render.

Finally, treat the template as a versioned artefact. It lives in tokenizer_config.json, it has been corrected upstream more than once across this family, and a correction changes the literal text your model receives. Hashing the template string at start-up and logging the hash costs nothing and turns “output quality changed and nobody deployed anything” into a one-line diagnosis. The revision-pinning walkthrough covers how to stop it changing at all.