Why Some Inference Servers Double-Emit Gemma's Stop Token

Gemma responses that end with a visible <end_of_turn>, responses that carry on and start writing the user’s next question, and responses that begin with a duplicated marker all get reported as the same bug. They are three different mismatches between the checkpoint and the thing serving it.

Three symptoms, three causes

• The response never ends. The model finishes its answer, emits <end_of_turn>, and keeps going with <start_of_turn>user and a question you did not ask, until the token limit stops it. Cause: the server stops only on <eos>.
• The marker is visible in the text. The answer is correct and ends with a literal <end_of_turn> in the string your application shows. Cause: the token is being decoded rather than treated as a stop, or special tokens are not being skipped on decode.
• A warning about a duplicated BOS, or a first token that looks doubled. Cause: the chat template already contains <bos> and the tokenizer adds another when it encodes the rendered string.

They share a root: Gemma’s special-token handling is split between the chat template, the tokenizer config and the generation config, and a serving stack has to honour all three.

The name people give the problem, a doubled stop token, is usually a description of the symptom rather than the mechanism. Genuine duplication does happen, and it happens when your prompt already ends with a turn terminator that the template also appends, so the model sees two and the transcript acquires an empty turn. Far more often the token is not duplicated at all: it is emitted once, correctly, and then either ignored by the stop logic or printed by the decoder. Working out which of those you have is the whole of the fix, and it takes one log line.

Gemma has two end tokens

The tokenizer’s nominal eos_token is <eos>. The token the instruction-tuned models were actually trained to emit at the end of a reply is <end_of_turn>. Both are real vocabulary entries, and the checkpoint’s generation config usually lists both as stop conditions:

from transformers import AutoTokenizer, GenerationConfig

tok = AutoTokenizer.from_pretrained("google/gemma-2-9b-it")
gc  = GenerationConfig.from_pretrained("google/gemma-2-9b-it")

print("eos_token       :", tok.eos_token, tok.eos_token_id)
print("end_of_turn id  :", tok.convert_tokens_to_ids("<end_of_turn>"))
print("generation eos  :", gc.eos_token_id)     # expect a LIST of ids

The third line is the diagnostic. If gc.eos_token_id is a list and your server is passing a single stop id, you have found the runaway-generation bug. Many inference servers were written against models with exactly one end token and treat eos_token_id as a scalar; the list silently collapses to its first element, which for Gemma is the wrong one for chat.

Where you cannot fix the server, you can usually add a stop string. Almost every serving API accepts one, and the string form works even where the id list is being mishandled:

{
  "model": "gemma-2-9b-it",
  "messages": [{"role": "user", "content": "Explain sliding window attention."}],
  "max_tokens": 512,
  "stop": ["<end_of_turn>"]
}

Why does the family have two end tokens at all? Because they mean different things. <eos> ends a document, and it is what the base checkpoints learned during pretraining. <end_of_turn> ends a turn inside a conversation that continues, and it was introduced by the instruction tuning. A chat model needs to end its turn without declaring the document finished, so both exist and only one of them is the right stop condition for chat. The turn structure they delimit is described on the chat-template page. Frameworks written before multi-token stop sets were common tend to assume the tokenizer’s eos_token is authoritative, and on Gemma it is not.

The doubled beginning-of-sequence token

Gemma’s chat template renders a leading <bos>, and its tokenizer is configured to add a BOS token when it encodes. Do both and the model sees two. This is the single most common Gemma integration bug, because the natural two-step is exactly wrong:

# WRONG: template adds <bos>, then the tokenizer adds another.
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
ids  = tok(text, return_tensors="pt")            # add_special_tokens defaults to True

# RIGHT (option 1): let apply_chat_template do the tokenizing.
ids = tok.apply_chat_template(messages, tokenize=True, add_generation_prompt=True,
                              return_tensors="pt")

# RIGHT (option 2): render to text, then encode without special tokens.
ids = tok(text, add_special_tokens=False, return_tensors="pt")

Confirm it rather than assuming, since the effect on output is subtle rather than catastrophic:

print(tok.convert_ids_to_tokens(ids["input_ids"][0][:4]))
# want: ['<bos>', '<start_of_turn>', 'user', ...]
# bug : ['<bos>', '<bos>', '<start_of_turn>', ...]

The symptom of a doubled BOS is not a crash. Recent transformers versions warn about it, but plenty of stacks do not, and the model simply produces slightly worse output than it should: a sequence beginning with two BOS tokens is a prefix it never saw in training, and everything downstream is conditioned on it. That is why this bug survives so long in codebases. Nothing is broken enough to investigate, and the four-token print above is the cheapest test in this entire cluster.

When the marker appears in your output

If <end_of_turn> is showing up in the string you display, generation stopped correctly and decoding is wrong. Two causes, and they are distinguishable.

The first is decoding with special tokens included. Pass skip_special_tokens=True and slice off the prompt so you decode only the new tokens:

out = model.generate(**inputs, max_new_tokens=512)
new = out[0][inputs["input_ids"].shape[-1]:]
print(tok.decode(new, skip_special_tokens=True))

The second is a stop string matched after the fact. A server that stops on the string <end_of_turn> rather than on the token id may include the matched text in the final chunk, and whether it does is a per-implementation choice. If you cannot change it, strip the marker at the boundary of your application rather than in twenty places downstream.

A related case worth naming: a GGUF or other quantised repack carries its own copy of the template and token metadata. An older conversion can disagree with the current upstream repository entirely, so the same model file that works in one runtime leaks markers in another. Re-converting from a known revision is usually faster than debugging the metadata.

Streaming deserves its own mention, because it is where a leaked marker most often reaches a user. A streaming decoder emits text chunk by chunk, and the stop token arrives in the last chunk. If your special-token filtering happens on the final assembled string rather than per chunk, the marker has already been sent to the browser by the time you strip it. Filter on the way out, per chunk, and stop forwarding at the first stop token rather than after it.

Fixing it in each layer

1. Establish which symptom you have. Log the raw token ids of the last few generated tokens, not the decoded string. The string cannot tell you whether the stop token was emitted and decoded or never emitted at all.
2. Check the stop set the server is using. Compare it with gc.eos_token_id from the checkpoint. A single id where the config has a list is the bug.
3. Add the stop string as a belt-and-braces measure. Passing stop: ["<end_of_turn>"] costs nothing and covers the case where the id set is wrong.
4. Fix the BOS duplication at the tokenization boundary. Either tokenize inside apply_chat_template or encode with add_special_tokens=False. Never both paths in one codebase.
5. Decode with special tokens skipped, and slice the prompt off first. This removes the leaked-marker class entirely.
6. Pin the revision so the template and generation config you debugged against are the ones you deploy with.