Why a Qwen Model Keeps Generating Past Its Stop Token

You asked Qwen a question, got a perfectly good answer, and then it kept going — inventing your next question, answering that too, and running until it hit the token limit. The answer is correct and everything after it is the model continuing a transcript, because nothing told the runtime to stop.

The symptom

The output looks like this, with the fabricated part starting exactly where a real conversation would have ended:

The capital of Portugal is Lisbon.
<|im_end|>
<|im_start|>user
What is the capital of Spain?<|im_end|>
<|im_start|>assistant
The capital of Spain is Madrid.
...

Sometimes the control tokens are visible, as above. Sometimes they are not, because the runtime is decoding with special tokens skipped, and then you get the same runaway transcript with no delimiters at all — just an answer followed by a plausible-looking follow-up question the user never asked. The second form is harder to recognise and easier to mistake for the model being chatty.

The tell either way is that generation stops only at your max_tokens, and that finish_reason comes back as length on every request rather than stop. If you are seeing length on a one-line answer, the model is not being verbose — it is not being stopped.

Two EOS tokens, one config

Qwen’s base and instruct checkpoints end sequences with different tokens, and this is the root of almost every instance of this problem.

• The base checkpoints were pretrained on documents separated by <|endoftext|> (id 151643). That is their EOS.
• The instruct checkpoints were post-trained on ChatML turns terminated by <|im_end|> (id 151645). That is what they emit at the end of an answer, and generation_config.json in those repositories lists both ids as terminators.

So the instruct model is doing its job. It emits <|im_end|> at exactly the right moment. The failure is that whatever is running the decode loop does not have 151645 in its set of stop ids, so the token is sampled, appended to the sequence, and the loop continues. Having just written a turn terminator, the highest-probability continuation is the start of a new turn — which is why the model then writes your side of the conversation. It is not confused; it is completing a transcript, which is the only thing it ever does.

The reason the configuration gets lost is that it lives in a different file from the one people copy. config.json carries the architecture; generation_config.json carries the EOS ids. A quantisation script, a GGUF conversion, a merge of a LoRA adapter, or a hand-written serving manifest can faithfully reproduce the first and drop the second. The weights are then perfect and the model never stops.

Diagnosing it in four steps

1. Confirm the model is actually emitting the token. Request the raw ids, or decode with skip_special_tokens=False. If <|im_end|> appears in the output at the right place, the model is fine and this is entirely a serving problem. If it never appears, skip to the last section.
2. Check you are on an instruct checkpoint. A surprising share of these reports are a base model being served with a chat template. Base checkpoints were never trained to emit <|im_end|> and no stop configuration will make them; the repository name ends in -Instruct or it does not.
3. Read the generation config that the runtime loaded, not the one in the original repository:

   from transformers import GenerationConfig
   print(GenerationConfig.from_pretrained("/path/to/your/local/model"))
   # expect eos_token_id to include 151645

4. Check the chat template rendered a generation prompt. A prompt that does not end with <|im_start|>assistant and a newline puts the model at a turn boundary with no role assigned, and it will open a turn itself before answering — which produces output that looks like this problem but is caused at the other end. See the ChatML template page for what the rendered prompt should look like.

The fix, per serving stack

In all cases the fix is the same idea — put <|im_end|>into the terminator set — and differs only in where that set is configured.

Transformers. Pass the terminators explicitly rather than relying on the loaded config:

terminators = [
    tok.convert_tokens_to_ids("<|im_end|>"),
    tok.convert_tokens_to_ids("<|endoftext|>"),
]
out = model.generate(**inputs, max_new_tokens=512, eos_token_id=terminators)

vLLM and other OpenAI-compatible servers. Send stop as a request parameter, which works regardless of what the server loaded:

{
  "model": "Qwen2.5-7B-Instruct",
  "messages": [{"role": "user", "content": "What is the capital of Portugal?"}],
  "max_tokens": 512,
  "stop": ["<|im_end|>", "<|endoftext|>"]
}

This is worth doing belt-and-braces even on a correctly configured server, because it costs nothing and survives a model swap. Note the difference between stop and stop_token_ids where a server offers both: string stops are matched against decoded text and are removed from the output, id stops are matched against sampled tokens and are the more reliable of the two for special tokens.

llama.cpp and GGUF. This is where it most often goes wrong, because the EOS id is baked into the GGUF metadata at conversion time and a conversion that guessed wrong is not repairable by editing a JSON file. Inspect it, and override at load time if needed:

llama-cli -m qwen2.5-7b-instruct-q4_k_m.gguf \
  --override-kv tokenizer.ggml.eos_token_id=int:151645 \
  -p "..."

Ollama and other Modelfile-based runners. The stop strings are declared in the Modelfile. A community-published model whose Modelfile omits PARAMETER stop "<|im_end|>" will show exactly this behaviour; add the line and rebuild.

When it is not the EOS token

If step one showed the model never emitting <|im_end|> at all, the stop configuration is not your problem and there are three other candidates.

• A repetition loop. Greedy decoding on a long generation can enter a cycle whose highest-probability continuation returns to its own start, and with no sampling randomness nothing breaks it. The model genuinely never reaches a stopping point. Raise the temperature off zero, or set a repetition penalty — see the repetition penalty parameter.
• A fine-tune that trained the token away. If the training data was built without <|im_end|> at the end of each assistant turn, or with the loss masked over it, the model has been taught not to emit it. This is a data bug and no serving flag fixes it.
• A completions endpoint instead of a chat endpoint. Sending a ChatML-formatted string to a raw text-completion route usually bypasses the chat stop configuration entirely. The behaviour is the same class of problem described in why Llama ignores a stop sequence, and the diagnosis is the same: find out which stop set the request path is actually using.

One thing not to do: leave max_tokens low as a workaround. Capping generation hides the runaway without fixing it, and the cost is that every genuinely long answer is now truncated mid-sentence with the same length finish reason you have trained yourself to ignore.