Why a Local Phi-3 Server Sometimes Misses the Stop Token

The model answered correctly and then kept going, inventing a follow-up question and answering that too. Nothing is wrong with the weights. Phi-3 has two plausible end-of-turn tokens and your server only knows about one of them.

The symptom

One of these, depending on whether your decoder strips special tokens:

Paris is the capital of France.<|end|><|assistant|>
Would you like to know its population?<|end|><|assistant|>
The population of Paris is approximately...

--- or, with skip_special_tokens=True ---

Paris is the capital of France.
Would you like to know its population?
The population of Paris is approximately...

The tell is that generation runs to exactly your token limit every time — 512 tokens, 1024 tokens, whatever max_new_tokens or num_predict is set to — and that the tail reads like a transcript of a conversation nobody had. The answer at the top is fine. It is what comes after it that is the bug, and you are paying latency and compute for all of it.

Two end tokens, one eos_token field

Phi-3’s instruct checkpoints carry two tokens that mean something like “stop”, and they are not interchangeable:

• <|endoftext|>, id 32000 on the 32K-vocabulary sizes. The tokenizer’s declared eos_token, inherited from the pretraining setup, and emitted at the end of a whole document.
• <|end|>, id 32007. What the instruct chat template puts at the end of every turn, and therefore what the model was tuned to emit when it has finished answering. See Phi-3’s chat template for where it sits.

The model does the right thing: it emits <|end|>. The failure is on the reading side. A stop condition is a set of token ids the runtime watches for, and where that set comes from depends on the runtime:

• generation_config.json can list several, as "eos_token_id": [32000, 32007]. Runtimes that read this file stop correctly.
• tokenizer_config.json has a single eos_token. A runtime that reads only this one watches for 32000, never sees it, and generates until it runs out of budget.
• A GGUF file carries one tokenizer.ggml.eos_token_id metadata value. Whichever id the conversion script chose is the only one baked in — and conversions made from an early revision of the repository, before the list form was added, chose 32000.

That last point is why this reappears long after it was “fixed”: the fix was a commit to the model repository, and a quantised GGUF someone converted before it, or a cached download from before it, still has the old behaviour. Pinning revisions makes this visible rather than mysterious — see pinning a Phi checkpoint.

Diagnose it in four lines

from transformers import AutoTokenizer, GenerationConfig

tok = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")
print(tok.eos_token, tok.eos_token_id)                       # <|endoftext|> 32000
print(tok.convert_tokens_to_ids("<|end|>"))                  # 32007
print(GenerationConfig.from_pretrained(
    "microsoft/Phi-3-mini-4k-instruct").eos_token_id)         # [32000, 32007] or 32000

If the third line prints a bare integer rather than a list containing the <|end|> id, you have found it. Also decode a raw generation with skip_special_tokens=False — seeing <|end|> in the output confirms the model emitted it and the runtime ignored it, which distinguishes this from the causes in the last section.

The fix, per runtime

transformers

eos_ids = [tok.eos_token_id, tok.convert_tokens_to_ids("<|end|>")]

out = model.generate(
    **inputs,
    max_new_tokens=512,
    do_sample=False,
    eos_token_id=eos_ids,     # a list is accepted
    pad_token_id=tok.eos_token_id,
)

vLLM and any OpenAI-compatible server

from vllm import SamplingParams
params = SamplingParams(max_tokens=512, temperature=0, stop_token_ids=[32007])

# over HTTP, by string:
curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "microsoft/Phi-3-mini-4k-instruct",
  "messages": [{"role": "user", "content": "Capital of France?"}],
  "max_tokens": 512,
  "stop": ["<|end|>", "<|user|>", "<|endoftext|>"]
}'

Prefer stop_token_ids to stop where you can. String stop sequences are matched against decoded text, so they are sensitive to how the detokenizer handles special tokens; an id match is exact. Include <|user|> in the string list as a belt-and-braces measure — if the model gets past the end token it will open a user turn next, and that catches it one token later.

llama.cpp and Ollama

# Ollama Modelfile
FROM ./Phi-3-mini-4k-instruct-q4_k_m.gguf
TEMPLATE """{{ if .System }}<|system|>
{{ .System }}<|end|>
{{ end }}<|user|>
{{ .Prompt }}<|end|>
<|assistant|>
"""
PARAMETER stop "<|end|>"
PARAMETER stop "<|user|>"
PARAMETER stop "<|endoftext|>"

For llama.cpp directly, pass the same strings in the "stop" array to llama-server, or inspect and override the baked metadata with --override-kv tokenizer.ggml.eos_token_id=int:32007 if you are stuck with an old conversion you cannot rebuild. Re-converting from a current revision is the better fix where it is available to you.

One thing that is not a fix: skip_special_tokens=True. It hides <|end|> from the decoded string while the model carries on generating behind it. The output looks tidier, the hallucinated extra turns are still there, and you are still paying for every token.

Verify it, and keep it verified

The fix is easy to apply and easy to lose, because nothing in your test suite is likely to notice its absence. A functional test that asks a question and checks the answer contains “Paris” passes perfectly well while five hundred junk tokens follow it.

Assert on the stop condition itself, which every runtime reports:

# transformers: the last generated id should be a stop token,
# and the length should be well under the cap
gen = out[0][ids.shape[-1]:]
assert gen[-1].item() in eos_ids, "generation hit the length cap, not a stop token"
assert len(gen) < 512

# OpenAI-shaped servers: read finish_reason
resp = client.chat.completions.create(model=MODEL, messages=msgs, max_tokens=512)
assert resp.choices[0].finish_reason == "stop"   # "length" means it never stopped

That second assertion is the one to put in a permanent test. A finish_reason of "length" on a question whose answer is one sentence is this bug, always, regardless of which of the causes below produced it. It is also worth reporting as a metric in production: a rising share of length-terminated completions is the earliest signal that a checkpoint, a template or a GGUF has changed underneath you.

The cost of not noticing is not only cosmetic. Every junk token is latency the user waits through and compute you paid for, and on a request billed by output tokens through a hosted route, it is a bill that is several times larger than it should be for output nobody reads.

Three other causes with the same symptom

• You loaded the base model. A non-instruct Phi-3 checkpoint never saw <|end|> in training and will not emit it. It continues text, because that is the only thing it was trained to do. Check the repository name for -instruct.
• Your prompt is not in the template. If the rendered prompt lacks the role tags — because you concatenated strings, or because you posted to /v1/completions instead of /v1/chat/completions — the model has no reason to think it is in a turn-taking conversation, and a turn-ending token is an unlikely continuation.
• You applied the wrong family’s template. ChatML tags on Phi-3, or Phi-3 tags on Phi-4, produce a prompt that looks structured and matches nothing the model was tuned on. The answer is usually still coherent, which is what makes this one take an afternoon.