Why Llama Sometimes Ignores a Stop Sequence Mid-Generation

You passed stop: ["\n\n"] and the model wrote four more paragraphs. Nothing malfunctioned: stop sequences are not something the model does. They are a string comparison in the server, and there are about six specific ways for that comparison never to run.

Two mechanisms, two places

Everything on this page follows from the fact that there are two unrelated ways generation ends, and they are implemented at different layers.

• Token-level. The sampler picks a token whose ID is in the end-of-sequence set. The loop stops because the model chose to stop. This is what makes an answer end naturally.
• String-level. The server decodes what has been generated so far and looks for one of your stop strings in the text. If it finds one, it halts the loop and truncates the output at the match. The model never knew about it and would have kept going.

The model is not aware of a stop string in any sense — it does not avoid it, aim for it, or treat it as an instruction. Asking for stop: ["END"] does not encourage the model to write END. If it never emits the string, the stop sequence never does anything, and that is the first thing to rule out.

Token-level stopping

For a Llama 3 Instruct checkpoint the relevant end-of-turn token is <|eot_id|>, ID 128009, and it is distinct from the base end-of-text token <|end_of_text|>, ID 128001. Which of these the runtime honours comes from generation_config.json in the checkpoint:

"eos_token_id": [128001, 128009]

A checkpoint or a serving config that lists only 128001 produces the most dramatic version of this bug: the model finishes its answer, emits <|eot_id|>, the loop does not stop, and it carries on by hallucinating the user’s next question and answering that too. If your output contains an invented dialogue, this is the cause, and it is a token-level problem that no stop string will fix reliably. The full story is in eot_id versus end_of_text.

Servers expose this separately from string stops. vLLM takes stop_token_ids as a list of integers; llama.cpp derives its set from the GGUF metadata; Ollama takes it from the Modelfile. And if you are using the built-in tool convention, <|eom_id|> belongs in this set as well, or tool calls run straight past their own terminator.

String-level stopping

The server holds the decoded text so far and, after each step, checks whether any stop string appears in it. Two design details of that check explain most of the confusion:

It happens after detokenization, so it is not token-aligned. This is good news — a stop string that falls across a token boundary is still found, because the comparison is on characters. It is also why a stop string can never be “missed” for being unaligned, contrary to a common explanation.

It happens after special tokens are skipped. Most servers decode with skip_special_tokens=True, so control tokens are not in the text being searched. Passing stop: ["<|eot_id|>"] therefore matches nothing — the token was removed before the comparison. That belongs in stop_token_ids, not in stop. This one mistake accounts for a large share of “my stop sequence is ignored” reports.

Whether the matched text appears in your output is a separate flag. On vLLM, include_stop_str_in_output defaults to false, so the output is cut before the match. That produces the opposite symptom — you see the stop working and conclude it did not, because the string you expected to see is missing.

Why yours is not firing

• The model never emitted it. Check by removing the stop parameter and reading the raw output. If the string is not there, the fix is prompting, not parameters.
• Whitespace does not match. The comparison is exact. "\n\n" does not match "\n \n", and "Observation:" does not match " Observation:". Prefer the shortest distinctive fragment over a long exact phrase.
• It is a special token in disguise. Covered above: use the ID.
• The parameter did not arrive. Ollama takes stops in options.stop, not at the top level; a Modelfile PARAMETER stop and a request-level list interact in ways worth checking. A misplaced field is ignored silently by most servers rather than rejected.
• You exceeded the server’s limit on how many. Several implementations cap the number of stop sequences — four is a common ceiling — and behaviour past the cap ranges from an error to quietly dropping the extras.
• It is a chat model and you are fighting the template. An Instruct checkpoint already ends its turn with <|eot_id|>. If you are adding string stops to control output shape, a structured output constraint is a sturdier tool than a stop string.

There is one more, specific to streaming. The server must buffer text that could be the beginning of a stop string, because it cannot un-send a chunk it has already flushed. Implementations differ in how much they hold back, so a long stop string can produce a visible stutter, and a client doing its own matching on already-rendered chunks will always be a step late. Do the matching on the server side, with the stop parameter, rather than in your stream handler — see the streaming response shapes.

What to use instead

A stop sequence is a blunt instrument: it cuts a stream at a string match and hopes the text before the cut is well formed. For most of the jobs people give it, something else is sturdier.

• To get a bounded answer — use max_tokens. It cannot be missed, and the finish reason tells you unambiguously that it fired.
• To get structured output — use a grammar or schema constraint rather than stopping at a delimiter. vLLM and llama.cpp both support constrained decoding, which makes the invalid tokens unsamplable instead of trimming the text afterwards. A JSON object that ends because the grammar closed it is complete; a JSON object that ends because a stop string matched is frequently truncated mid-value.
• To end a turn — use the model’s own end token, correctly configured. That is what it is for, and it fires whether or not you remembered to pass a parameter.
• To parse an agent scratchpad — a stop string on "Observation:" is the classic use and remains reasonable, because the format is your invention and the model was told to produce it. Keep the token short, keep it distinctive, and expect to also handle the case where the model produced it in the wrong place.

Worth stating plainly: a stop sequence is never a safety or correctness control. It stops generation after the model has already produced the text, and on a streaming endpoint some of that text may already have been sent. Anything that must not reach a user needs a check on the completed output, not a string match in the sampling loop.

Diagnosing it in order

1. Re-run with no stop parameters and max_tokens generous, and print the raw output with special tokens visible (skip_special_tokens=False if you are in transformers). This single step distinguishes “never emitted” from “not matched” and is worth doing before anything else.
2. If the string is absent, fix the prompt. If a control token is present but generation continued past it, fix eos_token_id / stop_token_ids.
3. If the string is present and generation continued, compare it byte-for-byte with what you passed — including leading spaces and newline style — and confirm the parameter is in the right place in the request body for that server.
4. Read the terminal event. llama.cpp’s stopped_eos, stopped_word and stopped_limit booleans, or an OpenAI-compatible finish_reason, tell you which mechanism ended the request — and if it says length, neither stop mechanism was involved and your output cap ended it.
5. Only then change the stop strings. Most of the time the fix is in one of the first four steps.