Stop Sequences in the Mistral API

stop takes a string or an array of strings and ends generation when one of them appears. The two things worth knowing are that the matched text does not come back to you, and that whether it matches at all is decided by tokenisation rather than by string comparison.

The parameter

Mistral’s chat completions reference documents stop as accepting a string or an array of strings, defaulting to null, with the behaviour given as “stop generation if this token is detected. Or if one of these tokens is detected when providing an array”.

curl https://api.mistral.ai/v1/chat/completions \
  -H "Authorization: Bearer $MISTRAL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistral-large-2512",
    "messages": [
      {"role": "system", "content": "Continue the transcript. Write only the next line spoken by Agent."},
      {"role": "user", "content": "Customer: My card was declined.\nAgent:"}
    ],
    "stop": ["\nCustomer:", "\n\n"],
    "max_tokens": 120
  }'

That is the canonical use: you have prompted the model into a format with a repeating structure, and you want one unit of it rather than the model continuing to play both parts. The stop sequences are the markers that would begin the next unit.

The match is removed from the output

The matched sequence is not included in the returned content. That is the behaviour you want — you asked to stop at the marker, not to include it — but it has a consequence that catches people building parsers: the response gives you no positive evidence of which sequence fired, or that one fired at all.

{
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": " I can look into that for you. Could you confirm the last four digits?"
      },
      "finish_reason": "stop"
    }
  ]
}

The content ends cleanly and nothing in it mentions "\nCustomer:". Note also that finish_reason is stop — the same value a natural end-of-turn produces. If you need to distinguish “the model finished” from “my stop sequence cut it off”, the API does not hand you that distinction on this endpoint, and the practical workaround is to design your stop sequences so that both outcomes are acceptable rather than trying to detect which happened.

What you do get for free is trailing whitespace. Generation stops at the first character of the match, so if your stop sequence is "\nCustomer:" the newline is consumed but any space the model emitted before it is not. Trim the result.

Why a stop sequence sometimes does not fire

This is the failure people report, and the explanation is the same on every provider even though the symptom feels model-specific. Your stop sequence is a string. The model does not emit strings; it emits tokens. The comparison happens against the decoded output, but the opportunity to compare only arises at token boundaries.

Suppose your stop sequence is "END" and the tokenizer has a single token for " ENDING". If the model picks that token, the decoded output now contains END — but it contains it in the middle of a longer unit that was emitted atomically. Depending on where the implementation performs the check, generation either stops having already produced ING, or does not stop at all because the boundary check never saw the substring in isolation. Either way, the behaviour is not what a naive reading of the parameter predicts.

The same mechanism explains the more common complaint: a stop sequence with leading whitespace behaves differently from one without. In most subword vocabularies, leading spaces are part of the token — "END" and " END" are different tokens. A stop sequence of "END" will therefore not match a token that is " END" until the tokenizer happened to split it that way.

Three rules follow, and they are worth applying mechanically:

• Prefer sequences that begin with a newline. "\nCustomer:" is far more reliable than "Customer:", because a newline is almost always its own boundary in the vocabulary.
• Prefer sequences that are unusual as text. A marker like "###" or "</answer>" is unlikely to appear inside a longer token that also contains ordinary content.
• Check your sequence against the tokenizer. Encoding your stop string with the tokenizer the model uses tells you immediately whether it is one token or a fragment of one — see Mistral’s tokenizer versions, since the answer differs between them.

Stop sequences and streaming

When you stream, the stop sequence is applied server-side, so you do not see the matched text arrive and then get retracted. The stream simply ends: a final chunk with finish_reason set, then data: [DONE], exactly as the streaming format describes for any other termination.

The mistake to avoid is implementing stop detection a second time in your client, scanning accumulated deltas for the marker. Deltas are token fragments, so your scan hits exactly the boundary problem described above, only now with a partial buffer — you will match on content split across two frames that the server correctly did not treat as a match, or miss one it did. Let the server do it and read finish_reason.

The open weights behave differently

stop is not a model feature. It is implemented by whatever is running the decoding loop, which on Mistral’s API is Mistral’s server and on your own hardware is your inference engine. The parameter name is shared across the ecosystem; the implementation is not, and neither is the behaviour at the edges.

Concretely, the things that vary between servings of the same Mistral weights are: how many stop strings are accepted, whether the match is checked against decoded text or against token ids, whether the matched text is stripped or returned, whether matching is case-sensitive, and what finish_reason the response reports when a stop sequence fired. Some engines expose a separate parameter for stopping on token ids rather than strings, which sidesteps the boundary problem entirely for markers that happen to be single tokens.

The consequence for anyone moving between hosted Mistral and self-hosted weights — a common path, given the Apache 2.0 licence on much of the line — is that a prompt template relying on stop sequences is one of the pieces most likely to need re-testing. It will not error. It will produce output that includes a delimiter it used to strip, or that runs one turn further than it used to, and the difference shows up in whatever parses the result rather than in the API call.

The mitigation is to make your parser tolerant rather than to make your stop sequences perfect: strip any trailing delimiter defensively even though the API says it will not be there, and treat a response that contains the marker as a parse case rather than an assertion failure. That single habit makes the same code correct on both sides of the move.

When to use something else

Stop sequences are the right tool for one job: cutting a completion at a structural marker you put in the prompt yourself. They are the wrong tool for three others.

For bounding length, use max_tokens. A stop sequence cannot enforce a limit because there is no guarantee the marker ever appears — if the model does not produce it, generation continues, and with max_tokens defaulting to null there is nothing else stopping it. The two parameters are complements, not alternatives; see what the default actually is.

For structured output, use a response format or a tool schema rather than prompting for a delimiter and stopping at it. A stop sequence whose characters can legitimately occur inside the payload — a brace, a quote — will truncate valid output mid-object, and JSON mode combined with a punctuation stop sequence is a reliable way to produce unparseable results.

And for stopping a runaway generation you did not anticipate, neither parameter helps, because you cannot enumerate a marker for output you did not predict. That is what a token bound and a timeout are for.