Mistral's max_tokens Default and Output Ceiling

max_tokens on a Mistral request is optional, and leaving it out does not mean “unlimited”. It means the ceiling is decided by arithmetic you did not do, which is fine right up until a long prompt makes the remainder smaller than the answer you needed.

What omitting it does

max_tokens is an optional integer that caps the number of tokens generated in the completion. It counts output tokens only — the prompt is not included in it. Omit it and no explicit cap is applied by your request; generation continues until the model emits its end-of-sequence token, until one of your stop sequences matches, or until it runs out of room.

“Runs out of room” is the part that is not obvious, and it is the reason the answer to “what is the default?” is not a number.

What actually caps the output

A transformer’s context window is a single budget shared by input and output. There is one sequence, and it holds your prompt and the generated continuation together. So the amount you can generate is:

available_output = context_window - prompt_tokens - overhead

# e.g. a 128k-context model, 100k of prompt:
#   131072 - 102400 - (chat template + tool schemas)
#   ≈ 28,600 tokens of headroom, whatever max_tokens says

Two separate limits are therefore in play and the smaller wins:

• The context remainder, computed per request from your prompt length. This moves with every request you send.
• A per-model output cap, where the model or the serving stack imposes one that is lower than the remainder. Some model families are trained and served with a maximum generation length well below their context length; others are not.

The overhead term is real and is routinely forgotten. The chat template wraps every message in control tokens, tool schemas are serialised into the prompt and can run to hundreds of tokens for a rich toolset, and a system prompt is charged like any other input. A request that looks like it has 3,000 tokens of headroom can have considerably less.

Reading the current number per model

GET /v1/models returns an entry per model your key can reach, and each entry carries its context length and capability flags. This is the authoritative, current answer, and it is three lines of code:

curl -s https://api.mistral.ai/v1/models \
  -H "Authorization: Bearer $MISTRAL_API_KEY" \
  | jq -r '.data[] | [.id, .max_context_length] | @tsv' \
  | sort

Take max_context_length as the total budget, subtract the token count of the request you actually intend to send, and you have your real ceiling for that request. Do the subtraction in code rather than in your head — the prompt length is a variable, so a hard-coded max_tokens that works in development fails on the one customer document that is four times the size of your test fixture.

Detecting a truncated answer

When the cap is what stopped generation, finish_reason on the choice is length. When the model finished of its own accord it is stop; when it is asking for tools it is tool_calls. Check the field on every response — a truncated JSON object is a parse error you can explain, and a truncated prose answer is a quality complaint you cannot.

resp = client.chat.complete(model=MODEL, messages=msgs, max_tokens=1500)
choice = resp.choices[0]

if choice.finish_reason == "length":
    # The answer is cut off mid-token-stream. Do not parse it as complete.
    raise OutputTruncated(
        f"hit cap; prompt={resp.usage.prompt_tokens} "
        f"completion={resp.usage.completion_tokens}"
    )

Streaming complicates this slightly and it is worth getting right, because a stream that stops looks the same to a naive consumer whether it finished or was cut off. The finish reason arrives on the final chunk, in the same finish_reason field on the choice. If your consumer breaks out of the loop as soon as content stops arriving and never inspects that last chunk, you have thrown away the only signal distinguishing a complete answer from a truncated one. Read it, and carry it out of the stream handler alongside the accumulated text.

A related interaction: when you are asking for structured output, a length finish reason and a JSON parse error are the same event seen twice. The model was mid-object when the budget ran out, so the braces do not close. Diagnose on the finish reason rather than on the parser — retrying the parse is pointless, and retrying the request with the same cap will fail identically. The fix is a larger budget or a smaller requested object, and the same applies to a schema-constrained response as to Mistral’s JSON mode.

The usage object on the response gives you prompt_tokens, completion_tokens and total_tokens. Logging those three for every request is the cheapest observability you can add here: it turns “the answers got shorter” from a hunch into a number you can plot against prompt size.

Choosing a value on purpose

Setting max_tokens explicitly is worth doing even when you do not need a limit, for reasons that have nothing to do with the ceiling:

• It bounds the bill. Output tokens are the expensive side. An unbounded cap on a prompt that accidentally invites an essay is a cost incident with no upper limit other than the context window.
• It bounds latency. Generation is sequential, so wall-clock time is roughly proportional to output length. A cap is a timeout you can reason about.
• It makes truncation legible. With an explicit cap, a length finish reason means “my budget was too small”. Without one it means “the request ran out of context”, which is a different fix.

It is also worth being clear that max_tokens is a hard stop and not a target. The model is not told about it. It does not plan a conclusion to fit the budget, it does not compress, and it will not wrap up early because the ceiling is close — generation simply ends mid-word when the counter runs out. This is the single most common misunderstanding of the parameter, and it is why a cap set to the length you actually want produces amputated answers rather than concise ones. Two mechanisms exist for shorter output and this is neither of them: ask in the prompt, and set a stop sequence at a boundary you control.

A reasonable default is to set it to roughly twice the longest answer you would accept. Tight enough to catch a runaway, loose enough that a slightly verbose but valid answer is not chopped. And do not use max_tokens to make answers shorter — a cap truncates mid-sentence rather than producing a concise answer. Ask for brevity in the prompt and keep the cap as a guard rail.