The Streaming Response Shape of the Mistral Chat API

Mistral streams server-sent events terminated by a data: [DONE] sentinel. The frames in between are chat completion chunks carrying deltas, and almost every bug in a streaming client comes from assuming a delta contains more than it does.

What comes down the wire

Set stream to true and the response content type becomes text/event-stream. Mistral’s chat completions reference describes the behaviour as tokens being “sent as data-only server-side events as they become available, with the stream terminated by a data: [DONE] message”.

curl -N https://api.mistral.ai/v1/chat/completions \
  -H "Authorization: Bearer $MISTRAL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistral-large-2512",
    "messages": [{"role": "user", "content": "Name three French cheeses."}],
    "stream": true,
    "stream_options": {"include_usage": true}
  }'

“Data-only” is the phrase to notice. The SSE specification allows named events via an event: line; Mistral does not use them. Every frame is a data: line, frames are separated by a blank line, and the payload is either a JSON object or the literal string [DONE]. A client that switches on an event name will find there is nothing to switch on.

The frames, in order

The stream for the request above has the following structure. Each JSON payload is a chunk object with the same envelope as a non-streaming completion, except that choices[].message is replaced by choices[].delta and the object type is chat.completion.chunk:

data: {"id":"cmpl-8a1c","object":"chat.completion.chunk","created":1786500000,
       "model":"mistral-large-2512",
       "choices":[{"index":0,"delta":{"role":"assistant","content":""},
                   "finish_reason":null}]}

data: {"id":"cmpl-8a1c","object":"chat.completion.chunk","created":1786500000,
       "model":"mistral-large-2512",
       "choices":[{"index":0,"delta":{"content":"Com"},"finish_reason":null}]}

data: {"id":"cmpl-8a1c","object":"chat.completion.chunk","created":1786500000,
       "model":"mistral-large-2512",
       "choices":[{"index":0,"delta":{"content":"té"},"finish_reason":null}]}

... many more content deltas ...

data: {"id":"cmpl-8a1c","object":"chat.completion.chunk","created":1786500000,
       "model":"mistral-large-2512",
       "choices":[{"index":0,"delta":{"content":""},"finish_reason":"stop"}],
       "usage":{"prompt_tokens":14,"completion_tokens":31,"total_tokens":45}}

data: [DONE]

Four properties of that sequence are load-bearing:

• The first delta carries the role and no content. A client that concatenates delta.content without a null check appends undefined to its buffer on the very first frame. This is the single most common streaming bug and it produces the distinctive output that begins with the word “undefined”.
• Deltas are increments, not snapshots. Each frame holds only the new fragment. You accumulate; you do not replace.
• A fragment is not a character or a word. It is whatever the tokenizer produced, so it can split a word, split an accented character across a multi-byte boundary in a badly decoded client, or contain leading whitespace that belongs to the previous word. Never do string matching on a single delta.
• finish_reason arrives on its own frame. It is null on every content frame and set on the last one, which typically carries an empty content delta. Read completion state from that field rather than from the arrival of [DONE].

Tool calls arrive in pieces

When the model calls a function during a stream, the tool_calls structure is built up across frames the same way content is. The first frame carrying a tool call has the id, the index and the function name; subsequent frames append fragments of the arguments string:

data: {"choices":[{"index":0,"delta":{"tool_calls":[
        {"index":0,"id":"D681PevKs","type":"function",
         "function":{"name":"retrieve_payment_status","arguments":""}}]},
      "finish_reason":null}]}

data: {"choices":[{"index":0,"delta":{"tool_calls":[
        {"index":0,"function":{"arguments":"{\"transaction"}}]},
      "finish_reason":null}]}

data: {"choices":[{"index":0,"delta":{"tool_calls":[
        {"index":0,"function":{"arguments":"_id\": \"T1001\"}"}}]},
      "finish_reason":null}]}

data: {"choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}]}

The consequence is that arguments is not parseable JSON until the stream ends. Accumulate by the index field — not by array position in the delta, which is only the position within that frame — and parse once, after finish_reason arrives. Because parallel tool calls are on by default, more than one index can be in flight at the same time and their fragments interleave.

Getting usage out of a stream

A streamed response does not report token usage by default. Mistral’s known limitations page states that token usage requires explicitly setting stream_options.include_usage. Without it, the chunks carry no usage object and you are left counting tokens yourself, which means reimplementing the right tokenizer version and getting the chat template overhead right — a lot of work to approximate a number the server already knows.

Set it once, globally, in whatever wraps your client. Cost attribution that silently omits every streamed call is worse than no cost attribution, because it looks complete.

What sits between you and the stream

A large share of “streaming does not work” reports are not about Mistral’s API at all. The frames leave the server correctly and something in the path holds them, and the symptom is distinctive: the whole response arrives at once, at the end, exactly as if you had never set stream.

• Buffering proxies. A reverse proxy that buffers responses will collect the entire body before forwarding it. This is the default in several common configurations and has to be turned off explicitly for the route that carries the stream.
• Compression. Gzip over an event stream works by filling a compression window before it emits anything, which reintroduces exactly the latency streaming existed to remove.
• Serverless response handling. Platforms that build a complete response object before returning it cannot stream by construction, whatever the client library does. This one is worth checking early, because no amount of client-side debugging will reveal it.
• Client-side buffering. An HTTP client that decodes the body as a single string is not going to give you frames. The -N flag in the curl example above exists for the same reason.

The diagnostic is to bisect the path rather than to reason about it. Run the curl command against Mistral directly from the machine hosting your service; if frames arrive incrementally there and not through your application, the problem is in your stack and not in the API. It is worth doing this before touching any code, because every layer in that list looks identical from inside the application.

One more property of the wire format matters here. Because frames are separated by a blank line and delivered as a byte stream, a single read from the socket can contain half a frame, or two and a half frames. Parse by scanning for the delimiter and keeping a remainder buffer, rather than assuming one read equals one event. Client libraries handle this; hand-rolled readers frequently do not, and the bug only appears under load, when frames start coalescing.

Every way a stream ends

A streaming client has to handle termination, and there are more terminations than the happy path:

• Normal completion. A final chunk with finish_reason set, then data: [DONE], then the connection closes. Mistral’s documented values include stop for a natural end, length when generation hit max_tokens, and tool_calls when the model is asking you to run a function.
• Idle timeout. The known limitations page states that streaming connections time out after ten minutes of inactivity. That is inactivity, not total duration — a stream producing tokens steadily can run longer, and a stream stalled behind a slow first token can be cut.
• Transport failure. The connection can drop mid-stream, with no final chunk and no sentinel. Your buffer holds a partial answer that looks superficially fine. This is why the check is “did I see a finish_reason” rather than “did the loop exit”.
• Client-side cancellation. Abandoning the read does not necessarily stop generation immediately, so a cancelled stream can still cost output tokens. Close the connection explicitly.