The Streaming Response Shape of the xAI API

Set stream: true on a chat completions request to xAI and what comes back is the OpenAI chunk shape, field for field. The parts worth knowing are the two places it is not.

The chunk shape

Streaming is Server-Sent Events. Each event is a data: line carrying one JSON object, and xAI documents the object as chat.completion.chunk with the familiar envelope:

data: {
  "id": "<completion_id>",
  "object": "chat.completion.chunk",
  "created": 1770000000,
  "model": "grok-4.5",
  "choices": [
    {
      "index": 0,
      "delta": { "role": "assistant", "content": "The " },
      "finish_reason": null
    }
  ],
  "system_fingerprint": "fp_xxxxxxxxxx"
}

The rules that follow from that shape are the ordinary SSE rules and they are worth stating because hand-rolled clients get them wrong. Frames are separated by a blank line, not by a newline. A field may be absent rather than null — delta.role typically appears once, on the first chunk, and not again — so read defensively rather than indexing. And finish_reason is null on every chunk until the last one for that choice, where it carries the reason the generation ended: stop when a stop token or a supplied stop sequence was hit, length when the token ceiling was reached.

Terminating the stream

xAI sends the OpenAI sentinel. The final event is the literal line data: [DONE], whose payload is not JSON and will throw if you hand every frame to a parser without checking.

for raw in response.iter_lines():
    if not raw:
        continue
    line = raw.decode() if isinstance(raw, bytes) else raw
    if not line.startswith("data: "):
        continue
    payload = line[len("data: "):]
    if payload == "[DONE]":
        break
    chunk = json.loads(payload)
    for choice in chunk.get("choices", []):
        piece = choice.get("delta", {}).get("content")
        if piece:
            print(piece, end="", flush=True)

Two failure modes hide in the difference between that loop and the naive one. A stream that ends without [DONE] ended abnormally — a dropped connection, a timeout, a server-side error — and a client that treats end-of-body as success will silently return a truncated answer as though it were complete. And an error mid-stream arrives after a 200 status has already been sent, because the headers went out before generation started. Check for [DONE] explicitly and treat its absence as a failure.

Getting usage out of a stream

A streamed response has no single body to read token counts from, so usage is opt-in. Send stream_options with include_usage: true and xAI emits a usage chunk before the stream closes. The documented object is richer than the non-streaming minimum:

"usage": {
  "prompt_tokens": 1483,
  "completion_tokens": 226,
  "total_tokens": 1709,
  "prompt_tokens_details": {
    "text_tokens": 1102,
    "audio_tokens": 0,
    "image_tokens": 381,
    "cached_tokens": 0
  }
}

That prompt_tokens_details breakdown is the only published route to several numbers xAI does not document anywhere else — it is how you find out what an image actually cost, and it is the basis of the image token page in this cluster. On a reasoning model, usage is also where reasoning_tokens appears, and reasoning tokens are billed whether or not the trace is shown to you.

Do not reconstruct completion tokens by counting the chunks. A chunk is a transport unit, not a token: one chunk can carry several tokens and the boundaries are not stable.

Two departures from OpenAI

Tool calls arrive whole. xAI documents that with streaming, a function call is returned in a single chunk rather than streamed across chunks. OpenAI fragments tool_calls[].function.arguments across many deltas, so most client code accumulates a string and parses it at the end. That code is still correct against Grok — one fragment is a valid sequence of fragments. Code written the other way, parsing each chunk as complete JSON, works against Grok and breaks the moment you route the same traffic to a provider that fragments.

Reasoning has its own events, on the other endpoint. The chat.completion.chunk shape above has no field for a reasoning trace. On the Responses endpoint, xAI documents typed streaming events instead — including response.reasoning_text.delta and response.reasoning_summary_text.delta alongside the output text — so a client that only reads choices[].delta.content will see a long silence during thinking and then the answer. That is not a stall, and building a progress indicator that assumes it is one is a common way to make a reasoning model look broken.

There is a third, quieter difference: xAI documents the usage object on a streaming chunk with a prompt_tokens_details breakdown into text, audio, image and cached tokens. Code that reads only prompt_tokens works, and code that assumes the sub-object is absent will throw the first time an image goes through. Read it defensively in both directions — treat every field in the usage object as optional, because which sub-fields appear depends on what was in the request and on the model that served it.

Errors arrive after a 200

This is the part of streaming that most non-streaming code has no equivalent for, and it is where a ported client usually breaks.

A streaming response commits to its status code before generation starts. The server has to send headers to open the event stream, and it sends them as soon as the request validates — which is before the model has produced a single token. Everything that can go wrong after that point goes wrong inside a response that has already told your HTTP client it succeeded. A rate limit hit mid-generation, a capacity failure, a content-side abort, a dropped upstream: all of them arrive in a 200 body.

So response.raise_for_status() is necessary and nowhere near sufficient. The three checks that actually matter run over the frames:

• Did [DONE] arrive? If the body ended without it, the stream ended abnormally, and whatever text you accumulated is a fragment. A client that returns accumulated text on end-of-body treats a truncated answer as a complete one, silently and forever.
• Did a frame carry an error object instead of choices? An error surfaced mid-stream is a JSON payload on a data: line like any other. Code that reaches straight for chunk[“choices”][0] raises a KeyError that gets reported as a client bug rather than as the upstream failure it is.
• What was the final finish_reason? It is null on every chunk until the last one for that choice, so the answer to “was this complete?” only exists at the end. Capture it as you go rather than hoping to find it later.

Retrying is the second-order problem. By the time a stream fails you have usually already shown the user part of an answer, so a naive retry produces a visible restart, and you are charged for the prompt again plus every token generated before the failure. Decide up front which you are building: buffer the whole answer and retry invisibly, at the cost of throwing away the latency benefit streaming exists for, or stream live and handle a mid-answer failure in the interface. Doing neither deliberately means doing the worst version of both.

Timeouts on reasoning models

xAI’s reasoning documentation carries an operational warning that belongs on this page: with reasoning models, extend your client timeout manually or the connection can close before the response completes.

The reason is the gap between headers and first visible token. A reasoning model can spend a long stretch producing trace tokens before anything reaches delta.content, and a default read timeout of thirty or sixty seconds measured against inactivity on the content field will fire during entirely healthy generation. Raise the read timeout, and if you have an idle-detection layer, feed it every frame you receive rather than only the ones that carry visible text.