The Shape of an OpenAI Streaming Chunk, Delta by Delta

A streamed completion is the same object as a non-streamed one, taken apart. Knowing exactly which field appears in which chunk is the difference between an accumulator that works and one that drops the first token, misses the finish reason, or crashes on the last line.

The transport

Set "stream": true and the response is text/event-stream rather than application/json. The body is a sequence of Server-Sent Events: lines beginning data: , each carrying one JSON object, separated by blank lines. The stream ends with a literal sentinel:

data: {"id":"chatcmpl-...","object":"chat.completion.chunk", ...}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk", ...}

data: [DONE]

[DONE] is not JSON. Feeding it to a parser is the most common first bug in a hand-written client, and it is why the official SDKs expose an async iterator rather than a raw body. Every chunk has "object": "chat.completion.chunk" — distinct from "chat.completion" — and shares the id of the completion, so chunks from concurrent streams can be attributed if you are multiplexing.

Two properties of the transport bite before you reach the JSON. Server-Sent Events are plain HTTP, which means anything in the path that buffers a response defeats them: a reverse proxy with response buffering on, a serverless function that returns a completed body, or a compression layer that waits for enough bytes to be worth compressing. The symptom is identical in all three cases — the request works, the content is correct, and the entire stream arrives at once at the end, which is exactly the behaviour streaming existed to avoid. And the connection is long-lived by design, so idle-timeout settings tuned for ordinary requests will cut a slow generation off mid-answer.

One stream, in order

A complete stream for a two-word answer, with nothing elided. This is the whole wire content:

data: {"id":"chatcmpl-A7","object":"chat.completion.chunk",
  "created":1770000000,"model":"gpt-4o-2024-08-06",
  "system_fingerprint":"fp_4e2b1c",
  "choices":[{"index":0,
    "delta":{"role":"assistant","content":"","refusal":null},
    "logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-A7", ... ,"choices":[{"index":0,
    "delta":{"content":"Hello"},"finish_reason":null}]}

data: {"id":"chatcmpl-A7", ... ,"choices":[{"index":0,
    "delta":{"content":" there"},"finish_reason":null}]}

data: {"id":"chatcmpl-A7", ... ,"choices":[{"index":0,
    "delta":{"content":"."},"finish_reason":null}]}

data: {"id":"chatcmpl-A7", ... ,"choices":[{"index":0,
    "delta":{},"finish_reason":"stop"}]}

data: [DONE]

Reading that in order, four things are worth naming.

• The first chunk carries the role and no text. delta.role appears exactly once, in the first chunk, with content as an empty string. A client that assumes every chunk has text will handle this fine; one that assumes every chunk with a content key has non-empty text will emit a spurious zero-length write.
• Content chunks carry no role. The role is established once and not repeated. Accumulate it from the first chunk.
• The finish reason arrives on its own chunk, with an empty delta. This is the part most often missed. The last chunk containing text has finish_reason: null; the reason comes afterwards in a chunk with "delta": {}. If your loop stops when the text stops, you never see it — and never learn that the response was truncated at "length". The values it can take are enumerated in the finish_reason reference.
• There is no usage. Not by default. See below.

Chunks are not tokens, either. A chunk usually carries one token but the API makes no promise of that, and multi-byte characters are not split across chunks in a way you have to reassemble — content deltas are valid UTF-8 strings. Concatenating them in arrival order gives you exactly the string a non-streamed request would have returned in message.content.

The delta object

delta is a partial message. Every field that can appear on an assistant message can appear on it, and each behaves the same way: it shows up when it has something new to say.

• role — first chunk only, always "assistant".
• content — a fragment of text, to be concatenated. Absent entirely on chunks that carry something else.
• refusal — a parallel string channel used when the model declines. It streams in fragments exactly as content does, and it is a separate accumulator: a stream can produce refusal deltas and no content deltas at all. A client that only accumulates content will render an empty response for a refusal.
• tool_calls — an array of fragments carrying an index to reassemble against, covered in detail in the parallel tool call format.

logprobs, when requested, arrives per chunk alongside the delta rather than accumulated at the end, which is the shape you want for a live confidence display. And system_fingerprint appears on every chunk, identical each time — it identifies the backend configuration and is the companion to the seed parameter, discussed in the seed parameter.

Getting usage out of a stream

By default a streamed response has no usage object anywhere, which historically forced people to re-tokenise the output locally to estimate cost. The fix is a request option:

{
  "model": "gpt-4o-2024-08-06",
  "stream": true,
  "stream_options": {"include_usage": true},
  "messages": [{"role": "user", "content": "Say hello."}]
}

With it set, one extra chunk is emitted after the finish-reason chunk and before [DONE], and it has a shape nothing else in the stream has:

data: {"id":"chatcmpl-A7","object":"chat.completion.chunk",
  "choices":[],
  "usage":{"prompt_tokens":11,"completion_tokens":3,"total_tokens":14}}

choices is an empty array. Any code that reaches for chunk.choices[0] without checking length will throw on this chunk, which is why turning on usage reporting sometimes breaks a client that was working. Guard the index access, then read usage where it appears.

What a correct accumulator does

let role = null;
let content = "";
let refusal = "";
let finishReason = null;
let usage = null;

for await (const chunk of stream) {
  if (chunk.usage) usage = chunk.usage;      // before the choices check
  const choice = chunk.choices[0];
  if (!choice) continue;                     // the usage-only chunk

  const d = choice.delta ?? {};
  if (d.role) role = d.role;
  if (d.content) content += d.content;
  if (d.refusal) refusal += d.refusal;
  if (choice.finish_reason) finishReason = choice.finish_reason;
}

if (finishReason === "length") throw new TruncatedOutput(usage);
if (refusal) throw new Refused(refusal);

Five behaviours in twenty lines, each corresponding to something in the walk above: usage read before the choices guard, an empty choices array skipped rather than indexed, role captured once, two independent string accumulators, and a finish reason that arrives after the last text. A client that does those five things handles every stream this API produces, including tool calls once the index-keyed reassembly is added for those.

The one failure this does not cover is a stream that stops mid-flight — a dropped connection, a timeout. There is no chunk for that; the iterator simply ends without a finish reason ever having arrived. That is why finishReason starts as null and is checked afterwards rather than inside the loop: a null at the end means the stream did not complete, which is a different problem from any of the documented finish reasons and deserves a different branch.