Why an OpenAI-Compatible Streaming Response Can Break a Parser Built for OpenAI

Your streaming client has been in production for a year against one provider. Pointed at a compatible endpoint it throws on the first response, or worse, hangs until the timeout. The cause is almost always in one of two layers, and knowing which halves the work.

The three crashes

These are the strings people paste into a search box, and each points at a different layer:

• IndexError: list index out of range, or in JavaScript TypeError: Cannot read properties of undefined (reading 'delta'). Something indexed choices[0] on a chunk whose choices was an empty array. Payload layer.
• json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0), or SyntaxError: Unexpected end of JSON input. Something passed a fragment of an event, a heartbeat comment, or an empty string to a JSON parser. Framing layer.
• No error at all — the stream completes, your handler never fires its completion callback, and the request hangs until a timeout. The terminator never arrived. Also framing layer.

Before debugging, capture the raw bytes. Everything below is visible in the output of a single command, and reading it takes less time than any amount of instrumenting your client.

curl -N -sS "$BASE_URL/chat/completions" \
  -H "authorization: Bearer $KEY" -H 'content-type: application/json' \
  -d '{"model":"'"$MODEL"'","stream":true,
       "messages":[{"role":"user","content":"count to five"}]}' \
  | cat -A | head -40
# cat -A shows line endings: $ for \n, ^M$ for \r\n.

Layer one: SSE framing

Server-sent events have a specification, and a parser written by reading one provider’s output rather than the specification embeds assumptions that provider happened to satisfy. The four that break:

• Splitting on the literal string "data: ". The space after the colon is optional in the SSE specification. An endpoint emitting data: with no space produces zero matches and your parser sees an empty stream. Strip the field name, then strip at most one leading space.
• Assuming one event per read. Network reads have nothing to do with event boundaries. One read can contain three events, or half of one. You must accumulate into a buffer and only process complete events, which are delimited by a blank line — meaning two consecutive newlines, and the line ending may be \r\n. A parser that splits each chunk on newline and parses every piece is the direct cause of the JSON decode error above.
• Not skipping comments. A line beginning with a colon is an SSE comment, used as a keep-alive by proxies and by some servers. It is not JSON. Discard any line whose field name is empty.
• Ignoring the event name. OpenAI’s chat completions stream sends unnamed events, so parsers written against it drop the event: field entirely. Other shapes — including OpenAI’s own newer streaming surface, and Anthropic’s — use named event types where the name tells you what the data field contains. A parser that discards names cannot tell a content delta from a completion signal from an error.

The terminator belongs here too. OpenAI’s chat completions stream ends with a literal sentinel event whose data is [DONE], which is not JSON and must be special-cased before parsing. Not every compatible endpoint sends it. A parser that only finishes on the sentinel hangs; a parser that only finishes on stream close but does not special-case the sentinel throws when it tries to parse it. Handle both: treat the sentinel as end-of-stream if it arrives, and treat the underlying connection closing as end-of-stream regardless.

One more transport-level cause worth eliminating early: if events arrive all at once at the end rather than progressively, nothing about your parser is wrong. Something between you and the server is buffering — a reverse proxy, a compression layer, a serverless platform that does not stream. That is a deployment problem and the transport page covers it.

Layer two: the chunk payload

Once events are being separated correctly, the remaining differences are in what each one contains.

• Empty choices. The most common single crash. OpenAI itself sends a final chunk with an empty choices array when you have requested usage in the stream options. Azure’s hosted version emits a leading chunk carrying content-filter annotations and no choices, which produced a long tail of index errors across client libraries — Langfuse issue 2833 is one of many public reports of exactly that crash, and the fix in every case is the same guard: skip chunks whose choices array is empty.
• The role delta. On the OpenAI shape the assistant role appears in the first delta and not thereafter. Compatible endpoints variously repeat it on every chunk or omit it entirely. Code that starts a new message whenever it sees a role field produces one message per chunk against a server that repeats it.
• Where the finish reason lands. Some servers send a final chunk whose delta is empty and whose finish reason is set; others attach the finish reason to the last content-bearing chunk. Read the finish reason wherever it appears rather than assuming a dedicated terminal chunk exists.
• Null versus absent. A field can be missing, present and null, or present and empty. In Python delta.content being None and being "" are different, and appending None to a string list is a crash three lines later rather than at the source. Guard with a truthiness check, not a presence check.
• Usage placement. Some endpoints put usage on the final content chunk, some on a dedicated trailing chunk, some nowhere. Read it defensively from any chunk that has it and keep the last value seen. The metadata mapping page covers the field names.

Tool calls, which fragment differently

Streamed tool calls are the part most likely to be wrong on a compatible endpoint, because the reassembly rules are subtle enough that a reimplementation rarely matches.

On the OpenAI shape, a delta carries a tool_calls array whose entries have an index. That index — not the position in the array, not the order of arrival — identifies which call a fragment belongs to, which is what makes several tool calls in one turn possible. The call id and the function name normally arrive on the first fragment for that index, and the arguments string arrives in pieces across many chunks, each piece being a substring of the eventual JSON rather than valid JSON itself.

The reimplementations diverge in predictable ways: omitting index entirely because the server only ever emits one call; sending the whole tool call in a single chunk with complete arguments; repeating the id and name on every fragment; or emitting the arguments as an object rather than a string. A parser that concatenates arguments keyed by index handles the first three gracefully if it defaults a missing index to zero, and the fourth needs an explicit type check.

The practical rule: never attempt to parse the arguments until the stream has finished, and always validate against the tool’s schema afterwards rather than trusting that reassembly worked. A partially reassembled argument object that happens to be valid JSON is the worst outcome available.

A parser that survives both

import json

def iter_sse(byte_iter):
    """Yield decoded data payloads from an SSE byte stream."""
    buf = ""
    for raw in byte_iter:
        buf += raw.decode("utf-8", errors="replace")
        while True:
            # events end at a blank line; tolerate CRLF
            idx = min(
                (i for i in (buf.find("\n\n"), buf.find("\r\n\r\n")) if i != -1),
                default=-1,
            )
            if idx == -1:
                break
            event, buf = buf[:idx], buf[idx:].lstrip("\r\n")
            data_lines, name = [], None
            for line in event.splitlines():
                if not line or line.startswith(":"):
                    continue                      # heartbeat / comment
                field, _, value = line.partition(":")
                if value.startswith(" "):
                    value = value[1:]             # the space is optional
                if field == "data":
                    data_lines.append(value)
                elif field == "event":
                    name = value
            if not data_lines:
                continue
            payload = "\n".join(data_lines)
            if payload == "[DONE]":
                return                            # sentinel, not JSON
            try:
                yield name, json.loads(payload)
            except json.JSONDecodeError:
                continue                          # never let one event kill the stream

text, tools, usage, finish = [], {}, None, None
for name, chunk in iter_sse(response.iter_content(chunk_size=None)):
    if chunk.get("usage"):
        usage = chunk["usage"]
    choices = chunk.get("choices") or []
    if not choices:
        continue                                  # usage / filter-only chunk
    choice = choices[0]
    finish = choice.get("finish_reason") or finish
    delta = choice.get("delta") or {}
    if delta.get("content"):
        text.append(delta["content"])
    for call in delta.get("tool_calls") or []:
        i = call.get("index", 0)                  # may be absent
        slot = tools.setdefault(i, {"id": None, "name": None, "args": ""})
        slot["id"] = call.get("id") or slot["id"]
        fn = call.get("function") or {}
        slot["name"] = fn.get("name") or slot["name"]
        args = fn.get("arguments")
        if isinstance(args, str):
            slot["args"] += args
        elif args is not None:
            slot["args"] = json.dumps(args)       # some servers send an object
# stream is over here whether or not [DONE] arrived

Every guard in that code corresponds to a failure named above, and none of them costs anything against a well-behaved endpoint. The two worth transplanting into an existing client immediately are the buffer on the event boundary and the empty-choices skip: between them they account for the large majority of these crashes, including the ones that happen against OpenAI itself the day you turn on usage reporting.

Finally, capture a real streamed response from every endpoint you use and check the raw bytes into your repository as a fixture. Replaying recorded bytes through the parser is a fast test, it needs no network and no key, and it is the only way to keep a fix for a quirk you saw once and cannot reproduce on demand.