Streaming Responses: SSE, Chunks and Backpressure

Streaming a completion looks like a five-line loop and is not. The transport is a text protocol whose framing does not respect your chunk boundaries, the client abort path is subtly wrong in most code, and hanging up does not necessarily stop the meter. All four problems have the same shape: the easy version works in development and fails under real network conditions.

What is actually on the wire

Chat completion streaming is server-sent events: a text/event-stream body of UTF-8 text, framed into events separated by a blank line, each event made of lines like data: {...}. Providers following the OpenAI convention send one JSON object per event and a final data: [DONE] sentinel. Lines beginning with a colon are comments, and are commonly used as keep-alive heartbeats so that intermediaries do not time the connection out during a long prefill.

Three properties of that format cause all the trouble. The frame delimiter is a blank line, not a chunk boundary. The data field may repeat within one event, in which case the values are joined with newlines. And the spec permits \r\n as well as \n, so a reader that splits only on the latter breaks against a standards-compliant server it has never met.

Four ways the naive reader is wrong

• Assuming one network chunk is one event. TCP gives you bytes, not messages. A chunk can carry three events and half of a fourth; the half must be kept and prepended to the next chunk. This bug shows up as intermittent JSON parse errors under load and never on localhost.
• Decoding each chunk independently. A multi-byte UTF-8 code point can be split across chunks. Calling new TextDecoder().decode(value) per chunk emits a replacement character; you need one decoder used with { stream: true } for the whole response. The symptom is mangled emoji and accented characters, only sometimes.
• Ignoring non-data lines. Heartbeat comments, event: names and id: lines are all legal and a reader that JSON-parses every line will crash on them.
• Leaking the connection on early exit. If the consumer stops reading — a break, a thrown error, a React component unmounting — the response body stays open and the socket stays allocated until something cancels it. In a server-side loop this exhausts the connection pool slowly enough to look like a memory leak.

A correct reader

export async function* streamChat(
  url: string,
  init: RequestInit,
  signal: AbortSignal,
): AsyncGenerator<string> {
  const res = await fetch(url, {
    ...init,
    headers: { ...init.headers, accept: "text/event-stream" },
    signal,
  });
  if (!res.ok || !res.body) {
    throw new Error("HTTP " + res.status + ": " + (await res.text()));
  }

  const reader = res.body.getReader();
  const decoder = new TextDecoder();          // ONE decoder for the whole body
  let buf = "";

  try {
    for (;;) {
      const { done, value } = await reader.read();
      if (done) break;

      buf += decoder.decode(value, { stream: true });
      buf = buf.replace(/\r\n/g, "\n");        // accept CRLF framing

      let sep: number;
      while ((sep = buf.indexOf("\n\n")) !== -1) {
        const frame = buf.slice(0, sep);
        buf = buf.slice(sep + 2);               // the remainder stays buffered

        let data = "";
        for (const line of frame.split("\n")) {
          if (line === "" || line.startsWith(":")) continue;   // heartbeat
          if (!line.startsWith("data:")) continue;             // event:, id:
          const v = line.slice(5).replace(/^ /, "");           // one space only
          data = data === "" ? v : data + "\n" + v;
        }

        if (data === "") continue;
        if (data === "[DONE]") return;

        let obj: any;
        try {
          obj = JSON.parse(data);
        } catch {
          continue;      // a provider-specific non-JSON frame; skip, do not die
        }
        if (obj.error) throw new Error(obj.error.message ?? "stream error");

        const piece = obj.choices?.[0]?.delta?.content;
        if (piece) yield piece;
      }
    }
  } finally {
    // Not optional. Releasing the lock leaves the socket open; cancelling the
    // body is what actually tears the connection down when the consumer
    // stopped early. Swallow the rejection -- an already-aborted body throws.
    await reader.cancel().catch(() => {});
  }
}

The finally block is the line most implementations are missing. Because this is a generator, it also runs when the caller breaks out of its for await loop, which is exactly the case that leaks otherwise.

Aborting, and what it does not stop

Cancelling is a client-side act with a server-side consequence you do not control. Aborting the fetch closes your connection. Whether the provider then stops generating — and stops charging — depends on the provider, and is worth knowing rather than assuming.

The rule to design around: tokens already generated are billable whether or not you read them. If a user closes the tab 200 tokens into a 900-token answer, you have certainly paid for 200. Whether you pay for the remaining 700 depends on how promptly the backend notices the hang-up. The defensive measures are the boring ones — set max_tokens to a real bound so the worst case is finite, and prefer aborting early over aborting late.

The second hazard is timeouts. A client timeout aborts your read; it does not necessarily halt generation. Retrying after a timeout can therefore mean two generations billed for one answer delivered, which is the double-billing hazard in full.

Wire the abort to every reason you might want to stop, not just the timer:

const ac = new AbortController();
req.on("close", () => ac.abort());              // the downstream client left
const stall = setTimeout(() => ac.abort(), 30_000);

try {
  for await (const piece of streamChat(url, init, ac.signal)) {
    stall.refresh();                            // reset on every token, not once
    await write(piece);
  }
} finally {
  clearTimeout(stall);
  ac.abort();                                   // idempotent; closes any leftover
}

Backpressure, and re-streaming to a browser

Most services do not consume tokens; they forward them to their own client. That reintroduces the problem the streams API exists to solve. If you write each piece into a downstream response without awaiting the write, and the downstream consumer is slower than the model — a phone on a poor connection — the difference accumulates in your process memory, per connection, unbounded.

The fix is to let the slow side set the pace. Awaiting the downstream write before calling reader.read() again means you simply stop pulling from the provider while the browser catches up, and TCP flow control does the rest. In Node’s stream API that is respecting the return value of write() and waiting for drain; with the Web Streams API, piping through a TransformStream handles it for you, which is the main practical reason to prefer it.

The other half is the path between you and the browser. Compression middleware, a reverse proxy with response buffering on, and any framework helper that collects a body before sending it will each convert your carefully streamed response into one large chunk delivered at the end. The symptom is a time to first token equal to total duration, and the culprit is almost never the model — check X-Accel-Buffering, your proxy’s buffering directives, and whether anything is gzipping the event stream.

A last structural point that catches people out when they build the forwarding layer. HTTP status is committed the moment the headers are sent, and for a stream that is before any generation has happened. There is no way to turn a 200 into a 500 after the fact — so a mid-stream failure can only be expressed inside the body, as an error frame or as a truncated stream. That has consequences on both sides. As a consumer, you must treat a stream that ends without its terminal sentinel as a failure, because a silently truncated response is otherwise indistinguishable from a short answer. As a producer, you need your own error frame convention and a documented terminator, or your callers cannot tell the difference either.

The same reasoning explains why providers send heartbeat comments during a long prefill: without them, an intermediary sees an idle connection and closes it, and the caller cannot distinguish that from a model that is thinking. A reader that treats comment frames as keep-alive — resetting its stall timer without emitting anything — gets this right for free. One that ignores them entirely will time out on exactly the slow requests it most wanted to wait for.