Streaming Through Your Own Backend Without Breaking It

It works locally. You deploy it, and the response that streamed beautifully now arrives in one lump after twelve seconds. Nothing in your code changed; something between your code and the browser decided to be helpful.

The symptom

You are proxying a provider’s server-sent event stream through your own backend, because the API key cannot go in the browser and because you want to log, meter and authorise the call. The provider streams to you correctly — you can see the chunks arriving in your handler. The browser sees nothing until the end.

Somewhere between your write and the client’s read, a layer is accumulating bytes. This is almost never a bug in that layer. Buffering a response is the right default for the ninety-nine per cent of responses that are documents, because it lets the intermediary set a content length, compress efficiently and free the upstream connection early. Streaming is the exception, and it has to be declared.

Who is buffering

Work outward from your process. Each of the following is documented behaviour rather than folklore, and each has an off switch.

• Your framework’s response object. Many server frameworks collect the body and send it on completion by default. You need the explicit streaming path: a ReadableStream as the response body in a fetch-style runtime, or writing to the response and flushing in a Node-style one. If your handler returns a string, nothing downstream can stream it.
• Compression middleware. A compressor is buffering by nature — it wants a block of input to compress well. Node’s widely used compression middleware exposes res.flush() precisely for this case; call it after each chunk, or exempt text/event-stream from compression entirely. This is the layer people forget, because it is one line in an app setup file they wrote months ago.
• nginx. proxy_buffering is on by default, which means nginx reads the upstream response into its own buffers before sending. The documented per-response override is the X-Accel-Buffering: no header, which nginx honours from the upstream and which is the right mechanism because it lets one endpoint stream without changing global config. Setting proxy_buffering off in the location block works too, and affects everything in that block. proxy_read_timeout is worth checking at the same time: its default is 60 seconds, which is shorter than some generations.
• The CDN or edge in front of it. Behaviour varies and changes, so verify against your provider’s current documentation rather than a blog post. What is generally true is that Content-Type: text/event-stream plus Cache-Control: no-cache, no-transform is the combination that asks every well-behaved intermediary not to buffer or rewrite, and no-transform is the half people omit.
• Serverless function response handling. Some function runtimes historically returned the complete response body to the platform rather than streaming it out, and support for streaming responses varies by platform, region and runtime version. Treat “does this platform stream” as something to confirm for your deployment rather than assume.

A useful diagnostic ordering: if curl -N against your deployment streams and the browser does not, the problem is a browser or client-side layer. If curl -N against the origin streams and against the public URL does not, the problem is an intermediary, and you can bisect it by hitting each hop directly.

The shape of a correct handler

export async function POST(req: Request) {
  const upstream = await fetch(PROVIDER_URL, {
    method: "POST",
    headers: { authorization: "Bearer " + key, "content-type": "application/json" },
    body: JSON.stringify(payload),
    signal: req.signal,        // client goes away -> upstream is cancelled
  });

  if (!upstream.ok || !upstream.body) {
    // Still before the first byte, so a real status code is possible.
    return new Response(JSON.stringify(normaliseError(upstream)), { status: 502 });
  }

  let sent = 0;
  const out = new TransformStream({
    transform(chunk, controller) { sent += chunk.length; controller.enqueue(chunk); },
  });
  upstream.body.pipeTo(out.writable).catch(() => { /* handled below */ });

  return new Response(out.readable, {
    headers: {
      "content-type": "text/event-stream; charset=utf-8",
      "cache-control": "no-cache, no-transform",
      connection: "keep-alive",
      "x-accel-buffering": "no",
    },
  });
}

The header block is the part to copy. The charset matters because some clients will not treat an unlabelled event stream as text; no-transform matters because it is the standard way of telling an intermediary not to recompress or rewrite; and x-accel-buffering is harmless where it is not understood.

Errors after a 200

The structural problem with proxying a stream is that the status code is committed before the outcome is known. You send 200 when the upstream connection opens; the failure happens eleven seconds later. There is no mechanism in HTTP to take it back.

So mid-stream failures must be in-band. Define an event type for them and make the client handle it — the same normalised error class you use everywhere else, delivered as a payload rather than a status:

event: error
data: {"class":"upstream_timeout","retryable":true,"at_token":184}

event: done
data: {"finish":"stop","tokens_out":412,"route":"provider-a/model-x"}

Two habits go with this. Always send a terminal event, so the client can distinguish “finished” from “connection died” — an SSE stream that simply stops is ambiguous, and the browser’s EventSource will reconnect on a dropped connection, which for a metered endpoint means a second generation unless you are prepared for it. And send a heartbeat comment line (a line beginning with :) every few seconds during long silences, because idle timeouts in intermediaries are counted from the last byte and a model thinking for forty seconds sends no bytes.

Cancellation is a billing feature

When a user closes the tab, the browser drops the connection. If your handler does not notice, or notices and does not propagate, the upstream generation continues to completion and you pay for every token nobody will read. On a page where users routinely abandon slow answers, this is not a rounding error.

The mechanism is the abort signal, and the discipline is to pass it all the way through: the incoming request’s signal into the upstream fetch, and an explicit cancel on the upstream body reader in your cleanup path. Test it — abandon a request and confirm the upstream connection closes — because the failure is invisible from the user’s side and shows up only on an invoice. Also note that cancelling does not necessarily refund what was already generated; providers differ, and the amount you avoid paying is the part not yet produced.

Confirming it rather than believing it

Buffering behaviour changes between versions and platforms, so treat every claim on this page as something to verify in your own deployment. The verification is cheap: an endpoint that emits one SSE event per second for ten seconds, and curl -N against it from outside your network. If the ten lines appear one per second, nothing in the path is buffering. If they appear together at the end, bisect by hop. Keep that endpoint in the codebase behind a flag — it takes twenty lines and it will answer this question every time somebody changes the infrastructure.