A Stream That Hangs and Never Finishes

A stream that hangs and never finishes is nearly always buffered rather than stalled: the tokens are being generated and something between the model and your reader is holding them. Which something is answerable in about two minutes with curl -N, and no amount of reading your own streaming code will answer it faster.

Two different hangs

Establish which you have before doing anything, because they share no causes.

The first is the common one and it is fully fixable. The second is about deadlines, and it is the reason the last section of this page exists.

Bisecting the hops with curl

A typical production path is: browser, CDN, load balancer, your application, possibly a gateway, then the provider. Any one of them can buffer. Test them from the inside out and stop at the first one that streams correctly.

1. The provider, from the machine your application runs on.

   curl -N -sS https://api.example-provider.com/v1/chat/completions \
     -H "Authorization: Bearer $PROVIDER_API_KEY" \
     -H "Content-Type: application/json" \
     -d '{"model":"MODEL","stream":true,
          "messages":[{"role":"user","content":"count slowly to 30"}]}'

   -N disables curl’s own output buffering, and without it you will misdiagnose your own terminal. Chunks should appear progressively. If they do not, the problem is upstream of everything you own — usually an egress proxy on that host.
2. Your application, from localhost. Same command against http://127.0.0.1:PORT/your/endpoint. If the provider streams and your app does not, the buffer is in your handler or your framework’s middleware stack.
3. Your application through the load balancer, by internal address. Isolates the proxy layer.
4. The public URL. If everything above streams and this does not, it is the CDN.

Four commands, one culprit. This is worth doing even when you are confident you know the answer, because the confident answer is wrong often enough to be expensive.

Buffering, the usual answer

Reverse proxies

nginx buffers proxied responses by default. For a streaming endpoint you need buffering off and a read timeout long enough for a slow generation:

location /api/stream {
    proxy_pass         http://app;
    proxy_buffering    off;
    proxy_cache        off;
    proxy_read_timeout 300s;
    proxy_http_version 1.1;
    chunked_transfer_encoding on;
}

The application can also request this per response by sending X-Accel-Buffering: no, which nginx honours and several hosted proxies honour too. That header is the better fix when you do not control the proxy configuration, and it is a one-line change.

Compression middleware

gzip middleware is a buffer by construction: it accumulates until it has enough to compress. On an SSE endpoint this converts a stream into a single delivery. Exclude text/event-stream from compression, or disable compression on the streaming route. This one catches people because compression was added for an unrelated reason months earlier.

Headers that make intermediaries behave

Content-Type: text/event-stream
Cache-Control: no-cache, no-transform
Connection: keep-alive
X-Accel-Buffering: no

no-transform matters: it asks intermediaries not to recompress or otherwise rewrite the body. Sending the wrong content type is enough on its own to make some CDNs buffer, because their default policy is content-type driven.

Platform-level buffering

Some serverless and PaaS response paths buffer whole responses regardless of what you send, and some support streaming only on specific runtimes or handler signatures. If the bisection points at the platform, the fix is a configuration or a runtime change, not a header — check the platform’s streaming documentation rather than adding more headers hopefully.

When the client is the buffer

• Python requests without stream=True reads the entire body before returning. With it, iterate iter_lines(). Note that iter_lines without a chunk size can still wait for a line terminator, which is fine for SSE because SSE is line-oriented, and wrong for anything that is not.
• Calling .text() or .json() on a fetch response awaits the whole body. Read response.body as a stream instead.
• A framework that renders the whole response. If your handler returns a list or a joined string rather than a generator or a streaming response object, you have serialised the stream in your own code.
• A generator that never yields early. Code that appends chunks to a list and yields at the end is a buffer wearing a generator’s clothes.

There is a related failure worth ruling out at the same time: text that arrives but is corrupted at chunk boundaries. That is a decoding bug rather than a buffering one, and it is covered in broken characters in output.

Idle connections and keepalives

Long gaps between tokens — a reasoning model thinking, a slow first token behind a queue — look identical to a dead connection to every intermediary in the path, and many of them will close an idle connection after 30 or 60 seconds. The result is a stream that dies exactly when the generation was hardest.

SSE has a comment syntax for precisely this. A line beginning with a colon is ignored by every conforming client and is still bytes on the wire:

: keepalive

data: {"choices":[{"delta":{"content":"Hello"}}]}

Emit one every 15 seconds while waiting. Blank lines terminate events, so the trailing newline matters. If you are proxying a provider’s stream rather than generating one, inject the keepalive in your handler during gaps rather than assuming the upstream sends them.

The disconnect nobody notices

The mirror image of a hung stream, and the one that costs money rather than patience: the reader closed the tab, and nothing in the chain told the generator to stop. The provider keeps generating, you keep paying, and the tokens go nowhere.

It is invisible in ordinary monitoring because the request looks successful. The signature is in the data rather than in an error: a population of requests with full token usage and a suspiciously short delivery duration, or a completion whose output was never written to whatever you persist.

# The generator must both stop reading upstream and close it.
async def proxy(request):
    async with provider.stream(**kwargs) as upstream:
        async for chunk in upstream:
            if await request.is_disconnected():   # framework-specific check
                await upstream.aclose()           # stop the upstream too
                log.info("client gone; upstream closed")
                return
            yield chunk

Two details decide whether this works. Closing your own iterator is not enough — the upstream connection has to be closed explicitly, or the provider goes on generating into a socket nobody reads. And the disconnect check has to run inside the loop; a check before the loop starts tells you nothing about the following thirty seconds.

Where a partial answer has value, persist what has arrived before returning, so a reconnecting reader resumes rather than restarting. That also removes the temptation to retry the whole generation, which on a long answer doubles both the latency and the cost — a recurring entry in bill triage.

Where the timeout belongs

This is the part that most streaming code gets wrong. A total-request timeout is the wrong instrument: set it short and you kill long legitimate answers, set it long and a genuinely dead stream hangs for minutes. What you want is an idle timeout — a deadline on the gap between chunks — plus a separate, generous cap on the total.

import asyncio

async def with_idle_timeout(agen, idle=30.0, total=600.0):
    """Fail fast on silence; still allow a long, healthy generation."""
    deadline = asyncio.get_running_loop().time() + total
    it = agen.__aiter__()
    while True:
        remaining = deadline - asyncio.get_running_loop().time()
        if remaining <= 0:
            raise TimeoutError("total stream budget exceeded")
        try:
            chunk = await asyncio.wait_for(it.__anext__(),
                                           timeout=min(idle, remaining))
        except StopAsyncIteration:
            return
        yield chunk

Thirty seconds of silence is a dead stream; ten minutes of steady tokens is a long answer. Only the two-deadline version can tell those apart. Retrying a failed stream needs its own care, because the user has already seen the partial output — retry strategy and timeouts cover the resumption problem, and proxying a stream covers the pass-through case.