Streaming a Model Response From Cloud Run

If tokens arrive at the browser in one lump at the end, Cloud Run is almost certainly not the reason. It streams by default; something between your model and your client is holding the bytes.

The platform side is already done

Google’s Cloud Run documentation states that Cloud Run supports streaming HTTP responses and that no configuration is required to enable the feature, with the server responding using Transfer-Encoding: chunked. Server-side HTTP and gRPC streaming have been supported since Google announced them in 2020, and server-sent events consumed through the browser EventSource API are an explicitly documented use case.

There is a second reason to care beyond perceived latency. Google documents a maximum HTTP/1 response size of 32 MiB when not using Transfer-Encoding: chunked or a streaming mechanism. Streaming is how a response exceeds that limit at all, so a long generation that returns as one body is not only slow to first byte, it has a ceiling.

A server that streams

The pattern is a generator that yields as the upstream produces, with nothing between it and the response that could accumulate.

from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()

async def token_stream(prompt: str):
    async for chunk in model.stream(prompt):      # upstream SSE or local decode
        yield f"data: {chunk.text}\n\n"
    yield "data: [DONE]\n\n"

@app.post("/chat")
async def chat(prompt: str):
    return StreamingResponse(
        token_stream(prompt),
        media_type="text/event-stream",
        headers={
            "Cache-Control": "no-cache",
            "Connection": "keep-alive",
        },
    )

1. Yield from a generator rather than building a list and returning it. The single most common bug is a comprehension that consumes the whole upstream stream before the first byte is written.
2. Do not set Content-Length. A response with a content length is a response whose size was known in advance, which means it was buffered. Frameworks set this for you if you hand them a complete body.
3. Use an ASGI server — uvicorn or hypercorn — not a synchronous WSGI worker, if the upstream is async. A sync worker will serialise the generator through a thread and frequently buffer it.
4. Deploy with a --timeout that exceeds your longest expected stream, because the whole stream is one request.

Proving it is not buffered

Do not verify this in a browser. Browsers, developer tools and JavaScript clients all add their own buffering, so a UI that renders all at once tells you nothing about where the delay is. Use curl with buffering disabled and timestamps, which localises the problem in one command:

curl -N --no-buffer -X POST https://SERVICE-HASH.REGION.run.app/chat \
  -H "Authorization: Bearer $(gcloud auth print-identity-token)" \
  -H "Content-Type: application/json" \
  -d '{"prompt":"count slowly to twenty"}' \
  | while IFS= read -r line; do printf '%s  %s\n' "$(date +%T.%3N)" "$line"; done

Timestamps that increase steadily mean the stream is working end to end. Timestamps that are all within a few milliseconds of each other at the end mean something buffered, and the next step is to run the same curl against the container locally. If it streams locally and not when deployed, the buffering is in front of Cloud Run. If it fails locally too, it is in your process, which is the more common case by a wide margin.

Also check the response headers with curl -sI. Seeing Transfer-Encoding: chunked confirms the platform is treating it as a stream; seeing Content-Length confirms it is not.

One more localisation step is worth the minute it takes. Send a request that streams something trivially slow and entirely under your control — a loop yielding a line a second, with the model call removed. If that streams and the real endpoint does not, the buffering is happening because of how you consume the upstream, not because of how you emit; the usual culprit is an upstream client that collects the whole response before handing it to you, which no amount of correct emitting can fix. This single test splits the problem in half more reliably than reading any amount of framework documentation.

Where the buffering actually is

• Your framework. Returning a string, a dict, or a fully-consumed iterator buffers by definition. Flask needs a generator passed to the response object; FastAPI needs StreamingResponse; Express needs res.write rather than res.json.
• Your WSGI server. A gunicorn sync worker in front of a generator will often accumulate before flushing. This is the classic Python case and it disappears with an ASGI worker class.
• Compression. A gzip middleware has to see enough bytes to compress usefully, so enabling it in front of a stream converts the stream into batches. Exclude text/event-stream from compression explicitly.
• A load balancer in front. An external Application Load Balancer or Cloud CDN sitting ahead of the service can buffer responses independently of Cloud Run. The tell is that the direct run.app URL streams and the custom domain does not — worth testing both before blaming the application.
• An intermediate buffer you wrote. A queue, a transform step, or a token accumulator added to clean up partial words re-introduces the batching you removed everywhere else. If a stage in your pipeline waits for a sentence boundary, the stream is now sentence-granular by design, which may be what you want but is not what you measured.
• The nginx habit. Adding X-Accel-Buffering: no is harmless and is not the fix here; Cloud Run is not nginx and does not read it. If adding that header appeared to help, something else changed at the same time.

The limits a stream runs into

Google documents a maximum request timeout of 60 minutes, and a stream counts as a single request for its entire duration — so an hour is the ceiling on any one generation, and your configured --timeout is the real one. Concurrency accounting works the same way: a slot is held for the whole stream, which is the interaction described in concurrency settings for inference. Google also documents HTTP/2 supporting up to 100 concurrent streams per client connection, which matters if you are multiplexing gRPC rather than opening a connection per stream.

One design note that outlives all of these numbers: a dropped connection mid-stream leaves you having generated tokens the client never received and having paid for them. If partial results have value, write them somewhere as they are produced rather than treating the HTTP response as the only copy — the alternative is a user who reloads and pays twice for the same answer.