WebSockets vs SSE vs Polling for AI Responses

The usual comparison is about overhead and bidirectionality, and for this workload both are nearly irrelevant. What decides it is what happens when a phone changes network in the middle of a forty-second answer that cost you real money to produce.

The comparison everyone makes is the wrong one

Transport comparisons usually rank on frame overhead, connection setup cost and messages per second. Those matter for a trading feed or a collaborative editor pushing thousands of small updates.

A token stream is not that. It is a few dozen to a few thousand small messages over tens of seconds, generated by something whose own latency dwarfs every transport difference by orders of magnitude. If you are choosing between transports on efficiency, you are optimising the cheapest component in the system.

The properties that actually differ in ways you will feel are: what happens on a dropped connection, whether the client can rejoin a stream in progress, whether the client can talk back mid-stream, and how much of your infrastructure understands the protocol.

Resumability is a storage problem

Start here, because it reframes the whole decision. Suppose a connection dies eighty per cent of the way through an answer. What can you do?

• Regenerate. You pay again, the user waits again, and — because the model is non-deterministic — the answer differs from the one they were reading. For a long or expensive generation this is the worst option on every axis.
• Resume. Send the tokens they missed and continue. This requires that the generation kept running after the client left, and that its output was written somewhere addressable by sequence number.
• Give up. Show an error and a retry button. Entirely defensible for short, cheap answers, and unacceptable for a three-minute report.

The critical observation is that resuming is not a transport feature. SSE’s Last-Event-ID header is a convention for telling the server where you left off; it does nothing unless the server has the missing events. If your handler pipes provider chunks straight to the client and holds nothing, the reconnect arrives and there is nothing to send.

So resumability means decoupling generation from delivery: the model call runs in a worker, appends numbered chunks to a durable log keyed by run id, and the connection — whatever transport — is a cursor over that log. Once you have built that, every transport becomes resumable, including polling. Once you have not, none of them do. This is why the transport question is downstream of an architecture question, and why arguing about it first is arguing about the wrong thing.

The three transports, honestly

Long polling — a request that the server holds open until there is something to say — sits between the last two and is worth remembering for environments where SSE is blocked. It gives near-push latency using request semantics your infrastructure already understands, at the cost of a held connection per waiting client.

A decision procedure

In order, stopping at the first that applies:

• Nobody is watching in real time — polling. Batch work, background enrichment, anything whose result is collected later. Do not hold a connection open for a result no one is reading.
• The client needs to interrupt, steer or send input mid-generation — WebSocket. Voice, live agent supervision, anything where the user acts on partial output. A second HTTP request can cancel a run, but it needs a run id and a route, and at that point the WebSocket is simpler.
• Several concurrent streams share one page — WebSocket. Multiplexing several runs over one connection avoids per-connection limits and gives you one reconnect path instead of several.
• Otherwise — one-way token stream to a browser — SSE. It is the smallest thing that works, it goes through ordinary HTTP infrastructure, and it fails in ways your existing tooling can see.

Notice that the default is the least capable option. That is deliberate: a WebSocket is a stateful connection that your load balancer, your logging, your rate limiter and your tracing all have to be taught about, and taking on that cost for a one-way stream of text buys nothing.

Traps in each

SSE

The browser’s EventSource reconnects automatically on a dropped connection. If your endpoint starts a fresh generation on each connection, an automatic reconnect is an automatic second purchase. Either make the endpoint a cursor over a durable run, or handle reconnect yourself. Also: EventSource is limited to GET without custom headers, so authentication has to work by cookie or query parameter — and a token in a query parameter ends up in access logs, which is its own problem.

WebSocket

Everything the spec does not give you is now yours. Heartbeats, or intermediaries will close an idle connection during a long think. Reconnect with jittered backoff, or a brief server restart produces a reconnect storm. Message framing, sequence numbers, and a resume handshake. Also, per-connection authorisation happens once at the handshake, so a long-lived socket outlives token expiry unless you re-check.

Polling

The failure mode is cost, on your side rather than the provider’s: a client polling every 500ms for a ninety-second job is 180 requests. Back off as the job ages, return a server-suggested next-poll interval, and make the endpoint cheap — ideally a single indexed read of the job row and its chunks after the cursor.

One property shared by all three: the client must be able to distinguish “finished” from “stopped talking”. Send an explicit terminal message with the finish reason. Silence is not a protocol.