What Changes About Streaming Backpressure When You Switch Providers

Backpressure is the property that a slow consumer slows the producer. In a streaming completion there are four places it can be applied and three places it can be silently discarded, and which of those you get is a property of the provider, the transport and your client library together. Change any one of them and a client that was fine starts growing heap or losing connections.

Where the pressure actually goes

Follow one token from generation to your application. The model produces it; the provider’s edge serialises it into an event and writes it to a socket; the kernel on their side buffers it; it crosses the network; your kernel buffers it in a receive window; your HTTP client reads it, decodes the framing, parses the event and hands an object to your code.

If your code stops reading, the pressure propagates backwards one hop at a time. Your receive buffer fills. TCP advertises a smaller window and eventually a zero window. The sender’s send buffer fills. The provider’s write blocks. Only at that point does anything on their side notice, and what happens next is entirely a provider decision: block the generation loop, buffer the completion in memory and keep generating, or give up on the request. Those three choices produce completely different behaviour for the same client code, and none of them is usually documented.

The one thing you can generally rely on is that generated tokens are billed whether or not you read them. If the provider buffers rather than blocks, a client that stops reading has not stopped spending — it has only stopped seeing. Check your provider’s own billing documentation on cancelled and abandoned streams rather than assuming; it is one of the few migration questions where the answer is written down and rarely looked up.

What differs at the transport

The chain above assumes one connection carrying one response. That is the HTTP/1.1 chunked case and the pressure story is exactly TCP’s. Over HTTP/2 there is a second window in front of it: each stream has its own flow-control window, and the connection has another above that. A slow reader closes its stream’s window without touching the connection, which is the point of the design — but it also means the amount of data that can be in flight before anything blocks is the sum of the stream window, the connection window, and both kernels’ socket buffers. That is a much larger cushion than the HTTP/1.1 case, so the same client absorbs a much longer stall before the provider notices anything at all.

This matters at a migration because HTTP/2 is often not a choice you made. Whether it is used depends on the provider’s edge, your client library and its configuration: undici, which backs Node’s global fetch, gates it behind an explicit allowH2 client option and exposes maxConcurrentStreams alongside it (Node.js, undici Pool documentation). Moving between providers can flip the transport underneath you without a line of your code changing, and with it the size of the cushion.

The SDK layer usually removes it entirely

Here is the part that catches people. Most streaming helpers present the response as an async iterator, and many of them decode eagerly: a background task reads the socket as fast as the network delivers, parses events and pushes them onto an in-process queue that your loop drains. If that queue is unbounded — and unbounded is the common default, because a bounded one requires deciding what to do when it fills — then there is no backpressure between your consumer and the network at all.

The consequence is that a slow consumer does not slow anything. It grows your heap. The provider sees a client reading at full speed, the transport never signals anything, and the only symptom is memory. A client written against a library that decodes lazily, straight out of the socket on demand, has genuine end-to-end backpressure; the same client code against an eager library has none. Neither library is wrong and the difference is rarely in the README.

So before assuming a provider changed something, establish which shape your client is. The test is simple: start a stream, sleep for several seconds inside the consumer loop without reading, and watch resident memory and the socket read counter. If bytes keep arriving while you sleep, you are on an eager decoder and the provider’s behaviour is irrelevant to you. That is the setup described in testing a slow consumer.

What the migration looks like from outside

• Memory growth under load that did not exist before. Same consumer, different library layer, unbounded queue. Look at the SDK before the provider.
• Streams dying mid-response when the consumer is slow. The new provider enforces an idle or total-duration limit that the old one did not, and a blocked write counts as idle. Symptoms look like a network fault and are not one.
• Latency that no longer degrades gracefully. If the old provider blocked generation on a slow reader, your slow path was self-limiting. If the new one buffers, the same overload now produces a burst of completed responses arriving at once, which moves the queueing problem into your process.
• Cancellation that does not free anything. Breaking out of the iterator may abandon the object without aborting the HTTP request, leaving the connection generating and billing until it finishes. Explicit abort is the only reliable form — the subject of cancelling a streaming request.

Designing a client that does not care

The portable design gives up on transport backpressure entirely and applies its own, because the transport story is the part you cannot control across providers.

1. Put a bounded queue between the reader and the consumer, with an explicit policy when it fills: block the reader, drop intermediate deltas, or abort. Choosing is the design decision; unbounded is choosing by accident.
2. Make abort the primary control. Hold the request handle or abort signal in scope for the whole stream and call it on every exit path, including exceptions and client disconnects.
3. Treat a slow downstream consumer as a cancellation trigger with a timeout, not as something to absorb. A user who closed the tab is not coming back for the rest of the tokens.
4. Instrument time-to-first-token and inter-token gaps separately. They move for different reasons and only the second one tells you about backpressure.