What Happens to In-Flight Requests During a Live Provider Cutover

A cutover is usually described as a moment: the flag flips and traffic moves. At the connection level there is no moment. There is a set of requests that were bound to the old provider before the flip and will stay bound to it until they finish or are killed, and everything that goes wrong during a cutover happens to that set.

The binding is made once, at admission

Your router resolves a provider when the request arrives. From that point the request owns a TCP connection and a TLS session to a specific host, has sent headers carrying a specific API key, and is waiting on bytes from that host. Changing a config value in your process changes what the next resolution returns. It cannot reach into an open socket and redirect it, and no amount of flag propagation speed changes that, because the state that matters is not in your process at all — it is in a connection and in a generation running on somebody else’s hardware.

So the population you have to reason about is: every request admitted in the window between the last flip and now, that has not yet completed. Its size is the arrival rate multiplied by the request duration, which is Little’s law and which is why long generations make cutovers harder than high traffic does. Fifty requests per second finishing in 200 ms leaves ten in flight. Two per second streaming for ninety seconds leaves a hundred and eighty.

The duration that matters is not the median. A cutover has to survive the tail, and for streaming workloads the tail is set by max_tokens and the model’s output rate, not by your typical response.

Failure one: a stream that ends without ending

This is the failure that does not look like one. If the process holding an open streaming response exits, or the connection is reset because a load balancer deregistered the instance, the client sees the stream close. At the transport layer, a stream that closed because the model finished and a stream that closed because someone killed the process are the same event: no more bytes.

The difference is only visible in the payload, and only if you look for it. A stream is complete when its protocol’s terminal marker arrives — OpenAI’s chat completions stream ends with a data: [DONE] line after the final chunk, and Anthropic’s Messages streaming format ends with a message_stop event after message_delta. Code that treats “the iterator finished” as “the generation finished” will happily persist a half-written answer, mark the job complete, and return a truncated summary to a user with no error anywhere in your logs.

The rule this produces is worth stating on its own, because it is useful far beyond cutovers: completion is a payload fact, not a transport fact. Track a per-stream boolean that is set only by the terminal event, and treat a closed stream without it as a failure. The library’s pages on testing that a stream closes cleanly and on drops mid-response cover how to exercise this deliberately; a cutover is simply the scheduled version of the same event.

Failure two: the retry that sends it twice

Now add the retry wrapper that almost every client has. A connection reset is exactly the condition it is designed to retry, so it retries — and because the config has already flipped, the retry resolves to the new provider. The prompt is sent a second time, to a different vendor, and you are billed for both.

Billing twice is the mild version. The serious version is a request in an agent loop, where the model’s output triggers a tool call with a side effect. If the first attempt got far enough to emit a tool call that your loop executed before the socket died, and the retry produces the same call again, the side effect happens twice. Idempotency keys do not save you here: they are scoped to one provider’s API, so a key that would have deduplicated a retry against the old provider means nothing to the new one. Deduplication has to be yours, keyed on your own request id, and it has to be checked before the effect rather than before the model call.

The mitigation that actually works during a cutover is narrower than a general idempotency scheme: do not let a connection error during the drain window be retried across the provider boundary. A retry back to the same binding is fine and usually correct. A retry that changes provider mid-request is a new request wearing an old request’s identity.

Failure three: the usage record dies with the socket

Token usage for a streaming request arrives at the end. On the Anthropic Messages stream the input count comes early, in message_start, and the output count arrives with message_delta near the close. On OpenAI’s chat completions stream there is no usage at all unless the request set stream_options: { include_usage: true }, in which case a final chunk carries it after the content chunks.

Either way, a stream killed mid-flight loses that record. The provider still bills for every token it generated before the kill — generation happened, and your disconnect does not un-generate it — but your own meter never sees a number, so the request contributes zero to your internal spend. During a normal week this is a rounding error. During a cutover it is concentrated: the requests that get killed are exactly the long ones, which are the expensive ones, so the reconciliation gap it opens is much larger than the count of affected requests suggests.

If you cannot avoid killing streams, at least estimate: count the content deltas you did receive, convert to tokens with the provider’s tokenizer, and record the result flagged as an estimate so that the daily reconciliation described in the budget-exposure page does not treat it as a discrepancy.

Failure four: half a conversation on each side

The last failure is the one that survives the request. If your unit of cutover is the HTTP request, a multi-turn conversation or a multi-step agent loop can straddle the flip: turn three went to the old provider, turn four to the new one. Several things do not round-trip across that boundary.

• Tool call identifiers. Each provider generates its own ids for tool calls and requires the corresponding results to reference them. A transcript containing one provider’s ids, replayed to another, is either rejected or accepted with the linkage silently lost.
• Assistant message structure. Content that was a single string on one side and a list of typed content blocks on the other has to be rewritten, and reasoning or thinking content generally cannot be reconstructed at all.
• Cache state. A prompt prefix that was warm on the old provider is cold on the new one. Nothing breaks, but the first turn after the switch costs full price and is slower, which looks like a regression if you are watching latency during the cutover.

The conclusion is that the atomic unit for a cutover is whatever your conversation state is, not the request. Pin the resolved provider onto the session at its first turn and honour that pin until the session ends, then let new sessions land on the new provider. That converts a hard problem into a waiting problem.

What follows from this

Every mitigation above reduces to the same shape: let the old population finish on the old provider, and change only what new requests bind to. That is connection draining, and the sequence — stop admitting, watch the in-flight gauge fall, cancel the stragglers deliberately rather than letting a process exit do it — is the subject of the drain procedure.

Two things are worth deciding before you start. First, your drain deadline, which is a number you can compute from your own duration histogram rather than guess. Second, what a straggler gets: a deliberate error with a Retry-After header is a far better outcome for a client than a truncated stream, because it is unambiguous and it tells the caller when to come back.