Draining Traffic Safely Before a Provider Cutover

Draining is the whole of a safe cutover. The switch itself is one boolean; the work is making sure that when it flips, nothing is left holding a connection it is about to lose.

What has to be true before you start

Three properties, and if any is missing the drain cannot work no matter how carefully you sequence it.

• The binding is pinned at admission. The resolved provider must be stored on the request context the moment it is chosen, and every later step — the initial call, retries within the request, the streaming loop, the usage record — must read the pin rather than re-resolving. A router that re-resolves on retry will move a request across the boundary in the middle of its own life, which is the duplicate-send failure described in the in-flight page.
• In-flight requests are counted. A gauge, incremented at admission and decremented in a finally, keyed by provider and by whether the request is streaming. You cannot drain what you cannot observe, and “wait a bit” is not a procedure.
• Streams are cancellable. Every outbound call carries an AbortSignal or its equivalent that you hold a handle to, so that ending a request is something you do rather than something the operating system does for you.

// admission: pin, count, and keep a handle
const binding = router.resolve(req);           // once, here, never again
const controller = new AbortController();
inflight.add({ id: req.id, binding, controller, startedAt: Date.now() });
try {
  return await callProvider(binding, req, controller.signal);
} finally {
  inflight.delete(req.id);
}

Computing the drain deadline

The deadline is how long you are willing to wait for the old population to finish before you start cancelling. It is not a convention; it comes out of data you already have, because you log request durations.

Take the duration distribution for the traffic that will be affected — filtered to the routes and models involved in the cutover, not your whole service — and read the high quantile. The relevant one is higher than you would use for an alert: at p99 you are choosing to cancel one request in a hundred, and during a cutover the survivors are disproportionately the long, expensive, user-visible ones. Take p99.9, add margin for the fact that the tail during a drain is worse than the tail in steady state, and cap it at the largest value your deployment system will actually allow, which is the constraint discussed below.

There is an upper bound worth computing as a sanity check: a streaming request cannot exceed max_tokens divided by the model’s sustained output rate, plus time to first token, plus any tool-call round trips in an agent loop. If that bound is far above your p99.9, the difference is telling you something useful — most requests stop early, and your worst case is set by a parameter you can lower for the duration of the cutover.

The sequence

1. Announce the intent, not the change. Set the flag that says “the old provider is draining”. Nothing about existing requests changes; the flag is only read by the resolver.
2. Stop admitting to the old binding. New requests resolve to the new provider. New sessions get the new provider; existing sessions keep their pin, per the conversation-boundary rule. Traffic to the old provider now decays rather than stops.
3. Watch the gauge. Poll the in-flight count for the old binding. It should fall monotonically. If it does not, you have found a code path that is still re-resolving, and you should stop and fix that rather than continue.
4. Wait for zero, or for the deadline. Whichever comes first. In the common case the gauge reaches zero well inside the deadline and nothing else happens.
5. Cancel the remainder deliberately. Walk the remaining entries, abort each controller, and record a distinct outcome — cancelled_for_drain — so these are countable afterwards and are not confused with provider errors in your dashboards.
6. Only now, revoke. Remove the old binding entry, rotate or delete its key, and cancel the contract if that was the point. Doing this before the gauge is zero converts a clean drain into a wall of authentication errors.

The grace-period trap

The most common way a correct drain fails is that something below you kills the process before the drain finishes, and every layer of a modern deployment has its own timer.

If the cutover coincides with a deploy or a restart, the container platform sends a termination signal and then waits a fixed grace period before sending an unblockable kill. On Kubernetes that is terminationGracePeriodSeconds, and its default is 30 seconds — a value chosen for HTTP services whose requests take milliseconds, and which is shorter than a single long generation. If your drain deadline is 120 seconds and your grace period is 30, your drain is decorative: the platform kills the process at 30 and every remaining stream is truncated rather than cancelled.

The same arithmetic applies to load balancer deregistration delay, to any proxy idle or read timeout in front of you, and to the timeout your own reverse proxy applies to an upstream response. Every one of them must exceed the drain deadline, and it is worth writing the chain down explicitly:

grace_period  >=  preStop_sleep + drain_deadline + shutdown_slack
lb_dereg_delay >= preStop_sleep
proxy_read_timeout > longest_expected_stream

The preStop sleep exists because deregistration is eventually consistent: the load balancer may keep sending you new connections for a short period after the platform has decided to stop you. Sleeping before you begin the drain lets that settle, and it is additive to everything downstream.

What a straggler gets

A cancelled request needs a response, and the choice matters more than it looks. Three options, in increasing order of quality.

• Nothing — the connection closes. This is what happens by default and it is the worst outcome, because a client cannot distinguish it from a completed generation without checking for the terminal event, and many clients do not.
• An error frame inside the stream. If you have already sent bytes you cannot change the status code, but you can emit a terminating event of your own that says the stream ended abnormally, before closing. Any client that parses your event types can then distinguish truncation from completion.
• A 503 with a Retry-After header, for requests that have not yet produced output. This is unambiguous, it is what the status code is for, and it tells the caller when the answer will be different. RFC 9110 defines Retry-After for exactly this case as well as for 429; the two formats it permits and the parsing trap are covered in the Retry-After page.

For requests that had begun streaming, the partial output is often still worth something. Persist it flagged as partial rather than discarding it, along with the token estimate for the deltas you did receive, so the request is not invisible in your spend accounting.

Rehearsing it

Run the whole sequence in a non-production environment against synthetic traffic whose duration distribution you control: a mix of short requests and a handful deliberately asking for the maximum output you allow. Start the drain in the middle. Then assert three things — that every stream either carried its provider’s terminal event or was recorded as cancelled_for_drain; that the in-flight gauge reached zero; and that no request’s outbound calls span two bindings.

The third assertion is the one that finds real bugs, and it is cheap: record the binding id on every outbound attempt and group by request id at the end. A request with two distinct bindings is a re-resolution you did not know you had, and it is far easier to find here than in the duplicate-charge investigation three weeks later.