What Timeout Defaults Change When You Switch Providers

The first week after a provider swap produces a class of incident that looks like the new provider being slow and is usually a timeout tuned to the old one. The number that fires is rarely the one you set: it is the smallest limit anywhere between your code and the model, and most teams do not know what that is.

The timeout is a stack, not a setting

At least five independent limits can end an inference request, and the effective timeout is the minimum of all of them. Enumerate yours before changing any of them:

• The SDK’s request timeout. The one you think of, and often the largest.
• The HTTP client underneath it, which has its own connect, read and pool-acquisition timeouts, and whose defaults the SDK may or may not override.
• Every proxy in the path. A load balancer idle timeout, an ingress read timeout, a CDN or edge proxy limit. These are set by a different team, are usually a small number of tens of seconds, and are invisible from the application.
• The runtime’s own execution limit if you are on a serverless platform, which caps the whole invocation regardless of what the client is doing.
• The caller’s deadline. A browser fetch, an upstream service’s timeout, or a user closing the tab. If this is shorter than everything else, the rest is academic and you are paying for tokens nobody receives.

A migration changes the latency distribution, not the stack — which is why the incident appears at the proxy layer that had never been the binding constraint before.

The defaults you inherit

The defaults matter because most code never sets these explicitly. Values documented by their maintainers at the time of writing:

• Python’s requests has no timeout by default; a request can hang until the socket dies. Its own documentation calls setting one essential for production code (requests quickstart).
• httpx defaults to a 5 second timeout on connect, read, write and pool — deliberately aggressive, and far shorter than a long generation (httpx timeout documentation).
• Go’s zero-value http.Client has no timeout, which the standard library documents explicitly.
• Node’s global fetch applies undici’s header and body timeouts rather than an overall deadline; an overall one is set with an AbortSignal. Check the value for your Node version rather than assuming.
• The official OpenAI and Anthropic SDKs set their own generous default — on the order of ten minutes — and their own default retry count, both overridable per client and per request. Both are documented in the SDK READMEs, and both have changed across major versions.
• Infrastructure defaults are the ones that bite: reverse proxies and load balancers commonly default to around a minute of idle time, and edge proxies to somewhat longer. Read your own configuration; do not take a number from a page like this one.

Streaming changes what a timeout means

A non-streaming request has one clock: the whole response arrives or it does not. A streaming request has two, and conflating them is the most common timeout bug in inference code.

The first is time to first byte, and it is the one worth being strict about, because a stream that has not started is a request that is queued or lost. The second is the inter-event gap — how long you will wait between chunks once flowing — and it must be far shorter than the total duration. A total-duration timeout on a stream is nearly useless: set it long enough for a legitimate long generation and it will not fire on a connection that silently stopped producing, which is exactly the failure it existed to catch. That is the case behind testing a stream that never finishes.

Providers differ on how they keep a slow stream alive — some emit periodic events with no content — so an inter-event timeout tuned against one provider’s keepalive behaviour can fire immediately against another that has none. Check for keepalive events on the new provider before you port the number.

Deriving a new value from your own traffic

A timeout copied from a blog post is a guess about somebody else’s prompt lengths. Derive yours:

1. Instrument the two latencies separately — time to first token and total duration — on your existing provider, tagged by route and by prompt-size bucket. Aggregate percentiles, not means; a mean latency tells you nothing about where to put a cut-off.
2. Shadow the new provider with real production prompts, sending the traffic without using the responses, until you have enough calls for the high percentiles to be stable. This is the only honest source of a latency figure for your workload, and it is why nobody else’s number is usable — comparing tail latency across providers is a procedure you run on your own traffic, not a table you look up.
3. Set the timeout above a high percentile of the shadow distribution with real headroom, and be explicit about what you are choosing: the timeout is the rate of requests you deliberately abandon. Round it to something a human will recognise in an alert.
4. Walk the stack and check that every other limit is larger, including the proxy ones you do not own. The value you set is only the effective one if nothing below it is smaller.
5. Add a canary that fails when a timeout fires more often than your chosen rate, so drift in the provider’s latency surfaces as an alert rather than as a support ticket.

Timeout and retry multiply

The number users experience is the timeout times the attempt count plus the backoff between them, and this is where an inherited SDK default does real damage. A ten-minute client timeout with two automatic retries is a worst case measured in half-hours, sitting inside a caller that gave up long ago. Set a total deadline for the operation and give each attempt a slice of the remaining budget, rather than giving every attempt the same generous timeout.

Retrying a timeout is also not free of side effects: you do not know whether the first request completed server-side, so you may be billed for tokens you never saw, and any tool the model called may have run. The safe version carries an idempotency key so a retry is deduplicated upstream where the provider supports it, which is one more reason idempotency keys need attention during a migration.