Migrating a Client Library's Connection Pool Settings Between Providers

A connection pool that was invisible for two years starts adding hundreds of milliseconds the week after a provider swap, and nothing in the application changed. The pool did not get worse; the workload underneath it did, and pool settings are a function of that workload rather than of the client.

Pool size is arithmetic, not a rule of thumb

The number of connections you need in flight is given by Little’s law: the average number of items in a system equals the arrival rate times the average time each spends there. For HTTP clients that is L = λ × W, where λ is requests per second and W is the mean request duration.

Assumption: steady 25 requests/second against one provider.

Provider A, mean completion duration 4.0 s
  L = 25 x 4.0 = 100 concurrent requests in flight

Provider B, same traffic, mean completion duration 8.0 s
  L = 25 x 8.0 = 200 concurrent requests in flight

With a pool capped at 100 connections, provider B's workload
queues 100 requests permanently. Each queued request waits, on
average, for a slot:

  wait = (200 - 100) / 25 = 4.0 s of added latency

...which then feeds back: longer W raises L further until the
system finds a new equilibrium at a much worse latency.

The feedback in the last line is why this failure appears suddenly rather than gradually. Below capacity the pool is invisible; above it, added queueing raises the observed duration, which raises the required concurrency, which lengthens the queue. There is no gentle slope between the two regimes.

The inputs are yours to measure. λ comes from your own traffic, and W should come from the candidate provider under your own prompt lengths — not from a published latency figure, because completion duration is dominated by output token count and yours is not anybody else’s. Take the mean, and then size for the p95 of W rather than the mean if the distribution is long-tailed, which for generation it always is.

For reference, the defaults you are probably running under: httpx caps at 100 total connections with 20 kept alive and a five-second idle expiry (Encode, httpx resource limits), and undici’s Pool documents its connections option as defaulting to unlimited (Node.js, undici Pool documentation). Those two defaults fail in opposite directions — one silently queues, the other silently opens as many sockets as the workload asks for until the provider or the file-descriptor limit objects.

Streaming changes what you are counting

For a non-streaming call, W is the time to a complete response. For a streaming call the connection is held for the entire generation plus however long your consumer takes to drain it — so W is bounded below by the full completion time even though the useful work started at first token. A streaming workload therefore needs a pool sized by concurrent streams, and the number is larger than the equivalent non-streaming workload at the same request rate, not smaller.

The mistake this produces at a migration is subtle: teams move to streaming and to a new provider in the same release, observe better time-to-first-token, and size the pool against that improved number. The pool then exhausts under load because time-to-first-token is not W. If the consumer is slow, see streaming backpressure differences, because a slow consumer extends W directly and pool exhaustion is the first place it shows up.

HTTP/2 changes the unit again

Over HTTP/1.1 the pool’s unit is a connection and a connection carries one request at a time, so concurrency and connection count are the same number. Over HTTP/2 one connection multiplexes many streams, bounded by the server’s advertised concurrent-stream limit, and your effective concurrency is connections times that limit.

A pool of 100 connections is then either 100 concurrent requests or many thousands depending on a value the server chose, and moving between providers can change both the transport and the advertised limit. undici exposes allowH2 and maxConcurrentStreams as client options for exactly this reason. The practical rule is to know which transport you are on before reasoning about the pool at all, because the arithmetic in the first section applies to concurrent requests and only coincidentally to sockets.

The idle-timeout race, and why it duplicates work

This is the failure that costs money rather than latency. Both ends hold an idle timeout on a keepalive connection. If the client’s is longer than the server’s, there is a window in which the server has already sent a close and the client still believes the connection is usable. The client writes a request into a socket that is closing and gets a connection reset.

Most HTTP clients retry that automatically, because a reset on an idle connection looks exactly like a transient network fault. On a GET this is correct and invisible. On a completion request it is a second charged inference, and if the call has a side effect — a tool invocation, a webhook, a row written — it is a duplicated side effect. The provider’s idle timeout is not usually documented and changes between providers, so this appears at a migration as intermittent duplicate work with no error in your logs.

Two fixes, both required. Set the client’s keepalive expiry strictly below the shortest plausible server-side idle timeout so the client closes first and the race cannot happen; httpx’s keepalive_expiry is that knob. And restrict automatic retries to failures that occurred before any request bytes were written — connection-establishment errors — rather than to any reset, which means turning off the client’s blanket retry and doing it yourself with an idempotency key. The library covers the key mechanism in request idempotency.

The settings actually worth revisiting

• Maximum connections, re-derived from λ × W on the new provider’s measured duration, at p95 rather than mean.
• Maximum keepalive connections, which governs how much of your steady-state traffic pays a fresh TCP and TLS handshake. Set below the total but above your trough-hour concurrency.
• Keepalive expiry, strictly under the server’s idle timeout, per the section above.
• Connect timeout, which should be short and is frequently confused with the read timeout; a generation taking 90 seconds is not a connect problem.
• Read or headers timeout, which for streaming must be expressed as time between events rather than total duration, or every long completion fails.
• Retry policy, narrowed to pre-write failures, and aligned with your backoff — the subject of backoff parameter migration, since a pool that queues and a backoff that retries will happily amplify each other.