What a Load Test Should Check Before a Provider Migration Goes Live

A load test against an inference provider is not measuring your software. It is measuring an allocation you have been granted, under a traffic shape you have not sent yet, with backpressure that arrives as an HTTP status rather than as slowness.

The question the test has to answer

“Can it handle our load?” is unanswerable as posed, because the constraint is usually administrative rather than physical. The answerable version is three questions:

• At our peak sustained rate, what fraction of requests are rejected, and does that fraction grow over a long run or stay flat?
• When our arrival rate spikes to its observed maximum for its observed duration, does the client recover within the request’s deadline or does the queue collapse?
• Under that load, what happens to the slowest requests — not the average, which is insensitive to the failure you care about?

Everything below is in service of those three. A test that produces a single throughput figure has not answered any of them.

Building the profile from your own traffic

Synthetic prompts produce synthetic answers. Inference latency and inference cost both scale with token counts, so a load test built from a fixed 50-token prompt tells you about a workload you do not have.

1. Sample a week of production requests, stratified by route. Keep the distribution of input token counts and the distribution of output token counts, not just the means — these are usually heavily skewed, and the skew is what fills the provider’s token bucket.
2. Redact and freeze the sample as a fixture set. Reusing the same set against both source and target is what makes the two runs comparable at all.
3. Reproduce the arrival pattern, not just the mean rate. Extract the peak one-minute rate and the peak ten-second rate from your access logs; those two numbers are the burst you must survive.
4. Include the mix of streaming and non-streaming calls in the ratio you actually send, since a streaming call holds a connection for its whole duration and a batched one does not.
5. Include tool-calling turns. A single user action that fans out into four sequential model calls consumes four units of rate limit, and multi-step routes are usually a bigger share of the limit than of the request count.

Two limits, not one

Inference providers typically enforce a request-rate limit and a token-rate limit simultaneously, and which one you hit depends on the shape of your traffic rather than its volume. A workload of many small classifications hits the request limit; a workload of few large summarisations hits the token limit at a request rate that looks trivially low. Testing at your average request rate with short prompts can pass while production fails immediately.

Both limits are usually reported in response headers, and the header families differ by vendor. OpenAI documents x-ratelimit-limit-requests, x-ratelimit-remaining-requests, x-ratelimit-reset-requests and the corresponding -tokens variants; Anthropic documents an anthropic-ratelimit- family with the same limit/remaining/reset structure plus a retry-after on rejection. Capture whatever your target actually returns on every request during the test, because the remaining-count series over time is a far better description of your headroom than the pass/fail count.

# record the limit headers alongside every response
curl -sD headers.txt -o body.json \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  --data @request.json \
  https://api.example.com/v1/chat/completions

grep -i 'ratelimit\|retry-after' headers.txt

The general mechanism, including how a token bucket refills and why a limit expressed per minute is not a permission to send everything in the first second, is covered in rate limits explained.

Burst behaviour and what a 429 tells you

Run the burst separately from the soak, because they fail differently. The burst test sends your observed peak ten-second rate from cold and asks one question: does the client’s retry policy convert rejections into successful requests within the deadline, or does it convert them into a retry storm?

Watch for the failure mode where retries make things worse. A fixed retry delay synchronises every rejected caller onto the same millisecond, so the second attempt is as large a burst as the first. Exponential backoff with full jitter is what breaks that synchronisation, and honouring the retry-after header when one is present beats guessing. Record the number of attempts per logical request, not per HTTP call — a test that reports 100% eventual success after nine attempts each has found a problem, not proved its absence.

Distinguish the status codes rather than counting failures. A 429 is backpressure and is expected under a burst; a 5xx from an overloaded upstream is a capacity signal and should be retried more conservatively; a 400 under load is a bug in your batching. If the target returns a distinct overload error separate from its rate-limit error, treat those as different series in the readout, because the right response to each differs.

Then test what happens when it does not recover. Trip the circuit breaker deliberately during the burst and confirm the fallback path carries the traffic. A migration is exactly when a fallback gets used, and a fallback that has never been exercised at load is not a fallback.

Long responses, streams and disconnects

The soak test is where long-response behaviour surfaces. Run at your peak sustained rate for at least thirty minutes — long enough for a token bucket to be genuinely exhausted rather than merely dented, and long enough for connection pools and any provider-side warm capacity to reach steady state.

• Concurrent open streams. A streaming request occupies a connection for its full duration. Concurrency at steady state is approximately arrival rate multiplied by mean duration, so a route that averages twelve seconds at forty requests per second needs roughly 480 simultaneous connections. Check that against your own pool limits, your proxy’s, and any per-connection limit on the path.
• Client and proxy timeouts. Defaults differ between SDKs and between providers, and a load test is the cheapest place to discover that a load balancer idle timeout is shorter than your longest legitimate generation. See timeout defaults across providers.
• Cancellation. Abort a fraction of streams mid-flight, as real users do when they navigate away. Confirm the connection is actually released rather than held until completion, and confirm you are recording the partial usage — an abandoned stream is billed for what was generated.
• Token accounting under load. Where usage counts arrive on a separate stream event, verify they are still captured when the run is saturated. Losing usage under load is how a bill and a dashboard diverge; the failure is described in making token counting match the bill.

The readout

Report per scenario, not per run: achieved requests per second and tokens per minute; rejection rate broken down by status; attempts per logical request at the 50th and 99th percentile; time to first token and total duration at p50, p95 and p99; peak concurrent open connections; and the minimum observed remaining-tokens headroom. Then state the sustained rate at which the rejection rate crosses your tolerance — that single number is the capacity answer, and it is the only output of the test that a go/no-go decision can be made from. Comparing the latency half of that readout against the incumbent is a separate piece of arithmetic, done in comparing tail latency before and after a migration.