Comparing Tail Latency Before and After a Provider Migration

The mean latency of an inference endpoint is close to useless and the p99 is easy to compute wrongly. Both problems have the same cause: generation time is dominated by output length, which is itself a random variable you did not hold constant.

What a percentile is measuring here

A p99 of 4.2 seconds means: sort every observed request duration ascending, and the value at the 99th percentile position is 4.2 seconds. One request in a hundred was slower. It is a statement about your sample, not a property of the provider, and it changes when the sample changes — which is exactly the trap in a before-and-after comparison, because the two samples are collected at different times under different conditions.

For an inference call, decompose the duration before comparing anything. Total time is approximately time to first token plus output tokens divided by the generation rate. Those two terms have different causes: time to first token is dominated by prefill and by queueing, and grows with input length; the second term grows with output length and is roughly linear in it. A provider can be worse on one and better on the other, and a single total-latency percentile hides which. Record both, always.

The fixed request set

Freeze a set of real requests — sampled from production, redacted, stored as fixtures — and send the same set to both providers. Sampling fresh traffic for each side means comparing two different input distributions, and input length moves time to first token directly.

1. Sample requests stratified by input token count, so the set spans the range you actually serve rather than clustering at the mode.
2. Pin the output cap to the same value on both sides, and set temperature to 0 on both. This does not make the two models produce the same text — they will not — but it removes sampling variance as a source of length difference.
3. Interleave the two providers request by request rather than running one set after the other. Two sequential runs differ by whatever changed in between, including the provider’s own load; interleaving makes the time-of-day confound common to both.
4. Run from the same client, in the same region, over the same number of connections, with retries disabled. A retried request’s duration is a different quantity and will distort the tail more than anything else in the experiment.
5. Record per request: input tokens, output tokens, time to first token, total duration, status, and the model string that served it.

How many samples a p99 needs

This is where most comparisons go wrong, and the arithmetic is simple enough to do on the page. Estimating the qth quantile requires enough samples that the region beyond it is populated. With n samples, the expected count above the 99th percentile is n multiplied by 0.01. At n = 100 that expectation is 1: your p99 is a single observation, and a single observation has no stability at all. At n = 1,000 it is 10 observations; at n = 10,000 it is 100.

A usable rule follows from the binomial spread. The count above the quantile has standard deviation of approximately the square root of n times p times (1 − p). With p = 0.01 and n = 1,000 that is about 3.1 against an expected 10 — roughly 31% relative noise on the tail population. At n = 10,000 it is about 9.9 against 100, or roughly 10%. Those relative figures are the honest precision of your p99, and they are why a p99 difference of a few percent between two providers, measured on a thousand requests each, is not a difference.

• p50 — a few hundred samples per side is comfortable.
• p95 — aim for at least 2,000 per side; the expected tail population is then 100.
• p99 — aim for at least 10,000 per side. Below a few thousand, report p95 and say so rather than reporting a p99 you cannot support.

Report a confidence interval rather than a point. The distribution-free way is the order-statistic interval: with n samples sorted ascending, the true qth quantile lies between the samples at ranks approximately nq − z√(nq(1−q)) and nq + z√(nq(1−q)), with z = 1.96 for 95% confidence. For n = 10,000 and q = 0.99 that is ranks 9,900 ± 19 — so quote the values at ranks 9,881 and 9,919 as the interval. If the two providers’ intervals overlap, you have not measured a difference.

Normalising for output length

Two providers given the same prompt will not produce the same number of output tokens, and one that is simply more verbose will look slower on total duration even at an identical generation rate. Worse, the two tokenizers may count the same text differently, so “output tokens” is not even the same unit on both sides — the mechanism is in tokenizer comparison.

So compute three series per side, and compare each:

• Time to first token. Independent of output length. This is the cleanest cross-provider comparison available and the one that dominates perceived responsiveness for anything streamed.
• Milliseconds per output token, computed as (total duration − time to first token) divided by output tokens. This is the generation rate, and it is the number that answers “is the new provider actually slower” independent of verbosity.
• Total duration. What the user experiences, verbosity included. Keep it, but never as the only series — a regression here with no regression in the other two means you have a prompt problem, not a provider problem.

A worked example with assumed inputs, to show the shape: assume the incumbent returns time to first token at p95 of 600 ms and 18 ms per output token, and the candidate returns 900 ms and 12 ms per output token. For a 100-token answer the incumbent takes about 2,400 ms and the candidate about 2,100 ms; for a 20-token answer the incumbent takes about 960 ms and the candidate about 1,140 ms. Same two providers, opposite conclusions, decided entirely by the output length distribution of your traffic. Substitute your own measured numbers — the two above are illustrative placeholders, not observations — and evaluate at your actual median and p95 output lengths.

What the tail does to a multi-call route

A per-request p99 understates what a user-facing route experiences, because a route usually makes more than one model call. If a route makes k sequential calls and each independently has probability 0.01 of landing beyond the per-call p99, the probability that at least one does is 1 − 0.99ⁿ. For k = 3 that is about 3.0%; for k = 5, about 4.9%; for k = 10, about 9.6%. In other words a ten-step agent loop meets a per-call p99 event on roughly one request in ten.

The same arithmetic applies to fan-out. When a request issues k parallel calls and must wait for all of them, its duration is the maximum of k draws, so the route’s median tracks the per-call high percentiles rather than the per-call median. Independence is an assumption here and usually an optimistic one — correlated slowness during a provider incident makes the real figure worse, not better.

The practical consequence for a migration: set the acceptance threshold on the route’s end-to-end percentile, computed from your own call graph, not on the per-call number the provider comparison produces. Wiring that threshold into CI so a future change cannot quietly cross it is covered in latency regression thresholds.

Deciding from the spread

Look at the ratio of p99 to p50 on each side, not only at the absolute values. A provider with a p50 of 800 ms and a p99 of 2,400 ms has a tail ratio of 3; one with a p50 of 600 ms and a p99 of 6,000 ms has a ratio of 10. The second is faster on average and much harder to build a timeout policy around, because there is no timeout value that both catches genuine hangs and does not cut off legitimate slow requests. For anything with a deadline, the narrower spread is usually worth more than the lower median.

Then be explicit about what the experiment cannot tell you. It was run at test load, not production load; providers behave differently when your own traffic is a rounding error on their capacity than when it is not. It was run over a window, and inference latency has a strong time-of-day component. Treat the result as a gate that a candidate must pass, and keep measuring from real traffic after the cutover — the post-migration series is the only one that answers the question you actually asked.