Measuring p50, p95 and p99 for LLM Calls

Averages describe symmetric distributions. Queueing produces distributions that are anything but, and inference is queueing all the way down. Every reporting mistake in this area comes from applying a statistic that assumes a shape the data does not have.

Why the mean fails here specifically

Consider 1,000 requests where 990 take 400 ms and 10 hit a cold start at 40 seconds:

mean = (990 * 0.4 + 10 * 40) / 1000  =  0.796 s
p50  = 0.4 s
p99  = 0.4 s          (the 990th value in sorted order)
p999 = 40 s

The mean is DOUBLE the experience of 99% of requests, and 50x better
than the experience of the worst 1%. It describes nobody.

This is not a contrived example — it is the standard shape of a system with cold starts, admission queueing and batch-composition effects. The distribution is multi-modal, and a mean of a multi-modal distribution lands in a valley between the modes, describing a request that does not occur.

There is a second, sharper argument. If a page makes ten model calls and each has a p99 of two seconds, the probability that a page hits at least one of those is 1 − 0.99^10 ≈ 9.6%. Your p99 tail is roughly one in ten of your users’ sessions. Tail latency is not an edge case in a system that fans out, which is why the tail is where the attention goes.

Percentiles of what, exactly

“p99 latency” for a streaming call is ambiguous, and the ambiguity hides the problem you have. Record these separately:

How many samples a p99 needs

A percentile is estimated from the observations that exceed it, and at the 99th percentile only 1% of your samples qualify. A rough and serviceable rule uses the standard error of a binomial count: with n samples you expect n·(1−q) exceedances, and you want enough of them that the count is not dominated by its own noise.

exceedances = n * (1 - q)
relative standard error ~= 1 / sqrt(exceedances)

want ~10% relative error -> need ~100 exceedances

  p50  ->  100 / 0.50  =    200 samples
  p95  ->  100 / 0.05  =  2,000 samples
  p99  ->  100 / 0.01  = 10,000 samples
  p999 ->  100 / 0.001 = 100,000 samples

A p99 computed over a 1-minute window at 5 requests/second has 300 samples
and 3 exceedances. It is not a percentile; it is the third-worst request,
and it will jump around wildly for reasons that are pure sampling noise.

The practical consequence is that low-traffic services should report p95 over a long window rather than p99 over a short one. A p99 on a one-minute window is a random variable with an enormous variance, and alerting on it produces pages that correlate with nothing. Widen the window until the exceedance count is respectable, and say which window you used.

Four ways aggregation lies

• Averaging percentiles. The mean of twelve five-minute p99s is not the hourly p99, and there is no weighting that makes it one — percentiles are not linear, so they do not compose. The only correct approach is to aggregate the underlying distribution: store histograms or sketches per window and merge those, which is exactly what HDR histograms and t-digest exist for. If your metrics system stores pre-computed percentiles per interval, every longer-window number it shows you is wrong.
• Mixing populations. One p99 across two models, three prompt-size buckets and both cold and warm paths is a percentile of a mixture, and it moves when the traffic mix changes even though no endpoint got slower. Segment first, then aggregate.
• Coordinated omission. If your load generator waits for a response before sending the next request, a slow response suppresses the requests that would have been sent during it — precisely the ones that would have been slow. The measured tail is far better than the real one. Send on a schedule and record intended start times, not actual ones; this is the single most common way a benchmark understates a tail.
• Censoring at the timeout. Requests killed by your own timeout are usually excluded from the latency series entirely, so the worst requests are the ones you do not count and every percentile is biased low. Record timeouts as observations at the timeout value, and report the timeout rate alongside the percentiles.

A note on the sketches themselves, since “store a histogram” hides a choice. Fixed-bucket histograms are exact about counts and approximate about values: your p99 is only as precise as the bucket it lands in, so buckets must be spaced logarithmically and must extend far enough that the tail is not all in the overflow bin. HDR histograms formalise that with a stated relative error across a stated range. t-digest takes the opposite approach, keeping more resolution near the extremes than in the middle, which is exactly the right bias here. Both merge correctly across windows and across hosts, which is the property that matters; a stored p99 does not.

One phrasing habit is worth adopting alongside them. “p99 under two seconds” and “99% of requests under two seconds” are the same statement, but only the second makes the denominator audible, and it is the denominator that people forget when the traffic is thin. State objectives in the second form and the sample-size question above answers itself.

What to record

The whole discipline reduces to: keep raw observations long enough to aggregate them properly, and keep the dimensions that let you split them.

per request, one row:
  model, provider, region          # dimensions to segment by
  prompt_tokens_bucket             # 0-1k, 1-8k, 8-32k, 32k+
  ttft_ms, total_ms
  output_tokens, ms_per_output_token
  max_inter_token_gap_ms
  queue_wait_ms                    # your own admission queue
  outcome                          # ok | timeout | 429 | 5xx | shed
  cold                             # boolean, if you can tell

then: histogram per (dimension tuple, minute); merge histograms across
      windows; compute percentiles from the merged histogram, never from
      other percentiles.

Report p50 for the typical experience, p95 for the one you are committing to, and p99 only where you have the traffic to support it. Publish the window and the sample count next to the number — a percentile without a denominator is not a measurement, and the habit of quoting both is what stops a dashboard from confidently reporting noise.

Alerting on these has one more trap. A percentile crossing a threshold for one evaluation window is, at realistic traffic volumes, mostly noise — which is why percentile alerts have a reputation for being ignored. Alert on a burn rate instead: how fast you are consuming the budget of allowed slow requests implied by your objective. “99% under two seconds” over a month permits a specific number of slow requests, and an alert that fires when you are spending them ten times faster than the month can afford is both quiet during normal operation and early during an incident. It also degrades gracefully at low traffic, because a burn rate over a long window still has a denominator when a five-minute p99 does not.

Finally, keep the outcome column in the same series as the timings. Latency and availability are usually reported by separate systems, and the separation hides the most common real-world pattern: a service whose latency looks excellent because its slowest requests are being shed. One table with both is the cheapest way to stop that story from being told.