A Test That Fails When Provider Latency Passes a Regression Threshold

Somebody adds a retrieval step, a longer system prompt or a second tool schema, and every request gets 300ms slower. No test fails, because no test looks at time. Six weeks later the p95 has doubled and nobody can say which change did it.

Why the obvious assertion flakes

The first attempt is always expect(ms).toBeLessThan(2000), and it fails within a week for reasons that have nothing to do with your code. A single sample of an inference call is drawn from a distribution with a long tail: queueing at the provider, cold routing, a noisy CI runner and network variance all land in that one number. The threshold then gets raised until it stops failing, at which point it is above any regression it might have caught and is pure ceremony.

Three properties are needed to make a latency assertion mean something. It must be a percentile of several samples rather than one measurement. It must compare against something measured in the same run, so provider-wide slowness cancels out. And it must be a ratio to a committed baseline rather than an absolute number, so “is this slower than it was?” is the question rather than “is this slow?”

Two latencies, measured separately

Averaging time-to-first-token and total time into one number hides which one moved, and the two have unrelated causes. Time to first token is dominated by prefill and therefore by prompt length: a longer system prompt, more tool schemas, more retrieved context all push it up. Total time adds the generation, which is driven by output length and the per-token rate. A change that adds 4,000 tokens of context and a change that makes the model answer twice as verbosely both look like “slower” and need different fixes.

For a streaming call, time to first token is the interval from sending the request to the first delta event that carries content. Be precise about which event you count: a stream typically opens with a message start and one or more block starts before any text arrives, and starting the clock stop at the wrong event gives you a number that is stable, meaningless and lower than the truth.

// src/measure.ts
export async function measure(run: () => AsyncIterable<StreamEvent>) {
  const t0 = performance.now();
  let ttft: number | null = null;
  let outputTokens = 0;

  for await (const ev of run()) {
    if (ttft === null && ev.type === "content_block_delta") ttft = performance.now() - t0;
    if (ev.type === "message_delta") outputTokens = ev.usage.output_tokens;
  }
  return { ttftMs: ttft!, totalMs: performance.now() - t0, outputTokens };
}

export function percentile(values: number[], p: number) {
  const sorted = [...values].sort((a, b) => a - b);
  const idx = Math.min(sorted.length - 1, Math.ceil((p / 100) * sorted.length) - 1);
  return sorted[idx];
}

Record outputTokens alongside the timings and normalise. Total time divided by output tokens is milliseconds per token, and a change that makes the model produce 40% more text will move total time without moving that ratio — which tells you immediately that the regression is in verbosity rather than in speed, and sends you to a different fix.

The control is what makes it a test

This is the part that separates a useful latency test from a flaky one. In the same job, at roughly the same moment, measure a fixed control request — the same model, a frozen prompt of known length, a pinned max_tokens, never edited — and express your result as a ratio to it.

If the provider is having a slow afternoon, both numbers rise and the ratio does not move. If your change made your prompt longer, your number rises and the control does not, and the ratio moves. That is exactly the discrimination you want, and it is unavailable to any assertion against a constant. Interleave the control and the subject rather than running all of one then all of the other, so a slowdown partway through the job affects both equally.

The test

import { describe, it, expect, beforeAll } from "vitest";
import { readFileSync } from "node:fs";
import { measure, percentile } from "../src/measure";

const baseline = JSON.parse(readFileSync("perf/baseline.json", "utf8"));
const SAMPLES = 15;
const TOLERANCE = 1.25;   // 25% slower than baseline before failing

async function samples(fn: () => AsyncIterable<StreamEvent>) {
  const out = [];
  for (let i = 0; i < SAMPLES; i++) out.push(await measure(fn));
  return out;
}

describe("latency regression", () => {
  let ratioTtft: number;
  let ratioPerToken: number;

  beforeAll(async () => {
    // Interleaved so provider-wide variance hits both equally.
    const control = [];
    const subject = [];
    for (let i = 0; i < SAMPLES; i++) {
      control.push(await measure(controlRequest));
      subject.push(await measure(productionRequest));
    }
    const cT = percentile(control.map((r) => r.ttftMs), 95);
    const sT = percentile(subject.map((r) => r.ttftMs), 95);
    ratioTtft = sT / cT;

    const perToken = subject.map((r) => r.totalMs / Math.max(1, r.outputTokens));
    ratioPerToken = percentile(perToken, 95) /
      percentile(control.map((r) => r.totalMs / Math.max(1, r.outputTokens)), 95);
  });

  it("p95 time to first token has not regressed against the control", () => {
    expect(ratioTtft).toBeLessThan(baseline.ratioTtft * TOLERANCE);
  });

  it("p95 milliseconds per output token has not regressed", () => {
    expect(ratioPerToken).toBeLessThan(baseline.ratioPerToken * TOLERANCE);
  });
});

Fifteen samples is enough for a p95 to be indicative and not enough for it to be precise, which is why the tolerance is 25% rather than 5%. Be honest about that trade: this test catches a change that made things meaningfully slower, and it will not catch a 6% creep. Catching the creep is a production monitoring job, against real traffic, over days — see defining latency SLOs. The test exists to stop the large, obvious regression from reaching the monitoring in the first place.

Owning the baseline

The committed perf/baseline.json is the part that decays if nobody looks after it, and there is one rule: updating it is a deliberate, reviewed commit with a sentence explaining what made the number legitimately worse. Never regenerate it automatically on a passing run — a baseline that follows the measurement can only ever ratify the current state, and a 5% regression every sprint is invisible to it forever.

Three things will make a run legitimately slower without any code of yours getting slower, and each needs a decision rather than a re-baseline. A retry that fired inside the SDK adds a full round trip to a single sample and will drag a p95 badly at fifteen samples; record the attempt count alongside each measurement and discard samples that retried, or the test is measuring the provider’s error rate. Concurrency in the harness inflates everything, because fifteen requests in flight at once queue behind each other at the provider — run the samples sequentially, and if that makes the job too slow, reduce the sample count rather than parallelising. And a cached prefix makes a request faster in a way that has nothing to do with the change under test: if prompt caching is enabled, the first sample pays a cache write and the rest read, so either discard the first sample or disable caching for this suite and say which you did.

Two practical additions. Print the measured ratios in the test output on success as well as failure, so the trend is visible in CI logs even while everything is green; a number that has drifted from 1.02 to 1.19 without failing is worth knowing about before it crosses. And record the model id and the date next to the baseline, because a provider serving the same model id on different hardware makes the old number incomparable — which is not a bug in your code and should be resolved by re-baselining with a note, not by widening the tolerance. The same discipline applies to quality thresholds, which is why the machinery is worth sharing with production quality regression tracking.