Load Testing an AI Endpoint

A load test of an inference endpoint that reports requests per second and a mean latency has measured almost nothing useful. Responses are streamed, so a user experiences two latencies rather than one; service time varies by an order of magnitude with output length; and the way most load tools generate traffic systematically hides overload. This page is about all three, and ends with a harness that avoids them.

Why the usual load test lies here

Three properties of AI endpoints break the assumptions built into general-purpose load tools.

• Service time is not roughly constant. A request producing 20 tokens and one producing 2,000 differ by a hundred times in cost and duration. Averaging them produces a number that describes neither, and a test whose output-length mix differs from production is testing a different service.
• The response is a stream, not an event. The user sees the first token long before the last. A single latency figure collapses the one number that decides perceived responsiveness into the one that decides throughput — time to first token versus tokens per second is the distinction.
• Real users think between requests. A conversation is a request, then twenty seconds of reading, then another request. A test with no think-time models a different arrival process entirely, and arrival process is what queueing behaviour depends on.

Open loop and closed loop

This is the distinction that decides what your test can tell you, and most tools implement only one of them without saying which.

They give different answers about the same system, and both are correct about different questions. A closed-loop test is self-limiting: it cannot drive the service into overload, because slowing the service slows the test. That makes it excellent for measuring capacity at a stable operating point and useless for measuring what happens beyond it.

An open-loop test can drive the queue past capacity, which is exactly the region where you find out whether your admission control works. If you are validating the bound and shedding behaviour from priority queueing, it must be an open-loop test — a closed-loop one will never reach the condition.

Run both, and label the results. “100 concurrent users” and “40 requests per second” are not two ways of saying the same thing.

Coordinated omission, worked

Coordinated omission is the measurement error that makes closed-loop results look far better than reality. The name is Gil Tene’s, and the mechanism is simple: when the system stalls, the test stops issuing requests, so the requests that would have experienced the stall are never sent — and therefore never measured.

Worked example. One virtual user, intending one request per second, over
a 10-second window. The service stalls for 5 seconds starting at t=2.

Closed loop, measuring from send to receive:
  t=0  request → 0.1 s      (recorded 0.1)
  t=1  request → 0.1 s      (recorded 0.1)
  t=2  request → 5.0 s      (recorded 5.0)   ← blocks until t=7
  t=7  request → 0.1 s      (recorded 0.1)
  t=8  request → 0.1 s      (recorded 0.1)
  t=9  request → 0.1 s      (recorded 0.1)
  Six samples. p95 ≈ 5.0 s, but the median is 0.1 s and the mean is 0.92 s.
  The four requests that a real user would have made at t=3,4,5,6 do not
  exist in the data set.

Open loop, measuring from INTENDED send time to receive:
  t=0  → 0.1        t=5  → 2.0   (queued from t=5, served at t=7)
  t=1  → 0.1        t=6  → 1.0
  t=2  → 5.0        t=7  → 0.1
  t=3  → 4.0        t=8  → 0.1
  t=4  → 3.0        t=9  → 0.1
  Ten samples. p95 = 5.0 s, median = 1.0 s, mean = 1.55 s.

Same outage, same service. The closed-loop median understates the user
experience by a factor of ten, because the samples that would have shown
the stall were never taken.

The correction is one line of code and it is the most important line in any load harness: record latency from the time the request was scheduled to be sent, not from the time it was actually sent. If your generator falls behind schedule, that lateness belongs in the measurement, because a real user would have experienced it.

The metrics a streaming endpoint needs

Report percentiles, never means. One cold start drags a mean past every request anyone actually experienced — latency percentiles covers why p50, p95 and p99 answer different questions.

The harness

Open-loop, Poisson arrivals, streaming-aware, coordinated-omission free. Standard library plus aiohttp for connection pooling.

# loadtest.py — open-loop harness for a streaming chat endpoint.
# python -m pip install aiohttp
import argparse, asyncio, json, os, random, statistics, time
import aiohttp

BASE  = os.environ["LLM_BASE_URL"]        # e.g. https://host/v1
KEY   = os.environ["LLM_API_KEY"]
MODEL = os.environ.get("LLM_MODEL", "gpt-4o-mini")

class Result:
    __slots__ = ("scheduled", "ttft", "end", "tokens", "itls", "error")
    def __init__(self, scheduled):
        self.scheduled = scheduled
        self.ttft = None; self.end = None
        self.tokens = 0; self.itls = []; self.error = None

async def one_request(session, prompt, max_tokens, scheduled, results):
    r = Result(scheduled)
    body = {"model": MODEL, "stream": True, "max_tokens": max_tokens,
            "messages": [{"role": "user", "content": prompt}]}
    last = None
    try:
        async with session.post(BASE + "/chat/completions", json=body) as resp:
            if resp.status != 200:
                r.error = f"http_{resp.status}"
                r.end = time.perf_counter(); results.append(r); return
            async for raw in resp.content:
                line = raw.decode("utf-8").strip()
                if not line.startswith("data:"):
                    continue
                payload = line[5:].strip()
                if payload == "[DONE]":
                    break
                delta = (json.loads(payload)["choices"][0].get("delta") or {})
                piece = delta.get("content")
                if not piece:
                    continue
                now = time.perf_counter()
                if r.ttft is None:
                    r.ttft = now - scheduled      # from SCHEDULED, not from send
                else:
                    r.itls.append(now - last)
                last = now
                r.tokens += 1
    except asyncio.TimeoutError:
        r.error = "timeout"
    except Exception as exc:
        r.error = type(exc).__name__
    r.end = time.perf_counter()
    results.append(r)

async def run(rate, duration, prompts, max_tokens, concurrency_limit=2000):
    """Open loop: arrivals are a Poisson process at 'rate' per second.
    Nothing here waits for a response before scheduling the next arrival."""
    results, tasks = [], []
    timeout = aiohttp.ClientTimeout(total=180, sock_read=60)
    conn = aiohttp.TCPConnector(limit=concurrency_limit)
    headers = {"Authorization": "Bearer " + KEY}
    async with aiohttp.ClientSession(connector=conn, timeout=timeout,
                                     headers=headers) as session:
        t0 = time.perf_counter()
        next_at = t0
        while next_at - t0 < duration:
            gap = random.expovariate(rate)        # exponential inter-arrival
            next_at += gap
            sleep_for = next_at - time.perf_counter()
            if sleep_for > 0:
                await asyncio.sleep(sleep_for)
            # scheduled time is next_at, EVEN IF we are late getting here
            tasks.append(asyncio.create_task(one_request(
                session, random.choice(prompts), max_tokens, next_at, results)))
        await asyncio.gather(*tasks)
    return results

def pct(xs, p):
    if not xs: return float("nan")
    xs = sorted(xs)
    k = max(0, min(len(xs) - 1, int(round((p / 100) * (len(xs) - 1)))))
    return xs[k]

def report(results, duration):
    ok = [r for r in results if r.error is None and r.ttft is not None]
    ttft = [r.ttft for r in ok]
    e2e  = [r.end - r.scheduled for r in ok]
    itl  = [x for r in ok for x in r.itls]
    toks = [r.tokens for r in ok]
    errs = {}
    for r in results:
        if r.error: errs[r.error] = errs.get(r.error, 0) + 1
    print(f"requests      {len(results)}  ok {len(ok)}  errors {len(results)-len(ok)}")
    print(f"achieved rate {len(results)/duration:.2f} req/s")
    for name, xs in (("ttft", ttft), ("e2e", e2e), ("itl", itl)):
        if xs:
            print(f"{name:5s} p50 {pct(xs,50)*1000:8.1f} ms   "
                  f"p95 {pct(xs,95)*1000:8.1f} ms   p99 {pct(xs,99)*1000:8.1f} ms")
    if toks:
        print(f"output tokens p50 {pct(toks,50)}  p95 {pct(toks,95)}  "
              f"total {sum(toks)}  ({sum(toks)/duration:.1f} tok/s)")
    for k, v in sorted(errs.items()):
        print(f"error {k:16s} {v}")

if __name__ == "__main__":
    ap = argparse.ArgumentParser()
    ap.add_argument("--rate", type=float, required=True, help="requests/second")
    ap.add_argument("--duration", type=float, default=120.0)
    ap.add_argument("--max-tokens", type=int, default=400)
    ap.add_argument("--prompts", default="prompts.txt")
    a = ap.parse_args()
    prompts = [l.strip() for l in open(a.prompts) if l.strip()]
    res = asyncio.run(run(a.rate, a.duration, prompts, a.max_tokens))
    report(res, a.duration)

The three lines that make it honest: random.expovariate(rate) for exponentially distributed inter-arrival times rather than a fixed tick, because real arrivals are bursty and a fixed tick understates queueing; scheduling the next arrival before awaiting the previous response, which is what makes it open-loop; and computing TTFT from next_at rather than from the moment the request was actually issued, which is the coordinated-omission fix.

Modelling the workload

The harness is the easy half. The traffic model is what makes the result mean something, and there are four inputs to get right.

1. Prompt length distribution. Take it from production logs, not from a fixed sample sentence. Prefill cost scales with input length, so a test using 50-token prompts against a service that normally sees 4,000-token prompts is measuring a different machine.
2. Output length distribution. Do not set max_tokens to one value for every request. Sample it from your real distribution, which is usually long-tailed, because the tail is what fills the batch and blocks the queue.
3. Cache-hit ratio. If you use prompt caching, a test that sends the same prompt repeatedly will hit the cache on almost every request and report throughput you will never see. Include genuinely distinct prompts in the proportion production has.
4. Think time, for closed-loop runs. Model it as a distribution, not a constant — reading a 400-token answer takes tens of seconds and varies enormously. A closed-loop test with zero think-time and 100 users is not 100 users, it is a much smaller number of much busier ones.

Reading the results

Run a rate ladder — say 2, 4, 8, 12, 16, 20 requests per second, a few minutes each with a gap between — and plot TTFT p95 against rate. The shape tells you everything.

• Flat, then a knee, then vertical. The normal shape. The knee is your capacity. Operate below it with headroom, because past the knee latency rises without bound for a small increase in load.
• Rising from the start. You are already at capacity at the lowest rate tested, or a per-request constant (a cold connection, a synchronous dependency) dominates. Retest with a lower floor.
• Flat past the point you expected the knee. Check the achieved rate matches the requested rate. A flat curve with a gap between the two is the generator failing, not the service succeeding.
• Error rate rising before latency. Good — that is admission control working. Confirm the errors are 429s with Retry-After and not timeouts or 5xx.

Finally, take the capacity number from the knee and feed it back into the sizing arithmetic in GPU autoscaling: the per-replica service rate in that derivation is exactly what this test measures, and the two pages are only useful together.