Load Testing an LLM API With k6

The default shape of a k6 script — some virtual users, a request, a sleep — measures something, but on an LLM endpoint it usually does not measure what you meant. When response times triple, a VU-based script issues a third as many requests, so the load you are applying falls exactly when you wanted to know what happens under pressure.

Open and closed workload models

A closed model has a fixed number of concurrent users, each of which waits for its response before issuing the next request. Throughput is an output: it emerges from concurrency divided by response time. That is the right model for a fixed pool of callers — a batch job with a worker pool, or an internal tool with forty users.

An open model has requests arriving at a specified rate regardless of how long previous ones take. Throughput is an input. That is the right model for anything driven by external demand, which is almost every public API, and it is the only model that can show you a queue building.

The difference matters more on an LLM endpoint than on most services because latency there is both high and highly variable — hundreds of milliseconds to tens of seconds depending on output length. In a closed model a few slow generations throttle your entire load generator. k6 offers both, and the arrival-rate executors are the ones that implement the open model. Grafana documents them under k6 scenario executors.

The script

This uses constant-arrival-rate, loads a corpus of realistic prompts, and records token counts as custom metrics so they can carry thresholds of their own.

// load/chat.js
import http from 'k6/http';
import { check } from 'k6';
import { SharedArray } from 'k6/data';
import { Trend, Rate, Counter } from 'k6/metrics';

const BASE = __ENV.BASE_URL;
const KEY = __ENV.API_KEY;

// SharedArray keeps one copy in memory across all VUs rather than one per VU.
const prompts = new SharedArray('prompts', function () {
  return JSON.parse(open('./prompts.json'));
});

const outputTokens = new Trend('output_tokens');
const inputTokens = new Trend('input_tokens');
const truncated = new Rate('truncated_responses');
const tokensBilled = new Counter('tokens_billed');

export const options = {
  scenarios: {
    steady: {
      executor: 'constant-arrival-rate',
      rate: 20,
      timeUnit: '1s',
      duration: '5m',
      preAllocatedVUs: 200,
      maxVUs: 600,
    },
  },
  thresholds: {
    http_req_failed: ['rate<0.01'],
    http_req_waiting: ['p(95)<8000'],
    truncated_responses: ['rate<0.02'],
    dropped_iterations: ['count<1'],
  },
};

export default function () {
  const prompt = prompts[Math.floor(Math.random() * prompts.length)];

  const res = http.post(
    BASE + '/v1/chat/completions',
    JSON.stringify({
      model: 'gpt-4o-mini',
      messages: [
        { role: 'system', content: prompt.system },
        { role: 'user', content: prompt.user },
      ],
      max_tokens: 512,
      temperature: 0.2,
    }),
    {
      headers: {
        'Content-Type': 'application/json',
        Authorization: 'Bearer ' + KEY,
      },
      timeout: '120s',
      tags: { name: 'chat_completion', shape: prompt.shape },
    },
  );

  const ok = check(res, {
    'status is 200': (r) => r.status === 200,
    'has a choice': (r) => {
      if (r.status !== 200) return false;
      const body = r.json();
      return Array.isArray(body.choices) && body.choices.length > 0;
    },
  });

  if (!ok) return;

  const body = res.json();
  outputTokens.add(body.usage.completion_tokens);
  inputTokens.add(body.usage.prompt_tokens);
  tokensBilled.add(body.usage.total_tokens);
  truncated.add(body.choices[0].finish_reason === 'length');
}

Four things in there are deliberate. The timeout is 120 seconds, not the default, because an LLM request that takes 45 seconds is slow rather than failed, and a default timeout turns latency into an error rate and hides the real distribution. max_tokens is set, because otherwise your test cost is unbounded and your latency distribution is dominated by whichever request rambled. The shape tag lets you break every metric down by prompt category afterwards. And dropped_iterations carries a threshold: k6 increments that metric when it cannot start a scheduled iteration because every VU is busy, which means your load generator, not the system under test, was the bottleneck. A run with dropped iterations is a run whose numbers you must not quote.

Think time, and where it belongs

The row this page answers asks about modelling realistic think time rather than hammering flat out, and the honest answer has two halves.

If you are testing the endpoint, think time is not a thing. Requests arrive at whatever rate real users collectively produce, and that aggregate rate is what constant-arrival-rate takes as its rate parameter. Adding a sleep to an arrival-rate scenario does not reduce the arrival rate — it makes each iteration hold a VU longer, so you need more preAllocatedVUs to sustain the same rate, and nothing else changes.

If you are testing a conversation — several turns where each depends on the last, and the realistic pattern is a user reading a reply before typing again — then think time is real and belongs inside a closed-model scenario, because you are modelling a population of sessions rather than a stream of requests. Draw the gap from an exponential distribution rather than a uniform one; human inter-action gaps are much better modelled as memoryless, and a uniform sleep produces artificial synchronisation across VUs.

import { sleep } from 'k6';

// Exponentially distributed think time with the given mean, in seconds.
// Inverse-transform sampling: -ln(U) / lambda, with lambda = 1 / mean.
function thinkTime(meanSeconds) {
  return -Math.log(1 - Math.random()) * meanSeconds;
}

export const options = {
  scenarios: {
    conversations: {
      executor: 'ramping-vus',
      startVUs: 0,
      stages: [
        { duration: '2m', target: 100 },
        { duration: '5m', target: 100 },
        { duration: '1m', target: 0 },
      ],
      gracefulRampDown: '120s',
    },
  },
};

export default function () {
  const turns = 1 + Math.floor(Math.random() * 4);
  const history = [];
  for (let i = 0; i < turns; i++) {
    // ... issue the request, push the reply into history ...
    sleep(thinkTime(8));
  }
}

Run both, and report them as answers to different questions. The open scenario tells you the rate at which the system falls over; the closed one tells you how many concurrent conversations it supports at an acceptable latency.

Metrics that mean something here

• http_req_waiting, not http_req_duration. Waiting is time from the request being sent to the first byte of the response, which on a non-streaming completion is essentially the whole server-side generation. Duration adds body download, which for a JSON response is noise. If you later test the streaming endpoint, waiting becomes time to first token, and it is the number users feel.
• Percentiles, and the maximum. LLM latency distributions are long-tailed and often multi-modal, because a cached prefix and a cold one take very different paths. A mean over that is meaningless. Look at p50, p95, p99 and max, and if p99 and max are far apart, look for what is different about those requests rather than dismissing them.
• Output tokens as a first-class metric. Latency on a generation endpoint is mostly a function of how many tokens came back. If your p95 latency rose between two runs, the first thing to check is whether p95 output length rose, because that is a change in what you asked for rather than a regression in the service.
• Error taxonomy, not an error rate. Tag 429s separately from 500s separately from timeouts. A load test that reports 8% errors where all of them are rate limits has found your quota, not your capacity; see how rate limits work.

What the run costs

A load test against a metered API spends real money and it is worth computing before you press go rather than discovering it on an invoice. From the scenario above — 20 requests per second for five minutes — and assuming a corpus averaging 1,000 input tokens and responses averaging 400 output tokens, at list prices of $3 per million input and $15 per million output tokens:

requests      = 20 * 300 s          =  6,000
input tokens  = 6,000 * 1,000       =  6,000,000
output tokens = 6,000 * 400         =  2,400,000

input cost    = 6.0M / 1M  * $3     =  $18.00
output cost   = 2.4M / 1M  * $15    =  $36.00
                                      -------
per run                               $54.00

A five-step ramp to find the knee, each run at this size:  $270
Nightly for a month:                                     $1,620

Token counts and prices are inputs — substitute your corpus and your contract. The conclusion usually survives the substitution: a load test that is cheap to run against a stateless HTTP service is not cheap against a generation endpoint, and it should be scheduled rather than left on a nightly cron out of habit.

What k6 will not do for you

Streaming is the significant gap. k6’s built-in HTTP module returns a complete response, so it cannot observe inter-token timing on a server-sent-event stream; you get time to first byte and total time and nothing in between. There is a community extension exposing an SSE module under k6/x/sse, maintained outside the core project, and it is the right place to look if streaming behaviour under load is the thing you need. If you would rather not depend on an extension, the equivalent test in a tool with a native streaming client is set out in load testing a streaming endpoint with Locust.

The other gap is semantic. k6 checks status codes and JSON structure; it has no view on whether the answers were any good, and under load that is a real question — providers degrade by routing to different capacity, and a test that only counts 200s will report success while quality falls. Keep the quality question in the eval suite and the capacity question here, and do not let a green load test stand in for either.