Simulating a Realistic Prompt Length Distribution in a Load Test

A load test that sends the same 200-token prompt fifty thousand times measures one point on a curve and reports it as the curve. Prompt length drives prefill time, which drives time to first token, which is most of what your p95 is made of — and it drives the input half of the bill directly. If the test’s length distribution does not look like production’s, neither number transfers.

Why one fixed prompt gives you the wrong number

Three separate errors compound when the prompt is constant.

• Latency is non-linear in the wrong place. Prefill cost grows faster than linearly with sequence length because attention is quadratic in it. Testing at the mean length and multiplying does not recover the tail; the long prompts that produce your worst requests are exactly the ones a mean-length test never issues.
• Prompt caching gets a free ride. Identical prompts share a prefix perfectly. Providers that cache prefixes will serve your second request onwards from a warm cache, so a constant-prompt test can report a time to first token that no real user will ever see. This one is worth naming explicitly because it makes the test look good.
• Batching behaves differently. A server batching uniform requests fills its batches neatly. Mixed lengths cause padding and scheduling effects that only show up under mixed load.

The fix is not to guess a better fixed length. It is to sample from a distribution taken from your own traffic, so the test issues the same proportion of 80-token and 9,000-token requests that production does.

The shape real prompt lengths have

Prompt length is a positive quantity with a floor, a dense body and a long right tail: a great many short turns, a moderate number of medium ones, and a thin tail of enormous requests carrying a pasted document or a full conversation history. That shape is what a log-normal distribution is for, and it is usually a good enough fit to be useful without pretending to be a model of anything.

You do not have to fit it analytically, and for most teams the better option is not to. If you have request logs with a token count on them, the empirical distribution is the distribution: export the counts, keep them in a file, and sample from that file. That removes every modelling assumption at once. Use a fitted log-normal only when you have no logs yet — a new endpoint, a new feature — and say in the test that you did.

# From your own request logs, one integer per line.
# psql -Atc "select input_tokens from requests
#   where created_at > now() - interval '7 days'" > prompt-tokens.txt

import statistics
lengths = [int(line) for line in open("prompt-tokens.txt")]
lengths.sort()

def pct(p):
    return lengths[int(len(lengths) * p) - 1]

print("n      ", len(lengths))
print("median ", statistics.median(lengths))
print("p90    ", pct(0.90))
print("p99    ", pct(0.99))

Look at the ratio between the median and the p99 before you go further. If it is a factor of two, prompt length is not your problem and a fixed prompt would have been fine. If it is a factor of thirty — which is ordinary for anything that lets a user paste text — then a single-length test has been telling you a story about the wrong endpoint.

Sampling lengths in a locustfile

Locust’s documented user class is HttpUser, with tasks declared by the @task decorator and pacing by wait_time. The part that needs care on an LLM endpoint is timing: a streamed response has two latencies — time to first token and total time — and the HTTP client can only report one per request. Locust’s documented escape hatch is to fire the request event yourself, with the keyword arguments request_type, name, response_time, response_length, response, context and exception. Firing it twice under two names gives you two separate percentile tables.

# locustfile.py
import random, time
from locust import HttpUser, task, between

# Sampled from real logs. Loaded once per worker process.
CORPUS = open("corpus.txt").read().split()
LENGTHS = [int(l) for l in open("prompt-tokens.txt")]

def sample_prompt():
    # ~0.75 words per token is a rough English ratio; if you need the
    # real count, encode with the tokenizer your provider documents.
    target_tokens = random.choice(LENGTHS)
    words = max(1, int(target_tokens * 0.75))
    body = " ".join(random.choice(CORPUS) for _ in range(words))
    return target_tokens, "Summarise the following in one sentence.\n\n" + body

class ChatUser(HttpUser):
    wait_time = between(1, 3)

    @task
    def chat(self):
        target_tokens, prompt = sample_prompt()
        bucket = "small" if target_tokens < 1000 else "large"
        start = time.perf_counter()
        first = None
        chars = 0
        try:
            with self.client.post(
                "/v1/chat/completions",
                json={
                    "model": "your-model",
                    "messages": [{"role": "user", "content": prompt}],
                    "max_tokens": 256,
                    "stream": True,
                },
                headers={"Authorization": "Bearer " + self.environment.parsed_options.api_key},
                stream=True,
                name="/v1/chat/completions [" + bucket + "]",
                catch_response=True,
            ) as response:
                for line in response.iter_lines():
                    if not line:
                        continue
                    if first is None:
                        first = time.perf_counter()
                    chars += len(line)
                if first is None:
                    response.failure("no tokens received")
                else:
                    response.success()
        except Exception as exc:
            self.environment.events.request.fire(
                request_type="POST", name="ttft [" + bucket + "]",
                response_time=(time.perf_counter() - start) * 1000,
                response_length=0, response=None, context={}, exception=exc,
            )
            return

        self.environment.events.request.fire(
            request_type="POST", name="ttft [" + bucket + "]",
            response_time=(first - start) * 1000,
            response_length=chars, response=None, context={}, exception=None,
        )

Two details in there are load-bearing. The name argument buckets requests into separate statistics rows, so you get percentiles for short and long prompts rather than one blended table in which the long tail is invisible. And the corpus is shuffled words rather than one repeated paragraph, because a repeated paragraph re-creates the prompt-cache problem the whole exercise exists to avoid.

Output length is a second distribution

Sampling the input and then pinning max_tokens to a constant fixes half the problem and quietly biases the other half. Output tokens are generated one at a time, so they dominate total time and they are usually the expensive half of the bill. A test where every response is exactly 256 tokens will report a total-time p95 with no spread in it at all.

Sample max_tokens from your logged completion lengths too, and be aware that this is a ceiling rather than a target: the model stops when it stops. To get a realistic spread of actual output lengths you need prompts that ask for realistically varied amounts of text, which is another argument for building the corpus out of real requests with the sensitive parts removed rather than out of a lorem-ipsum generator. If you are pulling real requests into fixtures, treat that as a data-handling decision as much as a testing one: redact before the corpus is committed, and prefer synthesised text of the right length over real user content wherever the content itself does not matter to the measurement.

Reading the result without fooling yourself

Report per-bucket percentiles, never a mean. A mean over a heavy-tailed distribution is dragged past almost every request that actually happened, and on this workload the tail is the whole point.

Two derived numbers are worth computing from the same run. The first is cost per thousand requests: multiply the mean sampled input length by the input price, add the mean observed output length times the output price, and multiply by a thousand. State both prices and the date you read them in the same sentence as the result, because they move. The second is the ratio of time to first token to total time in each bucket, which tells you whether prefill or decode is your constraint and therefore whether shortening prompts or shortening answers is the lever. On a large-prompt bucket those often point in opposite directions, and a blended report hides that completely.

Finally, keep the sampled corpus and the length file in version control next to the locustfile. A load test whose input distribution changed silently between runs is a benchmark that cannot be compared to itself, and the comparison is the reason you ran it twice.