Load Testing a Streaming Endpoint With Locust

Point Locust at a streaming chat endpoint with the obvious script and the report will show a median response time of forty milliseconds and a payload size of zero bytes. Neither number is wrong. Both are answers to a question you did not ask, and the reason is two specific lines in Locust’s HTTP session.

What the default report is telling you

Write the naive version and the numbers look extraordinary:

# Do not use this.
from locust import HttpUser, task, between

class NaiveUser(HttpUser):
    wait_time = between(1, 5)

    @task
    def chat(self):
        self.client.post("/v1/chat/completions", json={
            "model": "gpt-4o-mini",
            "messages": [{"role": "user", "content": "Summarise this..."}],
            "stream": True,
        }, stream=True)

The reported response time is the time to receive the response headers, which a streaming endpoint sends immediately — before the model has produced a single token. The reported payload size is zero. And because nothing consumed the body, the connection is returned to the pool or dropped with the generation still in progress, so you may not even be loading the system in the way you think.

Why, exactly

This is not a quirk to be worked around blindly; it follows from two decisions in HttpSession.request that are entirely reasonable for ordinary HTTP.

The first is that the clock stops when the underlying request call returns. With stream=True, the requests library returns as soon as the response headers have been read, deferring the body. So the recorded response time is time-to-headers by construction.

The second is how the payload size is determined. When streaming, Locust cannot measure the body without consuming it — which would defeat the point of streaming — so it takes the size from the content-length response header instead:

if kwargs.get("stream", False):
    request_meta["response_length"] = int(response.headers.get("content-length") or 0)
else:
    request_meta["response_length"] = len(response.content or b"")

A server-sent-event response is chunked and does not carry a content-length header, because its length is not known when the headers are written. The or 0 takes over, and every streamed request is recorded as zero bytes. Locust’s handling of custom protocols and the event API used below are documented under testing other systems.

The script

The fix is to consume the stream yourself, take your own timings, and report them as your own events. The built-in entry becomes a connection-establishment measurement, which is worth having under its own name.

# locustfile.py
import json
import os
import time

from locust import HttpUser, task, between

MODEL = os.environ.get("MODEL", "gpt-4o-mini")
API_KEY = os.environ["API_KEY"]


class StreamingChatUser(HttpUser):
    wait_time = between(2, 8)

    @task
    def chat_stream(self):
        payload = {
            "model": MODEL,
            "messages": [
                {"role": "system", "content": "You are a concise assistant."},
                {"role": "user", "content": "Explain retry budgets in three paragraphs."},
            ],
            "max_tokens": 400,
            "stream": True,
        }

        started = time.perf_counter()
        first_at = None
        last_at = None
        max_gap_ms = 0.0
        chunks = 0
        chars = 0
        finish_reason = None

        with self.client.post(
            "/v1/chat/completions",
            json=payload,
            headers={
                "Authorization": "Bearer " + API_KEY,
                "Accept": "text/event-stream",
            },
            stream=True,
            catch_response=True,
            name="POST /v1/chat/completions [connect]",
        ) as response:
            if response.status_code != 200:
                response.failure("status " + str(response.status_code))
                return
            try:
                for line in response.iter_lines(decode_unicode=True):
                    if not line or not line.startswith("data:"):
                        continue
                    payload_text = line[5:].strip()
                    if payload_text == "[DONE]":
                        break

                    now = time.perf_counter()
                    if first_at is None:
                        first_at = now
                    else:
                        gap_ms = (now - last_at) * 1000
                        max_gap_ms = max(max_gap_ms, gap_ms)
                    last_at = now

                    chunk = json.loads(payload_text)
                    choice = (chunk.get("choices") or [{}])[0]
                    chars += len(choice.get("delta", {}).get("content") or "")
                    finish_reason = choice.get("finish_reason") or finish_reason
                    chunks += 1
            except Exception as exc:
                response.failure(exc)
                return

            if first_at is None:
                response.failure("stream produced no data events")
                return
            if finish_reason == "length":
                response.failure("truncated at max_tokens")
                return
            response.success()

        ttft_ms = (first_at - started) * 1000
        total_ms = (time.perf_counter() - started) * 1000
        self._report("stream ttft", ttft_ms, 0)
        self._report("stream total", total_ms, chars)
        self._report("stream max inter-token gap", max_gap_ms, chunks)

    def _report(self, name, response_time_ms, length):
        self.environment.events.request.fire(
            request_type="SSE",
            name=name,
            response_time=response_time_ms,
            response_length=length,
            exception=None,
            context={},
        )

Two details are load-bearing. catch_response=True is what makes the built-in entry’s verdict yours, which matters because a stream that returns 200 and then dies after four tokens is a failure that no status code will report. And the failure branches return before firing the custom events, so a broken stream does not contribute a misleadingly fast time-to-first-token to the percentiles.

The three numbers worth reporting

• Time to first token. The number a user experiences as responsiveness. It is dominated by prompt processing, so it grows with input length and is the metric most affected by a long system prompt.
• Total stream duration. Governed mostly by how many tokens came back. Report it alongside the character or token count, because a total-time regression that is really a length increase is a different problem with a different fix.
• Maximum inter-token gap. The one people leave out and the one that catches the failure mode specific to streaming: a stream that stalls for eleven seconds mid-answer and then delivers the rest in a burst has the same start and end times as a healthy one, and is a much worse experience. Under load this is usually the first thing to degrade, because it is what batching and queueing upstream actually do to a generation in flight.

A fourth worth adding if your consumers parse as they go is a count of streams whose assembled body was not valid at the end — a stream that terminates early leaves a truncated JSON document, which is the subject of parsing streamed JSON.

Concurrency, which is now the control

Locust is a closed-model tool: a user issues a request, waits for it, waits its think time, and repeats. On a streaming endpoint the “waits for it” part now lasts as long as the generation, which changes what your user count means.

If the mean stream lasts 12 s and wait_time averages 5 s, one user
completes a request every 17 s, so:

    requests per second = users / 17

    100 users  ->  5.9 req/s   and ~71 streams open at any moment
    500 users  -> 29.4 req/s   and ~353 streams open

Concurrent open streams = users * (stream duration / cycle time)
                        = users * (12 / 17) = 0.71 * users

For a streaming service, concurrent open connections is usually the resource that runs out first — each one occupies a slot in the provider’s or your own scheduler for its whole lifetime. So the user count is the meaningful dial here in a way it is not for a request-response service, and reporting results as “N concurrent streams” rather than “N requests per second” is both more honest and more useful.

Note the feedback loop, though: if the system slows down, stream duration rises, cycle time rises, and your offered request rate falls — the closed-model behaviour discussed in load testing an LLM API with k6. For streams that is often the right model, because real users also wait. It is the wrong model if you are trying to find the point at which new connections start being refused.

Practical notes

• Locust runs on gevent, so a held-open stream blocks only its own greenlet. Thousands of concurrent streams from one process is feasible; the limit you hit first is file descriptors, not CPU. Raise the open-file limit before blaming the tool.
• FastHttpUser is available and is not obviously the right choice here. Locust’s documentation on increasing performance describes it as using geventhttpclient for considerably lower CPU per request, and its client does accept a streaming option. But the CPU saving matters at high request rates, and a streaming test is characterised by low request rates and long holds. Reach for it only if the load generator is genuinely CPU-bound.
• Use --processes to use more than one core. A single Locust process is bound to one core by the interpreter, and a load generator at 100% of one core produces timing measurements that are partly measuring itself.
• Vary the prompts. One fixed prompt gives you one output length, one cache behaviour and a latency distribution with no spread, which will not resemble production at all. Sample from a corpus; the reasoning is set out in using realistic prompt lengths in a load test.
• Set max_tokens and check the finish reason. Without a cap, the run cost and the total-time distribution are both unbounded. With a cap, some responses will truncate, and treating those as successes quietly biases your total-time percentile downward — which is why the script above fails them explicitly.