Load Testing an Embeddings Endpoint Under Batch Requests

Requests per second is the wrong unit. One embeddings request can carry one string or a thousand, and a service that sustains 50 requests per second at a batch size of one may be doing a twentieth of the work of the same service at 5 requests per second with batches of 200.

Why the chat load profile does not transfer

A chat completion test cares about time to first token, tokens per second during generation, and the tail of a long stream. An embeddings call has none of those properties. There is no streaming, so latency is a single number: the whole response arrives or it does not. There is no output length to vary, because the response size is fixed by the dimension count and the number of inputs. And the request is front-loaded — all the work is prefill over a payload that can be megabytes.

That last point moves the bottleneck. A chat test rarely hits a request size limit; a batch embeddings test hits it routinely. Anthropic publishes a 32 MB maximum request size for its Messages and token counting endpoints and 256 MB for the Batch API, and providers cap the number of inputs per embeddings call independently of the byte limit. A load test that ramps batch size without a guard eventually stops measuring throughput and starts measuring how quickly a 413 comes back.

• The rate limit is usually tokens per minute, not requests per minute. Doubling the batch size does not buy headroom; it spends the same allowance in half as many requests. A test that reports only RPS will show a service happily under its limit while the token budget is exhausted.
• Latency has a floor and a slope. There is a fixed per-request overhead — connection, auth, routing — and a component that grows with total tokens in the batch. Finding where the slope starts to dominate is the entire point of the exercise.
• Retries are expensive in a way chat retries are not. Retrying a failed batch of 500 re-sends 500 items. If the failure was caused by one oversized item, it will fail again identically. Split on failure rather than retrying whole.

Batch size is the independent variable

Design the run as a sweep, not as a single ramp. Pick a small set of batch sizes — 1, 16, 64, 256 is a reasonable spread — and for each one find the concurrency at which p95 latency crosses whatever your ingestion pipeline can tolerate. What you want out of it is a table of items per second and tokens per second at each batch size, because that is the number that tells you how long a backfill of ten million documents takes.

The interesting result is almost always a knee: per-item latency improves sharply from batch size 1 to some middle value as the fixed overhead is amortised, then flattens, then gets worse as payloads grow large enough to cost real time on the wire. Running your ingestion at the flat part rather than past the knee is usually worth more than any concurrency change.

A locustfile that varies batch size

Locust’s HTTP user exposes a session as self.client, tasks are methods decorated with @task, and wait_time controls the pause between them. Two features matter here. The name argument groups statistics under a label of your choosing rather than per URL, which is how you get one row per batch size. And catch_response=True gives you a context manager in which you can mark an otherwise-2xx response as a failure — necessary because a 200 that returned fewer vectors than you sent inputs is a failure your pipeline must not silently absorb.

from locust import HttpUser, between, task
import random

CORPUS = [line.strip() for line in open("fixtures/corpus.txt")]

class EmbeddingsUser(HttpUser):
    wait_time = between(0.1, 0.5)

    def _embed(self, batch_size: int):
        inputs = random.sample(CORPUS, batch_size)
        with self.client.post(
            "/v1/embeddings",
            json={"model": "text-embedding-3-small", "input": inputs},
            name=f"embeddings batch={batch_size}",
            catch_response=True,
        ) as response:
            if response.status_code == 429:
                response.failure("rate limited: " + response.headers.get("retry-after", "?"))
                return
            if response.status_code != 200:
                response.failure(f"status {response.status_code}")
                return
            vectors = response.json().get("data", [])
            if len(vectors) != batch_size:
                response.failure(f"expected {batch_size} vectors, got {len(vectors)}")

    @task
    def small(self):
        self._embed(16)

    @task
    def large(self):
        self._embed(256)

Run it headless with an explicit user count, spawn rate and duration — --users, --spawn-rate and --run-time — so the run is reproducible and can be scripted. See the Locust documentation on writing a locustfile for the current argument list.

Do not load test with identical inputs

Sending the same string ten thousand times is the classic way to produce a beautiful, meaningless graph. Any deduplication in your own client, any cache in front of the provider, and any request-level caching upstream will collapse the work, and you will measure a cache rather than a service. Sample from a real corpus with realistic length variance, because token count per item is the thing that actually drives cost and latency and a synthetic corpus of uniform short strings has none of it.

The same discipline applies to length distribution: if 5 percent of your real documents are ten times the median length, the fixture needs that tail, because those are the batches that hit the payload limit.

Reading the result

Report p50 and p95 per batch size, plus items per second and the count of 429s. Do not report a mean: one cold start or one retried batch drags a mean past every request anyone experienced. If the 429 count is non-zero at the concurrency you plan to run, the answer is not a bigger machine — it is a queue in front of the calls, which is a different thing to test — rate limiting an AI endpoint covers the shape of it. And if you are load testing to decide a batch size for a one-off backfill, check whether the provider offers an asynchronous batch endpoint first; a job that can wait hours usually should not be measured in p95 at all.

Record the run’s inputs alongside its outputs, because an embeddings throughput number is meaningless without them: the model id, the dimension count if the model supports reducing it, the batch sizes swept, the concurrency, the corpus and its median and p95 item length. Six months later somebody will ask whether ingestion got slower, and without those fields the only honest answer is that the two runs are not comparable. The dimension count matters more than people expect on the response side — a batch of 256 vectors at 3,072 dimensions serialised as JSON floats is a large response body, and if your client is spending measurable time parsing it, you are measuring your JSON parser as much as the provider.