Parallelising a Slow Prompt Regression Suite

Turning on your test runner’s parallel mode usually makes a prompt suite slower, noisier and more expensive. Both defaults are wrong for this workload: the degree of parallelism is derived from CPU count when the bottleneck is a token budget, and the isolation model assumes tests share nothing when several of yours share a cache, a cassette or a rate limiter.

The runner is parallelising the wrong axis

A prompt regression case spends almost all of its wall-clock time waiting for a network response, and the response time is dominated by how many tokens the model generates. There is essentially no local computation. So the resource that matters is in-flight requests, and the resource your runner is sizing against is cores.

The two numbers are unrelated, and being unrelated they are wrong in both directions. On a two-core CI runner, pytest -n auto gives you two workers, so a two-hundred-case suite runs a hundred cases deep on each — twelve minutes of pure waiting. On a thirty-two-core machine the same flag gives you thirty-two processes hammering the provider hard enough to earn a stream of 429s, each of which is retried, which costs money and adds latency.

Vitest exposes the right axis directly. test.concurrent and describe.concurrent run tests within a file concurrently, and maxConcurrency bounds how many run at once — documented default 5. That is a limit on in-flight work rather than on processes, which is what you want; the number just needs to come from the right calculation. fileParallelism separately controls whether whole files run in parallel, and it is the switch to reach for when files share state.

In Python the same shape is an async test with a semaphore sized to the token budget, or pytest-xdist with -n set to that number explicitly rather than to auto.

Constraint one: the token budget

Providers publish two limits and people watch the wrong one. Requests per minute is easy to reason about and rarely binds for a test suite. Tokens per minute binds constantly, because a regression case carries a system prompt, tool schemas and often a retrieved context, so a modest-looking suite pushes an enormous number of tokens through a narrow window.

Derive the ceiling rather than guessing it. With C requests in flight, each taking L seconds and consuming T tokens in total, throughput is 60C/L cases per minute and 60CT/L tokens per minute. Setting that below the limit:

C  <=  TPM_limit x L  /  ( 60 x T )

assumptions:  TPM_limit = 400,000
              L = 4 s per case (your p50, not a universal figure)
              T = 4,500 tokens per case (prompt + tools + output)

C  <=  400,000 x 4 / ( 60 x 4,500 )
    =  1,600,000 / 270,000
    =  5.9      ->  set concurrency to 5

Five. On a thirty-two-core machine, where the default would have given you thirty-two. And note the direction the terms pull: longer responses raise L, which raises the ceiling, because a slow case occupies the budget for longer per token. Bigger prompts raise T and lower it sharply. Adding a large tool schema to every request can halve your usable concurrency without changing a single test.

Leave headroom below the computed number, because the limit is enforced over a sliding window and your suite is bursty. Then handle the 429s you will still occasionally get: exponential backoff with jitter, honour Retry-After when the provider sends it, and cap total attempts so a rate-limit storm fails the run rather than extending it for an hour. Retries are not free — a retried request that partially generated is billed for what it produced.

Constraint two: state the workers share

Concurrency turns hidden coupling into intermittent failures. Four things are commonly shared in a prompt suite, and each breaks differently.

• The provider-side prompt cache. Cases sharing a system prompt warm each other’s cache. Serially, the first case pays full price and the rest hit the cache; concurrently, several issue before any has populated it, so which ones hit is a race. Any case asserting on cost or on cold-start latency becomes non-reproducible. Cache economics is a separate subject, but the testing rule is simple: cases that assert on cache behaviour must not run concurrently with cases that share their prefix.
• Recorded HTTP cassettes. Two workers replaying and rewriting the same cassette file corrupt it. In pytest-xdist the fix is a distribution mode: --dist loadgroup sends all tests carrying the same @pytest.mark.xdist_group(name=...) marker to one worker, so everything touching a given cassette stays on one process. --dist loadfile is the blunter version, grouping by containing file.
• Fixture stores. A vector index seeded per run, a temporary database, a fake tool backend holding counters. Give each worker its own namespace — xdist exposes a worker identifier for exactly this — or make the store read-only for the duration of the run.
• Spend counters and budget guards. A per-run cost cap implemented as an in-process counter counts one worker’s spend. It has to be external to the workers or it is decoration.

A quick diagnostic before you tune anything: run the suite with each file confined to one worker. If it goes green, your coupling is across files. If it is still flaky, the coupling is inside a file, and test.concurrent on a describe block that shares a fixture is the usual culprit.

Why sharding makes rate limits worse

The next instinct after in-process concurrency is to split the suite across CI machines. Vitest supports it directly: vitest run --shard=1/4 through --shard=4/4, four jobs, each running a quarter of the files.

For CPU-bound tests that divides the work. For this workload it does not, because the constraint is not on the machine. All four shards authenticate with the same API key against the same account, and the token-per-minute limit applies to the account. Four shards each running at the concurrency you derived above put four times that many tokens into the same window, and the provider throttles all four.

So divide the concurrency budget by the shard count. Four shards at concurrency 5 is not four times faster than one shard at 5; it is one shard at 20 with extra scheduling overhead and a much better chance of throttling. Four shards at concurrency 1 respects the budget and buys you only the fixed overhead reduction of shorter queues per machine.

Sharding does pay when the limits are genuinely separate — different provider accounts, different keys with their own quotas, or a suite where a large fraction of the time is local work such as embedding, parsing or scoring rather than waiting on the provider. Work out which of those you have before adding machines, because otherwise the CI bill goes up and the wall-clock does not go down.

Putting it together

1. Measure the two inputs the formula needs: your p50 completion latency for a typical case, and total tokens per case including the system prompt and tool schemas. Both are in your request logs already.
2. Compute the concurrency ceiling as TPM x latency / (60 x tokens_per_case) and take about three-quarters of it as your setting.
3. Set that number explicitly — maxConcurrency in Vitest config, or -n in xdist — and never auto. Divide it by the number of shards if you shard.
4. Group tests that share a cassette or a fixture store onto one worker: @pytest.mark.xdist_group with --dist loadgroup, or keep them in a single file with concurrency off for that file.
5. Add backoff with jitter and honour Retry-After, with a hard attempt cap so a throttled run fails fast instead of running for an hour.
6. Run the suite three times unchanged. Any case that is not green all three times shares state with another case; find it before you raise concurrency further, because everything above this line assumes the cases are independent.