Splitting a Slow Eval Suite Across Parallel CI Jobs

An eval suite is almost entirely waiting. Two hundred cases at three seconds of provider latency each is ten minutes of wall clock and perhaps twenty seconds of CPU, which is exactly the shape of work that splits across jobs well. What decides whether the split holds up over a year is the shard key.

What linear scaling actually requires

Two conditions, and most suites satisfy neither by accident. The first is that every shard takes roughly the same time, not that every shard holds the same number of cases. The second is that the assignment of a case to a shard is stable: the same case lands in the same shard tomorrow, after you have added forty new ones and deleted six.

Stability is the condition people skip, and it is the one that quietly costs the most. The default sharding in most runners is positional — the collected list of tests is cut into contiguous blocks. Insert one case near the front and every case after it moves, which means every per-case artefact keyed on shard membership is now wrong: recorded durations, cached responses, any cassette directory laid out per shard, any incremental result store that skips cases whose inputs did not change. The suite still passes. It just stops being fast, and nobody connects the slowdown to the case they added a fortnight ago.

A hash-based key fixes stability outright. Assignment depends only on the case’s own identity, so adding, removing or reordering cases moves nothing except the cases you touched. It gives you no help at all with the first condition, balance by time — that has to be handled separately, and the last section of this page is about how.

Hashing the case ID

The key has to be something a human wrote and will not change casually. A stable id field on each case in your dataset is the right choice. The file path is nearly as good until somebody renames a directory. The array index is the thing you are trying to get away from. The prompt text itself is tempting and wrong: editing a prompt to fix a typo would move the case to another shard and invalidate whatever you had cached against it.

Any stable hash works — you are not defending against an adversary, only against clustering. What matters is that it is the same function on every runner and in every language you might reimplement it in, so reach for something with a specification rather than the language runtime’s built-in string hash, which is allowed to be seeded per-process.

// eval/shard.ts
import { createHash } from "node:crypto";

/** Stable across processes, runners and releases. */
export function shardOf(caseId: string, total: number): number {
  const digest = createHash("sha256").update(caseId).digest();
  // Top 32 bits are plenty; readUInt32BE avoids BigInt.
  return digest.readUInt32BE(0) % total;
}

export function selectShard<T extends { id: string }>(
  cases: T[],
  index: number, // 1-based, to match --shard=1/4 conventions
  total: number,
): T[] {
  return cases.filter((c) => shardOf(c.id, total) === index - 1);
}

The important detail about running that under Vitest is that Vitest’s own --shard flag, documented as --shard=<index>/<count>, splits at the level of files. If your entire eval suite is one file that loops over a JSONL dataset — which is how most of them start — then --shard=2/4 gives shard 2 the whole suite and the other three shards nothing. That is not a bug and it is not obvious from the output; three jobs finish in nine seconds and look like a success.

So you have two coherent designs and should pick one. Either generate one test file per case group and let the runner shard files, or keep the single data-driven file and select cases yourself from environment variables the matrix sets. The second is less code and keeps the dataset as the source of truth:

// eval/quality.test.ts
import { describe, expect, it } from "vitest";
import cases from "./cases.json";
import { selectShard } from "./shard";

const index = Number(process.env.SHARD_INDEX ?? 1);
const total = Number(process.env.SHARD_TOTAL ?? 1);

describe.concurrent("answer quality", () => {
  for (const c of selectShard(cases, index, total)) {
    it(c.id, async () => {
      const result = await runCase(c);
      expect(result.schemaValid).toBe(true);
      expect(result.toolCalled).toBe(c.expectTool);
    });
  }
});

On the Python side, pytest-split takes the other approach and is worth knowing about: you record a run with --store-durations, which writes a .test_durations file, and thereafter --splits 4 --group 2 partitions by recorded duration rather than by count. It offers duration_based_chunks (the default) and least_duration as splitting algorithms. That is balance by time, which is the condition hashing does not give you — at the cost of a checked-in durations file that goes stale and has to be regenerated.

Wiring it to a matrix

The matrix itself is four lines. The two settings that matter are fail-fast: false, because the default cancels every sibling job the moment one fails and you want to see all the failures from one run, and a job-level timeout-minutes well under the platform default of 360, so a hung provider call costs you twenty minutes rather than six hours.

name: evals
on: [pull_request]

jobs:
  eval:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: npx vitest run --reporter=blob
        env:
          SHARD_INDEX: ${{ matrix.shard }}
          SHARD_TOTAL: 4
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: blob-${{ matrix.shard }}
          path: .vitest-reports/
          retention-days: 7

The if: always() on the upload is not decoration. A job cancelled by its own timeout-minutes skips every remaining step, so without it the one run you most want to inspect is the one that uploads nothing.

Merging the shard reports

Four green jobs are four separate verdicts, and an eval gate is usually a single aggregate: a pass rate over the whole dataset compared with a threshold. Shard 1 passing 48 of 50 tells you nothing about whether the suite cleared 92%. You need the four partial results in one place before the comparison happens.

Vitest ships this as a first-class flow: run each shard with --reporter=blob, which writes a machine-readable blob into .vitest-reports, then in a downstream job download all the artefacts and run vitest --merge-reports with whatever human reporter you want. The merge job needs if: always() too, since by definition you are running it after failures.

  report:
    needs: eval
    if: always()
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with: { node-version: 22, cache: npm }
      - run: npm ci
      - uses: actions/download-artifact@v7
        with:
          path: .vitest-reports
          pattern: blob-*
          merge-multiple: true
      - run: npx vitest --merge-reports --reporter=junit --outputFile=junit.xml
      - run: node eval/gate.mjs junit.xml --min-pass-rate 0.92

Keeping the threshold comparison in its own script rather than in a test assertion is worth doing for a second reason: the same script can write the run’s score to the history you chart from, which is the subject of storing eval history across runs.

Where it stops being linear

Two things reliably break the speed-up, and neither is fixed by a better hash.

• One case that is much slower than the rest. Hashing balances counts. If one case exercises a reasoning model that thinks for ninety seconds while the other 199 return in two, the shard holding it sets the wall clock and three runners sit idle. The fix is to stop pretending it is the same kind of work: give the slow model its own job with its own timeout, or attach a weight to each case and assign greedily to the currently-lightest shard, which is what pytest-split’s least_duration algorithm does with recorded times.
• The provider rate limit, which you have just multiplied. This is the one that surprises people. A suite that ran happily at ten concurrent requests now runs at ten concurrent requests per shard. Eight shards is eighty in-flight requests and a wall of 429 responses, and because most SDKs retry those automatically with backoff, the symptom is not an error — it is that eight jobs take longer than one job did. Your concurrency setting has to be a global budget divided by the shard count, not a per-process constant.

There is also a cost point worth being clear-eyed about. Sharding changes wall-clock time and changes nothing about spend: the same number of cases hit the same models. If the run is expensive rather than merely slow, sharding is the wrong tool and budgeting the run or selecting fewer suites is the right one.