Gating a Merge on an Eval Score in Buildkite

Buildkite runs on your agents, which changes the cost question. Nobody is metering your build minutes, so the expensive resource in an eval run is the provider bill, and the controls that matter are the ones that bound how many model calls can be in flight at once.

What the plugin model changes

A Buildkite command step is a shell command on an agent, and plugins are composable hooks around it — they run before or after the command in the same checkout. For an eval step that means the caching, the artifact handling and the container isolation are separate, declarative concerns rather than lines in your script. It also means the plugin version is part of your config surface: a plugin is referenced as a name and a tag, and floating that tag is how a pipeline changes behaviour without a commit.

The step itself still has to exit non-zero on a failing score. Plugins do not decide the result; the command does.

Pin the tag. A plugin reference is written as a name and a version — the cache plugin below is referenced with an explicit v1.12.0 — and pointing at a moving reference means an upstream release can change what happens around your eval step on a build that contains no commit of yours. That is unpleasant in any pipeline and worse in this one, because the failure it produces is a score that moved for a reason that is not in your history.

The pipeline

agents:
  queue: default

steps:
  - label: ":python: Unit"
    key: unit
    command: pytest -q tests/

  - label: ":chart_with_upwards_trend: Eval gate"
    key: eval-gate
    depends_on: unit
    command: |
      python -m evals.run \
        --cases evals/cases.jsonl \
        --thresholds evals/thresholds.json \
        --junit-out reports/eval.xml \
        --json-out reports/eval.json
    agents:
      queue: eval
    env:
      EVAL_MODEL: gpt-4.1-mini-2025-04-14
    artifact_paths:
      - "reports/*"
    timeout_in_minutes: 20
    concurrency: 1
    concurrency_group: "llm-evals"
    plugins:
      - cache#v1.12.0:
          path: .eval-cache
          manifest: evals/cases.jsonl
          restore: branch
          save: branch
    retry:
      automatic:
        - exit_status: 75
          limit: 2

depends_on references another step’s key, so the eval never starts on a branch whose unit tests are already red. The agents map targets a queue — useful when the eval agents are the ones with the provider credentials and the rest are not. That split is worth making even on a small fleet: it means the eval key exists on a handful of agents rather than on every machine that runs a build, and it gives you one place to rotate it.

The cache plugin’s manifest is the file whose contents decide cache identity, and restore and save take a level — file, step, branch, pipeline or all — that decides how widely an entry is shared. Restoring at branch level is the conservative default for eval results: a branch that changed the prompt does not inherit the parent branch’s scored outputs, so it pays for the cases it actually changed and no assertion is made about the ones it did not.

soft_fail turns a gate into a notice

Buildkite’s soft_fail marks a step as passed for the purposes of the build result even when the command exits non-zero, either unconditionally or for a listed set of exit statuses. It is a good feature and it is the wrong feature for this step, because a soft-failed gate produces a green build with a step that visibly failed — and green builds merge.

If the temptation to add it is coming from flakiness rather than from doubt about the threshold, the flakiness is the thing to fix. If it is coming from doubt about the threshold, lower the threshold in the committed thresholds file, where the decision is reviewable, rather than in the pipeline where it is invisible. There is one honest use: soft_fail with a specific exit_status that your runner reserves for “the eval could not run for an infrastructure reason” — but that only works if the runner never uses that status for a genuine score failure.

Concurrency is your spend limit

On a self-hosted fleet, ten pull requests updated at once means ten eval runs starting at once, each firing its whole case set at a provider. Two things break: your rate limit, and your budget. The concurrency and concurrency_group pair fixes both from the pipeline side — every step in the same named group across every build is limited to that many running at once, so the group acts as a global lock on eval spend.

A concurrency of 1 serialises them, which is usually right for a suite that finishes in a few minutes and wrong for one that takes twenty. The number to pick is the one that keeps your aggregate request rate under the provider’s limit, given the parallelism inside each run. Setting it too high does not fail cleanly — it produces 429s scattered across cases, which score as failures and look like a quality regression. Rate limits and retrying a flaky eval job both bear on this.

Retrying without laundering a failure

retry.automatic takes a list of exit_status and limit pairs, and the exit status matters more here than in an ordinary build. Retrying the whole step on any non-zero status means a genuine score regression gets three attempts at sampling its way to a pass, which is not a retry policy but a slot machine.

• Retry on transport, never on score. Reserve a distinct exit status in your runner for “did not complete” and list only that one.
• Retry inside the runner, per case. A single 529 on one case should be retried by the case, not by rerunning thirty-nine cases that already passed and paying for them again. A per-case retry with a short backoff is bounded work; a whole-step retry multiplies the cost of the run by the retry limit.
• Make the retried run visible. If a case only passed on its second attempt, the report should say so. A suite where three cases quietly need two attempts each is telling you something about the prompt.
• Make a retried request idempotent where you can. A retry of a request the provider already processed is a second charge for the same work, and across hundreds of cases that is not a rounding error. Where an idempotency key is supported, derive one from the case identifier and the request hash so a repeat is recognised rather than re-served.