Skip to content
Multigrid

Eval gates in CI

Blocking a merge on a score — where the gate goes, what it costs to run, and how to keep it from becoming the thing everyone overrides.

A prompt change has no compiler. Nothing about editing a system message tells you that you broke the tool-selection behaviour on the twelve cases that mattered, and nothing about swapping a model version tells you that the JSON it returns is now wrapped in a code fence. The only thing that catches either is running a set of cases and comparing the result to something you agreed on earlier — and the only thing that makes anybody actually do that is a red check on a pull request.

These pages are about the check. Where the gate goes in each CI product, what the runner has to exit with, what the number should be compared against, and the two ways this ends badly: a gate so expensive that somebody turns it off, or a gate so weak that it passes an eval which collected nothing. Both failures are configuration, not modelling, which is why they are worth writing down.

Gating a Merge on an Eval Score in CircleCI

A CircleCI 2.1 config with the eval run as a required job, and the workflow rule that silently removes it from the dependency chain.

9 min read

Gating a Merge on an Eval Score in Jenkins

A declarative Jenkinsfile stage that fails the build below a score threshold, and the UNSTABLE-versus-FAILURE distinction that decides whether it blocks anything.

9 min read

Gating a Merge on an Eval Score in Azure Pipelines

Azure Pipelines YAML for an eval gate, plus the branch policy that has to exist because a pr trigger in YAML does nothing on Azure Repos.

9 min read

Gating a Merge on an Eval Score in Buildkite

A Buildkite pipeline where the eval step is a plugin-composed command step, with concurrency limits doing the cost control that self-hosted agents do not do for you.

9 min read

Failing a Build When the Eval Score Drops Below a Threshold

The platform-agnostic pattern: the runner owns the comparison and the exit code, the threshold lives in a committed file, and there is no warning mode.

9 min read

Posting Eval Results as a Pull Request Comment

A single updated bot comment carrying the score table, the token permissions it needs, and the fork case where the obvious workflow cannot post at all.

9 min read

Blocking a Deploy on a Flat or Falling Eval Score

The difference between a gate that requires the score to hold or improve and one that only requires a score to exist, and how to set the band it is compared against.

10 min read

Running Evals Nightly Instead of on Every Commit

When a nightly cadence is the right trade against CI cost, what you give up in blame resolution, and the hybrid that keeps most of both.

10 min read

Caching Eval Results Between CI Runs to Save Cost

A content-addressed result cache keyed on prompt, inputs, model version and decoding parameters, so only genuinely new cases pay for a model call.

10 min read

A Pre-Commit Hook That Runs a Fast Subset of Prompt Tests

A local hook that catches the mistakes worth catching in under five seconds, by asserting on everything except the model.

9 min read

Splitting a Slow Eval Suite Across Parallel CI Jobs

How to shard an eval suite on a stable case-ID hash so adding a case does not reshuffle every shard, and what stops the speed-up being linear.

11 min read

Skipping the Eval Gate for Docs-Only Pull Requests

A path filter that stops a README typo paying for a full eval run, without leaving the required status check permanently pending.

10 min read

Retrying a Flaky Eval Job in CI Without Hiding a Real Failure

Why retrying one case and retrying the whole suite are different operations, and how to bound retries so a green build still means something.

11 min read

Storing Eval History Across CI Runs to Chart a Trend

An append-only score-per-commit log kept on an orphan branch, with the fields that make a drop interpretable rather than alarming.

11 min read

A GitHub Actions Matrix for Testing Against Multiple Providers

Run one contract suite against every provider you support in parallel matrix jobs, asserting on the things that must be identical rather than on the text.

11 min read

Secrets Management for API Keys in a Prompt Test Suite

Why a fork's pull request cannot see your provider key, what the contributor sees instead, and the three patterns that give forks a real signal anyway.

11 min read

Setting a Per-Branch Budget for Eval Spend in CI

A worked allocation model for eval spend, and where the cap has to live given that provider keys have no concept of a branch.

11 min read

Timing Out a Stuck Eval Job Without Losing the Whole Pipeline

Why a hung provider call runs to the platform's six-hour job limit, and the four timeouts you need because the innermost one does not fire.

11 min read

A Merge Queue That Reruns Prompt Tests Before Merging

Why testing the combination of queued pull requests matters more when the gate is a score than when it is a pass, and how to wire the merge_group trigger.

10 min read

Running Prompt Tests Only Against Changed Prompt Files

A git-diff-based selector that maps changed prompt files to the suites that cover them, including the include graph and the empty-matrix error.

11 min read

Other topics