Gating a Merge on an Eval Score in CircleCI

The CircleCI half of this is short: a job that exits non-zero fails the workflow, and a failed workflow blocks the merge. The part that costs people an afternoon is that a job removed by a branch or tag filter stops being a dependency for everything downstream of it, so the gate you thought was required is quietly not there.

The runner has to exit non-zero

Nothing in CircleCI knows what an eval score is. The whole contract between your scorer and the CI product is the process exit status, so the first thing to get right is that your runner compares the score to a threshold itself and exits 1 when it is under. A runner that prints a score and exits 0 produces a green job with a red number in the log, which is the most common shape of a gate that does not gate.

Be deliberate about what the score is before you wire any of this up. It should be an aggregate over a fixed set of cases — pass rate against a rubric, schema validity, the fraction of cases where the right tool was selected — not a similarity measure against a stored sentence. Model output is sampled, so an exact-match assertion on prose fails on a paraphrase that is entirely correct, and a team that gets burned by that twice stops trusting the gate. Score properties that are stable under paraphrase and the number stops jittering for reasons nobody can act on.

Have it emit two files as well: a JUnit XML report so CircleCI’s test tab shows which cases failed, and a JSON file with the raw per-case results so the number can be compared against a baseline later. The threshold itself belongs in a committed file rather than in the config — see failing a build when the eval score drops below a threshold for why that separation matters more than it looks.

The config

This is a complete .circleci/config.yml with two jobs: the normal unit suite and the eval gate. The provider key lives in a context rather than in a project environment variable, so the same key is not readable from every job in the org.

version: 2.1

jobs:
  unit:
    docker:
      - image: cimg/python:3.12
    steps:
      - checkout
      - run:
          name: Unit tests
          command: pytest -q tests/

  eval-gate:
    docker:
      - image: cimg/python:3.12
    resource_class: medium
    steps:
      - checkout
      - restore_cache:
          keys:
            - evalcache-v1-{{ checksum "evals/cases.jsonl" }}
            - evalcache-v1-
      - run:
          name: Install
          command: pip install -r evals/requirements.txt
      - run:
          name: Score the golden set
          command: |
            python -m evals.run \
              --cases evals/cases.jsonl \
              --thresholds evals/thresholds.json \
              --junit-out reports/eval.xml \
              --json-out reports/eval.json
      - save_cache:
          key: evalcache-v1-{{ checksum "evals/cases.jsonl" }}
          paths:
            - .eval-cache
      - store_test_results:
          path: reports
      - store_artifacts:
          path: reports
          destination: eval

workflows:
  pr:
    jobs:
      - unit
      - eval-gate:
          context: llm-eval-keys
          requires:
            - unit

store_test_results takes a directory, not a file, and CircleCI reads every XML file under it. store_artifacts keeps the JSON around after the container is gone, which is what you will want the first time somebody asks why the score moved.

Making it a required check

1. Merge the config on the default branch first. A required status check can only be selected in the host’s branch protection UI after it has been reported at least once, so a check that has never run is not in the list.
2. Open a throwaway pull request and let the workflow run once. The status appears against the job name, in the form ci/circleci: eval-gate.
3. In the repository host’s branch protection settings for the default branch, require that named check.
4. Verify by pushing a commit that lowers the score below the threshold and confirming the merge button is disabled — not that the job is red, which is a weaker claim.

The fan-out quirk

Here is the behaviour that catches people. CircleCI’s configuration reference states that when jobs you list as dependencies do not execute — because of a filter, for example — CircleCI ignores them as dependencies for other jobs. The dependency does not fail and does not block; it evaporates.

Applied to a gate, that is worse than it sounds. If you write the natural-looking thing — a fan-out where the eval job carries a branch filter so it only runs on pull requests, and a deploy job requires it — then on any branch the filter excludes, the deploy job’s dependency on the gate simply disappears and it runs anyway. The workflow is green, the deploy happens, and no eval was scored.

Two rules keep this from happening. First, do not filter the gate job: let it always run and decide internally whether there is anything to score, so the job always reports a status. Second, when a fan-out has jobs on different filters, check that every path through the graph still passes through something that can fail — a filtered job reaching the same downstream node is not the same as an unfiltered one. If your reason for filtering was to skip evals on documentation changes, do that inside the job by exiting 0 early after a path check, which keeps the status reported.

Telling which one you have is quick. Open the workflow in the CircleCI UI and look at whether the gate job appears at all: a job removed by a filter is not shown as skipped, it is absent from the graph, which is why reading the pull request page instead is misleading. On the pull request, the giveaway is a check list that is shorter on one branch than on another with the same config. If the branch protection entry still names the check, the pull request sits on a pending status forever; if it does not, the merge is simply unguarded. Those are two different bad outcomes from one config, and the first at least announces itself.

Tag filters have the same shape with an extra wrinkle: CircleCI does not run workflows for tags at all unless a job explicitly declares a tag filter, so a gate that exists only in the untagged workflow is not run on a tagged release build. If your release process builds from a tag, the eval that guards the release has to opt in to tags explicitly, and it has to do so on every job in the chain that leads to it — the same evaporating-dependency rule applies.

Keeping the job cheap

• Cache the scored results, not the dependencies. The expensive part of an eval run is the model calls, not pip install. Key a result cache on the prompt text plus the model id so that only genuinely new cases pay — caching eval results between CI runs covers the key design, which is the part that decides whether you get hits at all.
• Put the gate behind the unit suite. The requires: [unit] above is not about ordering; it stops a branch that does not compile from spending money on model calls before failing anyway.
• Give it a resource class, not a bigger one. An eval job is almost entirely waiting on network I/O. A larger container makes it no faster and costs more credits per minute; concurrency inside the runner is what shortens it.
• Set a timeout. A provider outage turns an eval job into a job that sits at ten minutes of retries per case. Bound the whole step, and see timing out a stuck eval job.