Blocking a Deploy on a Flat or Falling Eval Score

Most eval gates that have been running for a year are not gates. They check that the suite ran and produced a number, and the number is compared against something that was true when it was written and has been true ever since. The distinction that matters is whether anything the model could do would make the check red.

The gate that only checks a number exists

There is a family of failures here and they all end the same way. The threshold is set at 0.6 while the suite has been scoring 0.94 for eight months, so it has three points of headroom and would not notice a serious regression. The suite scores every case as passing because a scorer exception is caught and counted as a pass. The case file stopped being collected and the pass rate over zero cases is 1.0. The eval runs against a cached transcript rather than the model, so the model is not involved at all. Each of these produces a green check that carries no information, and none of them announces itself.

They share a diagnostic. Break the thing on purpose and see whether the gate goes red. Delete a required field from the system prompt, or point the runner at a deliberately worse model, and push it to a branch. If the check does not fail, you have learned that it has not been failing for other reasons either. Doing this once a quarter is cheap and it is the only evidence that the gate works, because a gate that has never failed is indistinguishable from a gate that cannot fail.

What the score is compared against

Two comparisons are available and they answer different questions. An absolute floor asks “is this good enough to ship”, is stable, and is easy to reason about, but it is blind to a slow slide that stays above the line. A comparison against a baseline — the score the target branch currently gets on the same cases — asks “did this change make things worse”, catches the slide, and is noisier because it involves two measurements instead of one.

Run both. The floor is the ship criterion and belongs in the thresholds file described in failing a build when the eval score drops below a threshold; the baseline comparison is the regression criterion and needs a stored history, which is what storing eval history for a CI trend is about. The baseline must be re-measured rather than remembered: providers update models behind a stable name, so a number recorded three months ago may be describing a model that no longer exists. That is the mechanism behind silent model updates, and it is why the honest form of the comparison is to score the base branch and the head branch in the same run, against the same model version, within minutes of each other.

The band has to survive sampling noise

A model gate compares two noisy numbers. Unless every call is made at temperature 0 — and even then, batching and hardware differences can move outputs — two runs of an unchanged suite will not agree exactly. If your rule is “fail if the score is lower than the baseline at all”, you will fail roughly half of the runs where nothing changed, and the gate will be overridden within a fortnight.

The resolution of the comparison is bounded by the case count, and this is worth doing on paper before choosing a rule. With 40 cases, one case flipping is 2.5 percentage points, so no threshold finer than that means anything — a rule saying “fail on a drop of more than 1 point” is a rule that fails on any single case flipping. With 400 cases the same flip is 0.25 points and a 1-point band is about four cases, which is a change worth looking at. If you want a band tighter than your suite can resolve, the answer is more cases, not a tighter rule.

The practical setting: run the unchanged suite several times, take the spread you observe between those runs as the noise floor for your own setup, and set the band above it. That is a measurement you make, not one anybody can make for you, because it depends on your cases, your decoding parameters and your provider. Pinning temperature to 0 for the eval narrows it considerably even though it makes the eval less representative of production sampling — a trade most teams should take for a gate.

Flat is information too

The row this page is named for pairs falling with flat, and the pairing is not decorative. A prompt change that was supposed to fix a class of failures and moves the score by nothing has told you something specific: either the cases that cover that class are not in the suite, or the change did not do what you thought. Both are worth a comment on the pull request, and neither is worth blocking on.

Where flat should block is a deploy rather than a merge. If the score has not moved across a month of merged changes while the suite is supposedly being extended, the most likely explanation is that the suite stopped running, or stopped being extended, or has saturated at the ceiling and no longer discriminates between good and bad. A saturated suite is a real end state, and the response is to add harder cases rather than to celebrate. Watching for an unchanging number is the cheapest detector of an eval that has quietly died, and it is related to the blind spots covered in eval blind spots.

Ratcheting, and when to let it down

The mechanism that keeps a gate honest over years is a ratchet: when a run scores above the current floor by a clear margin, the floor is raised to the new level in the thresholds file, as an ordinary commit. Improvements become permanent, so the next regression is measured against what you actually achieved rather than what you once promised.

Raise it automatically only if the raise arrives as a reviewable change; a bot that edits the threshold file with an explanation is fine, a job that silently rewrites it is a gate that redefines itself to always pass. And there is a legitimate case for lowering: a model swap that is worse on the suite but better on latency or cost, adopted deliberately. Lower it in one commit that says so, with the trade written in the commit message. The failure to avoid is the threshold that drifts down one point at a time in unrelated pull requests, each reasonable on its own, until it is 0.6 with the suite scoring 0.94 and nobody remembers why.