An Override Process for When an Eval Gate Blocks an Urgent Fix

At 02:00 a prompt change is dropping tool calls in production, the one-line fix is ready, and the eval gate is red because the fix moves three unrelated cases below their threshold. Every team hits this. The question is not whether you will ship anyway — you will — but whether the way you ship leaves a record.

Why the gate gets bypassed badly

An eval gate is a required status check: a job that scores a suite of cases and fails the build when the score drops. It is the right control, and it has one property that makes it dangerous in an incident. It is slow. A suite that takes eleven minutes is invisible during normal work and unbearable when a queue is backing up, so the pressure to route around it arrives precisely when judgement is worst.

The route people actually take is repository-admin merge. It works, it takes one click, and it produces nothing: no reason, no owner, no follow-up, and in most configurations no notification to anybody who was not watching. Six weeks later nobody can answer “which of our merged changes never passed the gate?” The second-worst route is a permanent escape hatch — a SKIP_EVALS=1 variable in the CI settings, added once during an incident and never removed, which quietly turns the gate off for everyone.

Both failures come from the same cause: the override is not a first-class thing that the system knows about, so it has to be improvised, and an improvised override is unlogged by construction.

What an override has to record

Design the artefact first and the mechanism follows from it. An override that is worth having produces, at the moment of merge, a durable record containing:

• Which cases failed, and by how much. Not “the gate was red” — the case ids and the scores, captured from the same run that failed. This is what makes the repayment possible; without it, restoring the gate later means re-deriving what was broken.
• A human reason, written by a person. A free-text field that a template cannot fill in. “Restoring tool calls for the checkout flow; the three failing cases are summarisation length and are not on this path” is a reason. “Urgent fix” is not.
• An owner. One named person, not a team alias. The follow-up is assigned to them at merge time, not negotiated afterwards.
• An expiry. A date by which the gate must be green again or the override is escalated. See below; this is the part that makes the difference between a process and a hole.
• The commit that carried it. So a later bisect over “when did this regress” can filter to changes that were never gated.

A mechanism that leaves an artefact

The implementation that fits a standard branch-protection setup is a second required check that passes when the override is properly formed. The eval job itself stays honest — it reports its real result and never conditionally skips — and a separate job decides whether the build is mergeable given that result. Because the override job is itself required, an unlabelled pull request with a red eval gate still cannot merge, and nobody needs admin rights.

#!/usr/bin/env bash
# .github/scripts/eval-gate-decision.sh
# Required check. Exit 0 to allow merge, non-zero to block.
set -euo pipefail

eval_status=$(cat eval-result/status)     # "pass" or "fail", written by the eval job

if [ "$eval_status" = "pass" ]; then
  echo "eval gate green"
  exit 0
fi

labels=$(gh pr view "$PR_NUMBER" --json labels --jq '.labels[].name')
if ! grep -qx 'eval-override' <<<"$labels"; then
  echo "eval gate red and no eval-override label present"
  exit 1
fi

body=$(gh pr view "$PR_NUMBER" --json body --jq .body)
reason=$(grep -m1 '^Eval-Override-Reason:' <<<"$body" || true)
owner=$(grep -m1 '^Eval-Override-Owner:'  <<<"$body" || true)
until=$(grep -m1 '^Eval-Override-Until:'  <<<"$body" || true)

for field in "$reason" "$owner" "$until"; do
  if [ -z "$field" ]; then
    echo "override incomplete: needs Reason, Owner and Until trailers"
    exit 1
  fi
done

if [ "${#reason}" -lt 60 ]; then
  echo "override reason is too short to be a reason"
  exit 1
fi

gh issue create \
  --title "Repay eval override on PR #$PR_NUMBER" \
  --assignee "$(sed 's/^Eval-Override-Owner: *//' <<<"$owner")" \
  --label "eval-debt" \
  --body "$(printf '%s\n\n%s\n\nFailing cases:\n%s\n' "$reason" "$until" "$(cat eval-result/failures.txt)")"

echo "override accepted and recorded"

Three details in that script are doing the real work. The eval job writes its failing case ids to a file that the decision job reads, so the follow-up issue carries the evidence rather than a link to a log that expires. The reason is length-checked, which is crude and works: it is enough friction to stop “urgent” and not enough to stop somebody with an actual incident. And the issue is created before the check passes, so the record cannot be skipped by merging quickly.

The expiry is the whole design

An override with no expiry is a permanently lowered gate that took a few extra keystrokes. The expiry converts it into a debt with a due date, and the enforcement belongs in a scheduled job rather than in anybody’s memory: a daily task lists open eval-debt issues, and for any whose recorded date has passed, escalates — posts to the team channel, pages the owner, or, if you want teeth, fails the next deploy of that service until the issue is closed.

There are two legitimate ways to close one. Fix the regression so the gate is green again, or change the gate deliberately: if those three summarisation cases were wrong to be blocking, the answer is to re-baseline them with a written justification, not to leave an override standing that also disables the checks that were right. That second path is why the record must name the specific cases. Without them the only available action is “look at the evals again some time”, which is not an action.

Where an override must not be available

Not every red check should be overridable, and lumping them together is how a reasonable process becomes a rubber stamp. Keep the override scoped to the score gate — the graded, thresholded, inherently fuzzy part. Deterministic checks stay unconditional: a schema validation failure, a guardrail case the suite claims to block, a secret detected in a diff, a test that asserts a tool call fires at all. Those are not judgement calls, so there is nothing for a human reason to add.

It is also worth separating the gate from the deploy. If your pre-deploy smoke test runs against the real API after the merge, an overridden eval gate still cannot ship a build with a broken API key or a model id that no longer exists — and that is the reassurance that makes people comfortable having an override at all. The gate you can bypass should be the slow, statistical one. The fast, binary ones stay closed.

Finally, count them. An override rate that trends upward is telling you the gate is mistuned, not that the team is reckless; a suite that blocks unrelated work is a suite with the wrong cases in it. One override a quarter is a process working. One a week is a signal that the threshold, not the discipline, is the thing to fix.