Gating a Merge on an Eval Score in Jenkins

Jenkins gives you something the hosted products do not: three build results instead of two. That is useful for an eval, because a score that dipped inside the noise band is genuinely different from a score that fell off a cliff. It is also the reason Jenkins eval gates leak, because only one of those three results blocks a merge.

The shape of the stage

The eval runner is an ordinary process. It reads a case file, calls the model, scores the outputs, writes a JUnit report and exits with a status. Jenkins’s job is to run it, publish the report, and turn the exit status into a build result. Everything interesting is in that last step, so capture the status rather than letting a bare sh step abort the stage the moment it is non-zero.

sh(script: ..., returnStatus: true) returns the exit code instead of throwing, which lets you distinguish exit 1 from exit 2 and decide differently. That distinction matters: a scorer that exits 1 because the score was low is a real signal, and a scorer that exits 4 because you passed it a flag it does not have is a broken pipeline being reported as a quality regression.

So give the runner a small, documented exit vocabulary and hold to it: 0 for at or above threshold, 1 for below threshold, and something distinct — 75 is a conventional choice for a temporary failure — for “could not complete”, meaning the provider was unreachable, the key was rejected, or the case file did not parse. Everything downstream in this page keys off that vocabulary. Without it, the only information Jenkins has is a boolean, and every policy you might want to write about retries, flakiness or notification collapses into the same branch.

The Jenkinsfile

pipeline {
  agent { label 'linux' }

  options {
    timeout(time: 20, unit: 'MINUTES')
    disableConcurrentBuilds()
    buildDiscarder(logRotator(numToKeepStr: '50'))
  }

  environment {
    EVAL_API_KEY = credentials('llm-eval-api-key')
    EVAL_MODEL   = 'gpt-4.1-mini-2025-04-14'
  }

  stages {
    stage('Unit') {
      steps {
        sh 'pytest -q tests/'
      }
    }

    stage('Eval gate') {
      when { changeRequest() }
      steps {
        script {
          def status = sh(
            script: '''
              python -m evals.run \
                --cases evals/cases.jsonl \
                --thresholds evals/thresholds.json \
                --junit-out reports/eval.xml \
                --json-out reports/eval.json
            ''',
            returnStatus: true
          )

          if (status == 1) {
            error('Eval score below the committed threshold. See reports/eval.json.')
          } else if (status != 0) {
            error("Eval runner failed to complete (exit ${status}).")
          }
        }
      }
    }
  }

  post {
    always {
      junit allowEmptyResults: false, testResults: 'reports/eval.xml'
      archiveArtifacts artifacts: 'reports/*.json', allowEmptyArchive: true
    }
  }
}

allowEmptyResults: false is doing real work there. If the eval run collects nothing — a bad glob, a case file that moved, a filter that matched zero cases — the JUnit step fails the build rather than reporting a cheerful zero-test success. A gate that passes because it ran no cases is the failure mode described in blocking a deploy on a flat or falling eval score, and this one flag closes it.

UNSTABLE is not a gate

Jenkins can end a build SUCCESS, UNSTABLE or FAILURE. The tempting design is to make a small regression UNSTABLE and a large one FAILURE, and it reads well in the Jenkins UI — a yellow ball, visibly not green.

The problem is what the repository host does with it. Jenkins reports UNSTABLE to a Git host as a status that most branch protection configurations treat as passing or as pending, not as failing, and the merge button stays enabled. So the yellow build is a note to a human who is not looking at Jenkins, and the change merges. If a score movement should block, it must produce FAILURE. error() does that; unstable() does not.

Keep UNSTABLE for the case where the score is fine but something about the run was not — a case that timed out and was retried, a provider returning a 529 on two of forty calls. Those are worth seeing and not worth blocking on, and catchError(buildResult: 'UNSTABLE', stageResult: 'FAILURE') around that narrower step expresses exactly that. That form sets the stage red in the Blue Ocean view while leaving the build result at your chosen value, so the person reading Jenkins sees precisely which stage was unhappy without the build claiming to be broken.

The same argument applies to the post block. failure runs on FAILURE and unstable runs on UNSTABLE, and they are different notification audiences: the first is “a change is blocked” and belongs to the author, the second is “the eval infrastructure is degrading” and belongs to whoever owns it. Routing both to one channel is how an UNSTABLE build becomes noise that everyone filters, and a filtered channel is where a real regression goes to be ignored.

Keys, and what leaks into the log

The credentials('llm-eval-api-key') helper binds a secret into the environment and adds it to the mask list, so a key echoed into the console appears as a row of asterisks. That masking is per-build and it is string matching, which means it protects the key and not a request body you printed that happens to contain it in a different encoding.

The practical rule for eval runners: never log the full request. Log the case id, the model id, the token counts and the score. A debug flag that dumps request bodies is the thing that puts a key or a customer record into a build log that fifty people can read, and build logs are retained. There is a general treatment of that in secrets management for prompt tests.

Running it only on pull requests

when { changeRequest() } restricts a stage to builds Jenkins knows are pull requests, which requires a Multibranch Pipeline or an Organization Folder — a plain single-branch Pipeline job has no change request to detect and the stage is skipped every time. If your gate never runs and never fails, check the job type before you check the condition.

changeRequest target: 'main' narrows it further to pull requests aimed at one branch, which is what you want once release branches exist and you do not want to score every backport against the main baseline. And disableConcurrentBuilds() in the options block is not tidiness: two builds of the same branch running the same eval simultaneously double the spend and race on any shared result cache.

One more Jenkins-specific trap worth knowing before it costs you a debugging session: in a Multibranch Pipeline the Jenkinsfile that runs for a pull request is normally the one from the merge result, not the one from the source branch. That is usually what you want, and it means a change to the eval configuration is itself evaluated against the target branch’s baseline. It also means a Jenkinsfile edit that only exists on the source branch can behave differently from how it read locally, so when a stage does something you did not expect, check which version of the file the build actually used before changing the file again.