A Merge Queue That Reruns Prompt Tests Before Merging

A pull request is tested against the target branch as it was when the run started. A merge queue tests it against the target branch as it will be when the merge lands. For a suite whose verdict is a score rather than a boolean, the gap between those two is larger than most teams expect.

What a merge queue does

When a pull request is added to the queue, GitHub creates a temporary branch containing the target branch plus the changes of every pull request ahead of it in the queue, plus this one, and runs the required checks against that. If the checks pass, the merge happens without further testing. If they fail, the offending pull request is removed from the queue and the temporary branches for the ones behind it are rebuilt without it.

Several pull requests can be grouped into one merge group, which is the setting that decides the cost: a group of five is one eval run rather than five. The repository’s merge queue configuration exposes the minimum and maximum number of pull requests to merge at once and how many merge groups may be building concurrently, and there is a setting controlling whether a failing pull request may be grouped with others at all.

The conflict that has no textual conflict

The general argument for a merge queue is the semantic conflict: two changes that are individually correct and jointly broken, with no overlapping lines for git to complain about. Prompt work produces an unusually clean example.

Pull request A tightens the system prompt to stop a verbosity problem. Pull request B adds twelve eval cases covering a customer complaint about answers being too terse. Both are green: A was tested against a dataset that did not contain B’s cases, and B was tested against a prompt that had not yet been tightened. They touch different files. They merge cleanly. The combination fails, and it fails on the target branch after both have landed, where the bisect points at whichever one happened to merge second.

Every pairing of “a change to what the model is told” and “a change to what we check” has this property, and those are the two most common kinds of change in a prompt repository. A merge queue is the only mechanism that tests the combination before it exists, which is why it is worth more here than in a codebase where most pull requests touch disjoint modules.

Scores do not compose

The sharper point is about the shape of the verdict. A deterministic test suite composes reasonably well: if A passes every test and B passes every test, the union usually passes, because each test is an independent boolean about a specific behaviour.

A threshold gate does not work like that. Suppose the bar is 92% over 200 cases. A lands at 93%, comfortably green — it fixed four cases and broke two. B lands at 93% too, having fixed three and broken two. Nothing says the combination sits at 93%; the breakages are additive while the fixes may overlap, and 91% is an entirely ordinary outcome from two green pull requests. There is no arithmetic that recovers the combined score from the two individual scores, because you would need to know which cases each affected — information the aggregate has thrown away.

This is the argument for the merge queue that is specific to probabilistic gates, and it generalises: any gate that aggregates across cases before comparing to a bar loses the per-case detail that would let you predict the combination. You can recover some of it by storing per-case results rather than only the aggregate — which is a good idea for other reasons too — but the only way to know the combined score is to run the combined tree.

The cost: a rerun is a fresh sample

The merge queue’s rerun is not a repeat of the earlier run. It is a new draw from the same distribution, against a different tree, and that has a consequence worth being honest about: a pull request can be evicted from the queue for sampling reasons alone. If the true combined rate sits just above the bar, some fraction of queue runs land just below it, and the queue removes a change that was fine.

Two mitigations, and they are the same ones that make any threshold gate workable.

• Put low-variance metrics in the queue. Schema validity, tool selection, refusal behaviour and redaction invariants are close to deterministic. Gate the queue on those and keep the judged-quality score as a tracked trend on the main branch. The queue then evicts for real defects and almost never for noise.
• Set the bar below the range you accept, not at the mean. A gate placed at the expected value fails roughly half the time by construction. This is the same reasoning as choosing a failure threshold, applied at the point where the consequence is eviction rather than a red check.

Resist the obvious workaround of retrying the queue run until it passes. In the queue it is worse than elsewhere, because the retry is unattended and the selection is invisible — you would be systematically admitting the favourable tail of every merge.

Wiring it up

The mechanical part is short, and there is exactly one trap.

1. Add the merge_group trigger to the workflow. The event has a single activity type, checks_requested. If a required check’s workflow does not listen for this event, the check never reports on the merge group, the merge group never completes, and pull requests sit in the queue until they time out. This is the trap, and it is documented: workflows performing required checks must be updated to include merge_group.
2. Select the tier by event name. The queue is where the full suite belongs, because it runs once per group rather than once per push.
3. Do not cancel merge-group runs by concurrency. A concurrency group keyed on github.ref with cancel-in-progress is right for pull request pushes and wrong here — cancelling a merge group check leaves the queue waiting.

name: evals
on:
  pull_request:
  merge_group:
    types: [checks_requested]

concurrency:
  group: evals-${{ github.ref }}
  # Cancel superseded PR pushes, never a merge group run.
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}

jobs:
  eval:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with: { node-version: 22, cache: npm }
      - run: npm ci
      - name: Run evals
        run: |
          if [ "${{ github.event_name }}" = "merge_group" ]; then
            npx vitest run eval/ --reporter=junit --outputFile=junit.xml
          else
            EVAL_SAMPLE=40 npx vitest run eval/ --reporter=junit --outputFile=junit.xml
          fi
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

One last practical note: the check name must be identical on both triggers, because branch protection matches on name. If you split the full and sampled runs into two differently named jobs, the required check will be satisfied by one of them and absent on the other, which reproduces the deadlock described in skipping the gate on docs-only pull requests. One job, one name, a branch inside it.