Skipping the Eval Gate for Docs-Only Pull Requests

Path filtering a workflow is two lines of YAML. Path filtering a workflow that is also a required status check will stop every affected pull request from ever being mergeable, and the error message does not say why. Do the second part first.

What a docs-only run actually costs

It is worth being specific about what you are avoiding, because it decides how much complexity is justified. A full eval run charges you three separate things: provider tokens for every case, runner minutes for the wall clock, and the reviewer’s attention while a two-line README change sits behind twelve minutes of checks. The third is usually the one that actually hurts. A gate people learn to bypass because it is slow on trivial changes is a gate that stops working on the changes that matter.

There is also a correctness argument, and it points the other way. Skipping is only safe if the set of files you excluded genuinely cannot change model behaviour. That is a stronger claim than it looks, and the fourth section is about where it turns out to be false.

The required-check deadlock

GitHub’s paths and paths-ignore filters work by deciding whether the workflow runs at all. A workflow that does not run reports no status. A required status check that reports no status is not treated as passed — it is treated as pending, and the pull request shows “Expected — Waiting for status to be reported” indefinitely. The merge button stays disabled and nothing in the interface connects that to the paths filter you added.

The mechanism is worth holding on to because it generalises: branch protection asks “did a check with this name report success on this SHA”, and every way of making a workflow not run answers “no check with that name reported anything”. Deleting the workflow, filtering it out by path, gating it behind an if: at the workflow level, or having it skipped as part of a cancelled run all produce the same deadlock.

The consequence is a rule: the check must always run and must always report. What you skip is the expensive work inside it, not the job that carries the name branch protection is looking for.

The job that always runs

The shape is a cheap detection job that always executes, and an expensive job that is conditional on its output, with the required check name attached to a final job that depends on both and succeeds when the expensive one was legitimately skipped.

1. Detect, in a job with no path filter. Compute the changed file list against the merge base and set an output. Doing this with git diff rather than a third-party action is a deliberate choice; see the note at the end of this section.
2. Gate the eval job on that output with a job-level if:. A job skipped this way is reported as skipped, not failed, and — crucially — its dependents can still run.
3. Add a summary job named after the required check. It runs with if: always(), depends on the eval job, and fails only when the eval job’s result is failure or cancelled. A result of skipped is a pass.

name: eval-gate
on:
  pull_request:

jobs:
  detect:
    runs-on: ubuntu-latest
    outputs:
      run_evals: ${{ steps.changes.outputs.run_evals }}
    steps:
      - uses: actions/checkout@v7
        with:
          fetch-depth: 0            # merge base must exist locally
      - id: changes
        run: |
          BASE="${{ github.event.pull_request.base.sha }}"
          FILES=$(git diff --name-only "$BASE"...HEAD)
          echo "$FILES"
          if echo "$FILES" | grep -qE '^(prompts/|eval/|src/llm/|package-lock\.json|\.github/workflows/eval-gate\.yml)'; then
            echo "run_evals=true" >> "$GITHUB_OUTPUT"
          else
            echo "run_evals=false" >> "$GITHUB_OUTPUT"
          fi

  eval:
    needs: detect
    if: needs.detect.outputs.run_evals == 'true'
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v7
      - run: echo "run the suite here"

  gate:                              # this is the required check
    needs: [detect, eval]
    if: always()
    runs-on: ubuntu-latest
    steps:
      - run: |
          if [ "${{ needs.eval.result }}" = "failure" ] || \
             [ "${{ needs.eval.result }}" = "cancelled" ]; then
            echo "eval gate failed"; exit 1
          fi
          echo "eval gate satisfied (result: ${{ needs.eval.result }})"

Two details in there earn their keep. fetch-depth: 0 is required because the default shallow checkout does not contain the merge base, and git diff base...HEAD silently produces the wrong set — or an error — without it. And the workflow file lists itself in the trigger set, so a change to the gate always exercises the gate.

What counts as touching the model

The tempting filter is “skip if only docs/ and *.md changed”. Write it as an allow-list of files that do trigger evals instead, and the failure mode inverts from “silently skipped something important” to “ran the suite when it need not have”, which is the cheaper mistake.

The list is longer than most teams first guess. Prompt templates and the eval dataset are obvious. So is the client wrapper. Less obvious:

• The lock file. A provider SDK minor bump can change a default — a timeout, a retry count, which API surface a helper calls. A dependency change that alters model behaviour with no source change is exactly what the gate exists to catch.
• Model configuration held as data. A YAML or JSON file naming model IDs, temperatures or a routing policy is a behaviour change wearing config’s clothing.
• The workflow file itself, for the reason above.
• Anything a prompt template includes. If prompts are assembled from shared partials, editing a partial changes prompts that do not appear in the diff. The path-filter version of this handles it by listing the whole prompt directory; the per-suite selector has to resolve the include graph properly.
• Documentation that is a prompt. If a Markdown file in docs/ is read at runtime and injected as context, it is not documentation, whatever directory it lives in.

An override, and why you need one

Any static rule will be wrong occasionally, and the version that is wrong is usually “this looks like a docs change but I know it isn’t”. Give people a way to say so without editing the workflow: a label such as run-evals on the pull request, tested alongside the path result.

    if: >-
      needs.detect.outputs.run_evals == 'true' ||
      contains(github.event.pull_request.labels.*.name, 'run-evals')

Adding a label does not re-trigger a workflow on its own unless you also listen for the labeled activity type, so include types: [opened, synchronize, reopened, labeled] on the pull_request trigger if you want the label to take effect without an empty push.

Finally, decide separately what happens at merge time. Skipping on a pull request is a judgement about one diff; the combination of several diffs is a different question, and running the full suite once per merge group rather than once per push is usually the better trade — see the merge queue page.