Running Prompt Tests Only Against Changed Prompt Files

Twelve prompts, twelve eval suites, one pull request that changes a single template. Running all twelve is twelve times the cost and eleven twelfths of the wait for no information. The selector is straightforward; the mapping is where the correctness lives.

A different question from skipping

There are three reductions people conflate, and they compose rather than substitute. Skipping on docs-only changes is a binary decision about whether to run anything. Sharding splits a fixed amount of work across jobs and reduces no work at all. This page is the third: given that something relevant changed, run only the suites that could possibly be affected.

The property to preserve is that the selector may over-select freely and must never under-select. A suite that runs unnecessarily costs money; a suite that should have run and did not is a regression released. Every design decision below resolves in favour of running more.

Getting the diff right

You want the files this branch changed relative to where it diverged from the target, not relative to the target’s current tip. Those differ as soon as anything else merges, and the second includes other people’s changes, which will select suites your branch never touched.

# Three dots: changes on HEAD since the merge base with main.
git diff --name-only origin/main...HEAD

# Two dots would also report everything that landed on main meanwhile.
git diff --name-only origin/main..HEAD    # wrong for this purpose

The failure everyone hits first is that this produces nothing useful in CI, because the default checkout is shallow and the merge base is not in the local history. Either check out the full history or fetch enough of it:

      - uses: actions/checkout@v7
        with:
          fetch-depth: 0        # full history; the merge base exists

On a pull request event there is a shortcut worth knowing: github.event.pull_request.base.sha is the merge base GitHub itself computed, so git diff --name-only $BASE...HEAD against it is exact. On a push event you do not have that and must use github.event.before, which has its own edge case — it is all zeroes for the first push to a new branch, and your script must treat that as “run everything”.

Mapping files to suites

The mapping can be a convention or a manifest, and a manifest is worth the extra file. A convention — prompts/summarise.md is covered by eval/summarise.cases.jsonl — is invisible, so breaking it is invisible too: rename one side and the other silently stops being selected, and the suite quietly never runs again. A manifest breaks loudly, shows up in review, and can express things a naming convention cannot.

// eval/suites.json
{
  "suites": {
    "summarise":  { "paths": ["prompts/summarise/**", "src/llm/summarise.ts"] },
    "extract":    { "paths": ["prompts/extract/**",   "src/llm/extract.ts"] },
    "classify":   { "paths": ["prompts/classify/**",  "src/llm/classify.ts"] }
  },
  "runsEverything": [
    "prompts/_partials/**",
    "src/llm/client.ts",
    "src/llm/schema.ts",
    "eval/suites.json",
    "package-lock.json",
    ".github/workflows/evals.yml"
  ]
}

The runsEverything list is the part that makes this correct rather than merely clever. A shared partial included by every prompt changes every prompt, and none of those prompt files appear in the diff. So does the client wrapper, the schema definitions, the manifest itself, and the lock file — a provider SDK bump can change a default without a line of your source moving.

If your prompts are assembled from includes and you would rather not maintain that list by hand, resolve the include graph in the selector: parse each template, follow its includes, and treat a partial as belonging to every suite that transitively includes it. That is more code and it is the correct version, particularly if prompts live in a registry where the composition is already data.

// eval/select.mjs — prints a JSON array of suite names.
import { readFileSync } from "node:fs";
import { minimatch } from "minimatch";

const changed = readFileSync(0, "utf8").split("\n").filter(Boolean);
const { suites, runsEverything } = JSON.parse(
  readFileSync("eval/suites.json", "utf8"),
);

const all = Object.keys(suites);
const hitsEverything = changed.some((f) =>
  runsEverything.some((p) => minimatch(f, p)),
);

const selected = hitsEverything
  ? all
  : all.filter((name) =>
      suites[name].paths.some((p) => changed.some((f) => minimatch(f, p))),
    );

process.stdout.write(JSON.stringify(selected));

Feeding it to a dynamic matrix

A matrix can be computed at run time: a job emits a JSON array as an output, and a downstream job expands it with fromJSON. The detail that catches everyone is that an empty array is not an empty matrix — it is an error, along the lines of the matrix vector not containing any values — so the downstream job needs an if: guarding against it.

jobs:
  select:
    runs-on: ubuntu-latest
    outputs:
      suites: ${{ steps.pick.outputs.suites }}
    steps:
      - uses: actions/checkout@v7
        with: { fetch-depth: 0 }
      - run: npm ci
      - id: pick
        run: |
          BASE="${{ github.event.pull_request.base.sha }}"
          SUITES=$(git diff --name-only "$BASE"...HEAD | node eval/select.mjs)
          echo "selected: $SUITES"
          echo "suites=$SUITES" >> "$GITHUB_OUTPUT"

  eval:
    needs: select
    if: needs.select.outputs.suites != '[]'
    runs-on: ubuntu-latest
    timeout-minutes: 20
    strategy:
      fail-fast: false
      matrix:
        suite: ${{ fromJSON(needs.select.outputs.suites) }}
    name: eval (${{ matrix.suite }})
    steps:
      - uses: actions/checkout@v7
      - run: npm ci
      - run: npx vitest run eval/${{ matrix.suite }}
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

  gate:                                # the required status check
    needs: [select, eval]
    if: always()
    runs-on: ubuntu-latest
    steps:
      - run: |
          test "${{ needs.eval.result }}" != "failure" || exit 1
          test "${{ needs.eval.result }}" != "cancelled" || exit 1

The gate job exists for the same reason as in the docs-only case: branch protection needs a check name that reports on every pull request, including the ones where the eval matrix was empty and every eval job was skipped.

The cases that break it

• Deletions and renames. git diff --name-only lists a deleted file’s old path and a rename’s new path, and your glob matching must cope with a path that no longer exists. Do not stat the files; match on the strings.
• A merge commit on the branch. Merging the target branch into your feature branch, rather than rebasing, moves the merge base and can shrink the diff to almost nothing. The three-dot form handles this correctly; a hand-rolled comparison against the previous commit does not.
• Changes that are not files. A repository or environment variable holding a model ID, edited in the web interface, changes behaviour and produces no diff at all. Anything that can alter model behaviour and is not version-controlled defeats this entire technique, which is the strongest practical argument for keeping model configuration in the repository.
• The selector’s own dependencies. eval/select.mjs and its manifest must be in runsEverything. A change to the selection logic must run everything, because you cannot trust the new logic to decide what its own change affects.
• Third-party changed-file actions. A diff is four lines of shell; a dependency in the one job that must always run is a supply-chain surface. In March 2025 the tags of tj-actions/changed-files up to v45.0.7 were repointed at a commit that dumped runner secrets into build logs, tracked as CVE-2025-30066 and later added to CISA’s Known Exploited Vulnerabilities catalogue; see the GitHub advisory. If you use one, pin it to a full commit SHA rather than a tag.

Finally, do not let selection apply at merge time. The whole premise — “these files changed, so only these suites can be affected” — is a statement about one diff in isolation, and the point of a merge queue is that the combination of diffs behaves differently. Select on pull request pushes; run everything before the merge.

That full run is also how you find out the selector has quietly gone wrong. A selector is a claim about coverage, and nothing in the workflow checks the claim: if a mapping entry rots, the suite it names simply stops being selected and every pull request goes green faster than before. The nightly full run catches it, but only if you compare — so have the nightly job record which suites it ran and which the selector would have chosen for the same diff range, and report the difference. A suite that has not been selected by any pull request in a month is either dead or unmapped, and both are worth knowing.

The other check worth adding costs one line: fail the selector if it names a suite directory that does not exist, and fail it if a suite directory exists that no manifest entry covers. Both are the kinds of drift a rename introduces, and both are silent otherwise. An unmapped suite is the more dangerous of the two, because it looks exactly like a suite that is passing.