Running a Regression Suite Only on Changed Prompts

A suite that takes twenty minutes gets run once a day. The same suite, selected down to the cases a commit can actually affect, takes ninety seconds and gets run on every push. The obstacle is almost never the CI configuration; it is that nothing in your repository knows which tests depend on which prompt.

Selection needs a real dependency edge

Every test selector works the same way. It takes the set of files a commit touched, follows a dependency graph outward from them, and runs the tests it reaches. The graph is the whole trick. If the graph does not contain an edge from your prompt to your test, the selector does one of two things, and both are useless: it runs nothing, or it runs everything.

The common repository layout guarantees no edge exists. Prompts live in prompts/refund.txt or a YAML file, and the test loads them at run time with a path string. Nothing static links the two. Edit the text file and no import graph changes, because the text file is not in the import graph. This is the actual reason selection is rarely set up, and the fix comes before any tooling.

Make prompts modules, not data files

Put the template in a source file that exports it, and have the test import it. That single change creates the edge every selector needs, and it has a second benefit worth as much: the prompt now has a compile-time identity you can hash, which is what tagging cases to a prompt revision is built on.

// prompts/refund.ts
export const REFUND_SYSTEM = `You are a refunds assistant.
Reply with JSON matching the RefundDecision schema.
Never quote a figure that does not appear in the order.`;

export function renderRefund(order: Order) {
  return [
    { role: "system" as const, content: REFUND_SYSTEM },
    { role: "user" as const, content: JSON.stringify(order) },
  ];
}

If you need the prompt as plain text for a non-code audience, keep the text file and generate the module from it in a build step, with the generated file committed. The generated module is what the tests import, so the edge survives.

Selecting with the runner

Vitest ships two commands for this. vitest related takes source files and runs the tests that cover them; --changed derives that file list from git itself, and accepts a ref so you can compare against the branch you are merging into rather than against your working directory.

# tests that import a specific prompt module
npx vitest related prompts/refund.ts

# tests affected by everything on this branch
npx vitest run --changed origin/main

Then tell it which files must defeat selection entirely. Vitest’s forceRerunTriggers is a glob list; a match runs the whole suite regardless of the graph. Your model configuration and the selector script itself belong in it.

// vitest.config.ts
export default defineConfig({
  test: {
    forceRerunTriggers: [
      "**/package.json/**",
      "**/vitest.config.*/**",
      "**/src/model-config.ts",
      "**/scripts/select-tests.mjs",
    ],
  },
});

Python has no equivalent shipped with pytest. The commonly used third-party option is pytest-testmon, which records which tests executed which lines and reselects from that; check its own documentation for the current flag set before wiring it into CI, since it is an out-of-tree plugin. If you would rather not add one, the manifest below works everywhere.

The portable fallback: a manifest

Declare the mapping instead of deriving it. This is less clever and strictly more reliable, because a human wrote it and a human can read it in a review.

// scripts/select-tests.mjs
import { execSync } from "node:child_process";

const MAP = {
  "prompts/refund.ts":     ["tests/regression/refund"],
  "prompts/triage.ts":     ["tests/regression/triage"],
  "src/tools/schemas.ts":  ["tests/regression/tools"],
};
const ALWAYS = ["src/model-config.ts", "package-lock.json"];

const base = process.env.BASE_REF ?? "origin/main";
const changed = execSync(`git diff --name-only ${base}...HEAD`)
  .toString().trim().split("\n").filter(Boolean);

if (changed.some((f) => ALWAYS.includes(f))) {
  console.log("tests/regression");   // full suite
} else {
  const picked = new Set(changed.flatMap((f) => MAP[f] ?? []));
  console.log(picked.size ? [...picked].join(" ") : "");
}

Two details in that script are the whole difference between it working and it lying. A prompt file with no entry in MAP selects nothing and the build goes green, so add a check that every file under prompts/ appears as a key and fail the build when one does not — a new prompt with no tests is a fact worth surfacing rather than a silent pass. And a changed test file must select itself, since editing an assertion is a change whose effect only that test can show.

Note the three-dot origin/main...HEAD. Two dots compares the two tips, which on a branch that has fallen behind reports every file changed on main as changed by you and selects far too much. Three dots compares against the merge base, which is the set of files this branch actually touched.

What must never be selected away

Selection is an optimisation on the pull-request gate. It is not an optimisation on the record of whether the system works, and confusing the two is how a selective suite becomes a suite that has quietly stopped testing anything. The failure is asymmetric, too: selecting too much costs you a few minutes, while selecting too little costs you the entire value of the suite and does so invisibly. Every choice below is biased towards over-selecting for that reason.

• Run everything on the default branch. A merge is the point at which the combination of several selectively-tested branches exists for the first time.
• Run everything on a schedule. A nightly run against an unchanged tree is the only thing that can tell you a provider changed the model under you, because it is the only run where you did not change anything.
• Force a full run on infrastructure change. Model id, SDK version, temperature defaults, tool schemas, the retrieval corpus, and the selector script. A bug in selection is invisible by construction.
• Mind the merge queue. If merges are queued, the base moves after your selection was computed. Either recompute at the head of the queue or run the full suite there.

Putting it together

1. Move each prompt into a module that exports its template, and change every test to import it rather than read it from disk. Until this is done nothing else in this page will select correctly.
2. Add forceRerunTriggers entries for your model configuration, your lockfile and the selection script.
3. Add a pull-request CI job that runs npx vitest run --changed origin/main, or that shells out to the manifest script and passes its output as the path argument. Handle the empty-output case explicitly: no selected tests should exit zero, not run everything.
4. Add a second job on the default branch and a scheduled job that both run the full suite with no selection.
5. Verify the wiring rather than trusting it. Touch a single prompt module, push, and read the CI log for the list of test files it chose. Then touch src/model-config.ts and confirm the same job runs the whole suite.

Step five is the one people skip, and a selector that silently selects nothing looks exactly like a fast green build. Print the selected file list into the CI output every run so the count is visible in the place people already look.