A Pre-Commit Hook That Runs a Fast Subset of Prompt Tests

A pre-commit hook has about five seconds before people start using --no-verify by reflex, and once that reflex exists the hook is gone whether or not it is installed. Five seconds is not enough to call a model, and that constraint turns out to define a genuinely useful hook rather than a crippled one.

The budget decides the design

Work backwards from the number. A single model call with a modest prompt is commonly a second or more, and ten of them concurrently against a rate-limited key is not reliably faster than doing them in sequence. Add process start-up and dependency import and the honest conclusion is that no hook that calls a provider fits in the budget on a bad network, and a hook whose latency depends on someone else’s availability is a hook that will one day block every commit in the team during an outage.

There is a second reason, independent of speed: a hook that calls a paid API means every local commit spends money, on a developer machine, with a key that has to be present in every developer’s environment. That is both a cost surface and a credential surface for the least valuable tier of testing you own. Do not do it. The model calls belong in CI, behind the gate described in failing a build when the eval score drops below a threshold.

What is worth asserting offline

Removing the model removes less than you expect, because most of what breaks a prompt change is structural and visible without inference. These are the checks that pay for themselves in a hook:

• Every template renders. Load each prompt template and render it with a fixture set of variables. This catches a renamed variable, an unbalanced brace and a template that references a field the caller no longer supplies — the single most common way a prompt edit breaks in production.
• No variable is left unsubstituted. After rendering, assert the output contains no residual placeholder markers. A prompt shipped with a literal placeholder in it usually still produces plausible output, which is why this one survives review.
• Tool schemas are valid and names match the code. Validate each tool definition against the JSON Schema subset your provider accepts, and assert that every tool name referenced in a prompt has a handler registered. A prompt telling the model to call lookup_order when the tool is registered as lookupOrder is a silent failure at run time.
• The rendered prompt fits the budget. Count tokens with the tokeniser rather than by characters and fail above a committed ceiling. This catches the few-shot example somebody pasted that quadrupled the system prompt, and it is the check most likely to save real money — see the token cost of tool schemas.
• Replayed cases still pass their structural assertions. If you keep recorded responses from CI, replay ten of them through the parsing and validation path. This asserts on your own code, not the model, and it catches a schema change that would break every response — the approach in testing without the model.

What is deliberately absent is any assertion about the content of a model response. Even with a recorded transcript, asserting that the output contains a particular sentence is a test of a paraphrase, and it will fail on a correct answer the day the recording is refreshed.

The hook

The pre-commit framework runs the hook and manages the git plumbing. Define it as a local hook so it runs your own code from the repository with no external hook repository involved.

# .pre-commit-config.yaml
repos:
  - repo: local
    hooks:
      - id: prompt-smoke
        name: prompt smoke tests
        entry: python -m evals.smoke
        language: system
        pass_filenames: false
        stages: [pre-commit]
        files: ^(prompts/|evals/|src/tools/)

files restricts the hook to commits that touch prompts, eval fixtures or tool definitions, so a commit to the CSS does not pay for it. pass_filenames: false is right here because the smoke suite decides its own scope; leave it at the default and pre-commit appends the staged filenames to the command, which your runner would have to parse. And language: system uses the interpreter already on the path rather than building an isolated environment, which is the difference between a hook that starts instantly and one that does not.

1. Add the configuration file above to the repository root.
2. Run pre-commit install so the git hook script is written into .git/hooks. This is per clone; it is not carried by the repository, so it belongs in your setup instructions.
3. Run pre-commit run --all-files once to check the hook passes on a clean tree before anybody meets it mid-commit.
4. Time it. If it is over five seconds, cut cases until it is not.

Stage names changed

If you copy a configuration from an older post you will see stages: [commit], and depending on your version that is either deprecated or rejected. The pre-commit documentation records that the stage values were changed to match the git hook names: commit, push and merge-commit became pre-commit, pre-push and pre-merge-commit, in version 3.2.0. Write the new names.

The related decision is which stage to use. pre-commit gives the fastest feedback and the tightest budget. pre-push is worth considering for anything in the two-to-ten second range, because a push is a less frequent and more deliberate act than a commit and people tolerate more latency there — a reasonable split is the structural checks on commit and the replayed cases on push.

The escape hatch, and not fighting it

Both escape hatches are legitimate. SKIP takes a comma-separated list of hook ids, so SKIP=prompt-smoke git commit skips this one and leaves the rest running, and git commit --no-verify skips everything. Do not try to close them. A hook that cannot be bypassed blocks somebody committing a work-in-progress on a plane, and the response to that is not compliance, it is uninstalling the hook.

The reason this is safe is that the hook is not the gate. It exists to shorten the loop on the mistakes that are annoying to discover in CI ten minutes later, and everything it checks is checked again in the pipeline where the bypass does not exist. If you find yourself wanting the hook to be unskippable, what you actually want is that check in CI, where it belongs.