Bisecting a Prompt Change That Introduced Flakiness

A suite that was stable starts flaking, and the change that did it does not look like it could have. Bisect is the right tool, but the standard invocation quietly assumes the test is deterministic — and when it is not, bisect will still return a commit, confidently, and it may be the wrong one.

Why bisect breaks on a stochastic test

git bisect is a binary search over commits, and a binary search needs an oracle that gives the same answer every time it is asked about the same commit. A flaky test is not that oracle. Ask it once about a good commit and it may say “bad”; the search then discards the entire half of history that contains the real culprit and never revisits it. Bisect has no mechanism to detect that it took a wrong turn, so it terminates normally and prints a first-bad-commit that is simply incorrect.

Because the search is logarithmic, the number of probes is small — around 10 for a thousand commits — but every single one must be right. If each probe is individually correct with probability r, the whole search is correct with probability r^probes. At r = 0.95 and 10 probes, that is 0.95^10 = 0.599: a search that reaches the wrong answer two times in five. Probe reliability is the entire game, and it is why the naive git bisect run pytest -k the_test is worse than useless here.

Designing the probe

The probe must decide, for one commit, whether the flake rate is high or low — not whether one run passed. That is the same measurement problem as running a test N times and requiring K passes, and the same binomial arithmetic applies, but the reliability bar is higher because the errors compound across probes.

from math import comb

def p_green(q, n, k):
    return sum(comb(n, i) * q**i * (1 - q)**(n - i) for i in range(k, n + 1))

# assumed: pre-regression pass rate 0.98, post-regression 0.70
# probe verdict: GOOD if at least k of n attempts pass
for n, k in [(5, 5), (10, 9), (15, 13), (20, 18)]:
    print(n, k,
          round(p_green(0.98, n, k), 4),      # want close to 1  (good stays good)
          round(1 - p_green(0.70, n, k), 4))  # want close to 1  (bad reads bad)

# 5  5   0.9039  0.8319
# 10 9   0.9838  0.9999
# 15 13  0.9970  1.0000
# 20 18  0.9994  1.0000

Read the first row against the compounding rule above: five attempts requiring all five to pass gives a probe reliability around 0.90 on the good side, and 0.90 to the tenth power is 0.35 — a search that is wrong about two times in three. Ten attempts requiring nine gets probe reliability to roughly 0.98 on both sides, and 0.98 to the tenth is 0.82, which is a search worth running. That step from five to ten attempts is the difference between a bisect you can act on and a plausible-looking wrong answer.

The cost follows immediately: ten probes times ten attempts is a hundred model calls, plus whatever the rest of the suite costs, before you have a commit. Bound the search first — git log the prompt directory and start the bisect at the oldest commit that touched it, rather than at a release tag from three months ago.

The runner script

git bisect run reads the script’s exit status, and its contract is specific: 0 means the current commit is good, anything from 1 to 127 except 125 means bad, and 125 means this commit cannot be tested and should be skipped. Codes outside that range abort the bisection. That 125 is the feature that makes this workable — a commit that will not build, or where the prompt file does not exist yet, is skipped rather than misclassified.

#!/usr/bin/env bash
# bisect-probe.sh — exit 0 good, 1 bad, 125 untestable
set -u

N=10
K=9
TEST="tests/test_router.py::test_router_emits_a_tool_call"

# Untestable: the prompt this test needs does not exist at this commit.
[ -f prompts/router.md ] || exit 125

pip install -q -r requirements.txt >/dev/null 2>&1 || exit 125

passes=0
for i in $(seq 1 "$N"); do
  if pytest "$TEST" -q -x --no-header >/dev/null 2>&1; then
    passes=$((passes + 1))
  fi
done

echo "commit $(git rev-parse --short HEAD): $passes/$N passed"

if [ "$passes" -ge "$K" ]; then
  exit 0    # good
else
  exit 1    # bad
fi

Three details are load-bearing. The script must be executable and must live outside the working tree or be untracked, because bisect checks out each commit and a tracked script would change underneath the run. It echoes the pass count on every probe, so git bisect log plus the console output gives you the evidence rather than just the verdict. And it exits 125 rather than 1 when the environment cannot be prepared, which is the difference between skipping a commit and blaming it.

Running it

1. Pin everything that is not under bisect. Set the model to an explicit version rather than an alias, fix temperature and any seed, and pin your dependency versions — otherwise you are bisecting your lockfile as well as your prompts.
2. Find the bounds. git log --oneline -- prompts/ gives the commits that touched the prompts; pick a good that predates the flakiness and verify it by running the probe script there by hand. Verifying the bounds is not optional: a bad lower bound guarantees a wrong answer.
3. Start the search: git bisect start <bad-sha> <good-sha>. Git treats the first argument as bad and the second as good, and --term-old and --term-new let you rename them if “good” and “bad” read wrongly for a flakiness hunt.
4. Run it: git bisect run ../bisect-probe.sh, with the script outside the repository.
5. Save the evidence with git bisect log > bisect.log before git bisect reset. If you later doubt the result you can re-run the search from that log with git bisect replay instead of paying for a hundred model calls again.
6. Verify the answer directly. Check out the reported commit and its parent, run the probe on each with a larger N, and confirm the pass rates differ. Bisect narrows the search; it does not prove the result.

When bisect cannot help

Bisect assumes the property being searched for is monotone in commit order — good everywhere before a point, bad everywhere after. Several common causes of new flakiness are not commits at all, and no amount of probe repetition will find them:

• The provider changed. A model alias resolving to a new version, or a serving-stack update, affects every commit equally. The tell is that your verified “good” commit also fails when you re-run it today, which is check two of telling a flake from a regression. Confirm this before starting the bisect, not after ten probes.
• The cause is an unpinned dependency. Bisecting a repository whose install step resolves versions at probe time means every probe runs against a slightly different environment. Pin, then bisect.
• It was always flaky and traffic changed. The suite started running four times as often, or in parallel, so a pre-existing one-in-two-hundred flake became visible. There is no first bad commit because there was no change in the rate — only in the number of draws.
• Two changes interact. Bisect finds one boundary. If flakiness requires a retrieval change and a prompt change together, the reported commit is whichever landed second, and the fix is not necessarily to revert it.

In every one of those cases the useful next step is the same: stop searching history and start splitting the boundary instead, using the two-replay diagnostic to establish which side of the HTTP boundary the variance is on. Git bisect can only ever find things that are in git.

Primary documentation for the exit-code contract and the --term-old / --term-new options is the git-bisect manual page maintained by the Git project, which is worth reading once before trusting a script to it.