AI in CI: Automated Review That People Don't Mute

Every automated review bot is one week away from being collapsed by default. The design problem is not making it find more; it is making almost everything it says worth reading.

The arithmetic of a muted bot

A missed issue costs what the issue costs, discounted by the chance a human catches it anyway. A false comment costs the reviewer’s attention, plus a small permanent decrement in how seriously the next comment is taken. The second cost compounds and the first does not, which inverts the usual precision-recall tradeoff.

Put numbers on it — all of them assumptions, substitute your own. A bot posting n = 10 comments per pull request at precision p = 0.3 produces 7 wrong comments per PR. A reviewer on 30 PRs a month reads 210 wrong comments; at 20 seconds each to read and dismiss, that is 70 minutes of pure waste. But the behavioural cost arrives long before the 70 minutes: after the third PR in which most comments were noise, the thread gets collapsed unread, and from then on the bot’s recall is effectively zero regardless of what it finds.

Now n = 3 at p = 0.7: 0.9 wrong comments per PR, 27 a month, and comments that are read. The second bot finds fewer real issues per PR in principle and more in practice, because its output is consumed. Design for the second one.

Two mechanisms implement it. Make the model emit a confidence and a one-line justification per finding, and drop everything below a threshold you tune upward until complaints stop. Then cap the count: keep the top three by confidence and discard the rest, even good ones. A budget is not a limitation to apologise for; it is the feature.

Deterministic tools first, always

Never spend a model call on something a tool decides. Formatting is Prettier or gofmt. Unused imports and shadowed variables are the linter. Type errors are the type checker. Known vulnerable patterns are Semgrep or CodeQL. Dependency advisories are the audit tool. All of those are deterministic, instant, free, and never wrong in a way that requires discussion.

Run them first, and feed their output into the model’s request with an explicit instruction not to repeat anything already reported. Otherwise the model, given a diff and no other signal, will comment on style — because style is the thing that is visible without understanding, and it will fill its budget with it.

# .github/workflows/review.yml — order matters
- run: npm run lint -- --format json > /tmp/lint.json   || true
- run: semgrep --config auto --json > /tmp/semgrep.json  || true
- run: node ci/ai-review.mjs \
         --diff "$(git diff --unified=8 origin/main...HEAD)" \
         --already-reported /tmp/lint.json,/tmp/semgrep.json \
         --max-comments 3 --min-confidence 0.7

What it is actually good at

Ask for the classes where a language model has a genuine advantage over static analysis — those requiring an understanding of intent — and prohibit the rest explicitly, because the prohibition is what keeps the budget for the good ones.

Prohibit: naming opinions, “consider extracting this”, architecture commentary, performance speculation without a measurement, and anything phrased as a question. A bot that asks questions creates an obligation to answer, which is the fastest route to being muted.

What to put in the request

A bare diff is the most common configuration and the worst one. With only changed lines the model cannot tell whether a new call site respects a convention, whether an error type already exists, or whether the thing it is about to suggest was deliberately removed last month. Include:

• The diff with generous context — --unified=8 at minimum, and the full enclosing function where you can extract it.
• The PR title and description, which carry the intent the diff does not.
• The repository’s conventions file, so “inconsistent with the codebase” has a referent — the same file the coding tools read.
• The output of the deterministic tools, marked as already reported.
• For each changed file, the names of its existing tests. It is a cheap way for the model to notice that a behaviour change arrived with no test change.

Exclude lockfile, snapshot and generated-file diffs entirely. They are enormous, uninformative, and they will consume the comment budget on the one thing nobody wants an opinion about.

The one honest metric

Not comments posted, not issues found, not developer satisfaction surveys. Resolution rate: the share of posted comments that were followed by a change to the lines they pointed at, in the same pull request. It is imperfect — a change may be coincidental, a correct comment may be legitimately declined — and it is the only signal available that is behavioural rather than self-reported.

# For each bot comment: did the file+line region change after it was posted?
gh api "repos/:owner/:repo/pulls/comments?per_page=100" --paginate \
  --jq '.[] | select(.user.login=="ai-reviewer")
        | {id, path, line, commit_id, created_at}' > comments.json

# then, per comment, compare the blob at head against commit_id:
#   git diff <commit_id>..<head> -- <path>  -> did <line> +/- 3 change?
# resolution rate = touched / posted, tracked weekly per finding category

Track it per category rather than in aggregate, because that is what makes it actionable: if “missing error handling” resolves at 0.6 and “inconsistency” at 0.1, turn the second one off. A bot with two categories that work beats one with nine that mostly do not.

One hard rule: never fail the build on a model’s opinion. The same diff can produce a different verdict on a re-run — even at temperature zero — and a gate that is sometimes wrong and cannot be argued with is a gate that gets bypassed, then removed, taking the useful part with it. Advisory comments, blocking checks only for deterministic tools.