Refactoring Large Codebases With AI

A model can produce a 3,000-line refactor in ninety seconds. Nobody can review one. Everything useful about doing this well comes from taking that asymmetry seriously instead of hoping the tests cover it.

The constraint: review does not scale

Refactoring is defined by behaviour preservation, and behaviour preservation is exactly the property that a diff does not show. Class 1 on the review checklist — a comparison flipped or a return moved inside an otherwise mechanical change — is invisible in proportion to how large and mechanical the rest of the diff is. A large refactor is the worst case for human review by construction.

So the question is not “how do I get the model to refactor well”. It is: what can I make trustworthy other than the output? Two things can be. A transformation reviewed once and applied mechanically. And a test suite that would fail if behaviour changed. Everything below is one of those two.

Have it write the codemod, not the edit

This is the highest-leverage move in the whole cluster and it is consistently skipped. If a change is mechanical across many files, ask the model for the transformation program. You then review forty lines instead of four thousand, and the four thousand are produced by a deterministic tool that cannot get bored, hallucinate an import, or quietly improve something.

# JS/TS: jscodeshift or ts-morph (AST, type-aware with ts-morph)
npx jscodeshift -t codemods/logger-to-structured.ts src/ --dry --print

# Language-agnostic, structural, no AST to learn: comby
comby 'logger.info(:[msg])' 'log.info({ msg: :[msg] })' .ts -i

# Python: libcst    ·    Go: gofmt -r 'a.Foo(b) -> a.Bar(b)'
python -m libcst.tool codemod rename_kwargs.RenameKwargs src/

Run it with --dry first, read the printed output for three files chosen at random, then apply. The review target is the codemod plus a sample, and the sample is meaningful precisely because the transformation is uniform.

The judgement call is when a codemod is worth it. Rough rule: if the change touches more files than you would read carefully — call it fifteen — write the program. Below that, a scoped agent pass with a test after each file is fine.

Sample the output deliberately rather than by scrolling. The transformation is uniform, so the interesting cases are the atypical inputs, not random ones: the longest changed file, the shortest, the one with the most occurrences, and any file where the change count differs from what you predicted. Four files chosen that way tell you far more than forty read in order.

git diff --numstat | sort -k1 -rn | head -3        # biggest changes
git diff --numstat | awk '$1 != $2'                # lines added != removed,
                                                   # i.e. not a 1:1 rewrite
git diff --stat | tail -1                          # sanity: file count

The second command is the one to internalise. A mechanical rename should change the same number of lines it removes; a file where those differ is a file where the transformation did something structural, and that is exactly where a codemod’s edge cases live — a call inside a template literal, a re-export, a use in a comment, a name that collided with a local.

Where a codemod is not the answer: changes that require judgement per site. Splitting a function whose call sites each need a different argument, or migrating error handling where the right behaviour depends on the caller. A codemod applied to those produces uniform code that is uniformly wrong, and the uniformity makes it harder to spot, not easier. Those are the cases to do file by file with a test after each one — and to accept as genuinely slow work.

A ratchet that only moves one way

Large refactors fail by stalling: 60% converted, then a quarter passes and new code arrives in both styles. The fix is a mechanism that makes the old pattern impossible to add, installed the day the first file is converted rather than the day the last one is.

Where a lint rule exists, use it. Where one does not, a count in CI is a perfectly good ratchet:

#!/usr/bin/env bash
# ci/ratchet.sh — the count may fall, never rise.
set -euo pipefail
BUDGET=$(cat ci/legacy-logger.budget)     # e.g. 212
COUNT=$(git grep -c 'logger\.info(' -- 'src/**/*.ts' | awk -F: '{s+=$2} END{print s+0}')

if [ "$COUNT" -gt "$BUDGET" ]; then
  echo "legacy logger calls rose: $BUDGET -> $COUNT"; exit 1
fi
if [ "$COUNT" -lt "$BUDGET" ]; then
  echo "$COUNT" > ci/legacy-logger.budget
  echo "budget lowered to $COUNT — commit ci/legacy-logger.budget"; exit 1
fi

Failing when the count falls is deliberate: it forces the budget file into the same commit, so the number in the repository is always true and the progress is visible in the history. The whole mechanism is fifteen lines and it converts a refactor from a project into a background process that anyone can push forward by one file.

Slicing a refactor that changes an API

For anything crossing a module boundary, use expand-migrate-contract — three commits, each independently deployable and revertible:

• Expand. Add the new function, type or column alongside the old one. Nothing calls it yet. This commit cannot break anything, which means it can be merged without ceremony and reduces the size of the risky one.
• Migrate. Move call sites in batches, sliced by owner or by directory, with the ratchet counting down. Each batch is small enough to review and green on its own. This is where agent or codemod work belongs.
• Contract. Delete the old path once the count reaches zero. This commit is mostly deletions and is the one where a stray caller shows up as a compile error rather than a runtime one — which is the point of doing it last and separately.

Slice the middle stage by module boundary rather than by file count. A batch that stops at a boundary keeps the test suite meaningful; a batch of “the next twenty files alphabetically” cuts through the middle of a subsystem and leaves you unable to tell whether red means broken or half-done.

The silent improvement problem

The characteristic model failure in refactoring is helpfulness. Asked to extract a function, it also renames a variable for clarity, adds a null guard that changes behaviour on an input you relied on, tightens a type, and removes a branch it judged dead. Each is defensible. The aggregate is that “this is a pure refactor” is no longer true, and you have reviewed it as though it were.

Two counters, both cheap. First, instruct explicitly and ask for an exception report: change no observable behaviour; if you believe a behaviour change is necessary, stop and list it rather than making it. Models comply with this reasonably well and the list is often the most interesting output of the session.

Second, make behaviour preservation checkable before you start. If the area has thin tests, record characterisation tests against the current implementation first — real inputs and their current outputs, asserted as-is, warts included. They are not good tests and they are not meant to be; they are a fingerprint of current behaviour, and they are deleted when the refactor lands. Generating them is a good use of a model, because the oracle is the running system rather than anyone’s judgement.

Finally, watch the diffstat. A refactor that removes 400 lines and adds 520 has done something other than restructure, and git diff --stat tells you that before you read a line.