Refactoring an AI Feature After Six Months

Six months after shipping, an AI feature is running on a model that has been superseded, a prompt that has accumulated clauses nobody can justify, and a set of assumptions about what models could not do that were true when they were made. Each of those decays by a different mechanism, at a different speed, and knowing which is which decides where the afternoon goes.

Three layers, three decay mechanisms

An AI feature has three layers that age. The point of separating them is that their decay is not a common phenomenon of “things get stale” — each has a specific cause, and the cause tells you the remedy.

The distinction that matters most is the third. Prompt and model decay are visible — a deprecation notice arrives, a prompt file gets long. Architectural decay is invisible, because working code does not announce that its reason for existing has expired. A three-stage pipeline that splits a document because the model of the day could not hold it keeps working perfectly, at three times the necessary cost and latency, indefinitely.

Which one goes first, and why

Ranked by urgency, which is not the same as by size:

• The model, because it has a date. Providers announce deprecations with a window, and a feature pinned to a retiring model has a forced migration whether or not anyone planned for it. This is the only layer where inaction produces an outage on a known day, which makes it the only one with a genuine deadline.
• The prompt, because it blocks the model change. A prompt tuned to one model does not transfer cleanly, so prompt debt is what makes the forced migration expensive. A prompt with no cases attached cannot be migrated at all in any principled way — you change the model, the output changes, and nothing tells you whether it changed for the better.
• The architecture, because it is the largest prize and has no deadline at all. Removing a stage removes its cost, its latency and its failure modes permanently. Nothing will ever prompt you to do it, which is exactly why it needs a scheduled review rather than a trigger.

There is a fourth thing that ages and is not a layer: the evaluation set. A case set built at launch represents the traffic of six months ago. If usage has broadened, the set is now measuring a slice, and every decision made against it inherits that bias — which makes refreshing it the precondition for trusting any of the work below.

The six-month review

An afternoon, with the logs open. Six questions, in this order, because each one’s answer changes what the next one means.

• Is the traffic what we thought? Input length distribution, language mix, the top intents, volume by segment. Compare against launch. A shifted distribution invalidates everything downstream, including the eval set.
• What does a successful outcome cost now? Not cost per call — cost per outcome, including retries, fallbacks, the verifier, the classifier and the review time. Compare to the number in the original plan, if one was written.
• Which clauses in the prompt still earn their place? Take the longest prompt, and for each paragraph ask which case would fail without it. Any paragraph with no answer is a candidate for removal — and removal is testable if the cases exist, which is the argument for the cases.
• Which stages exist because of an old constraint? For each step in the pipeline, ask what it works around and whether that limit still applies. Chunking, multi-pass summarisation, regex-repairing JSON and elaborate retry ladders are the usual suspects.
• What is the model deprecation horizon? Check the announced dates for every model in use, including fallbacks. A fallback pinned to a retired model is a degradation path that fails when it is finally needed.
• What have the review queue and the failure logs been saying for six months? The corrections are a labelled dataset nobody asked for. Read a sample; the failure modes will not be the ones the prompt is defending against.

The order to change things in

Changing two of these at once makes both unmeasurable, and that is the characteristic way a refactor of an AI feature goes wrong: everything moves, quality shifts, and nobody can attribute it.

0. refresh the eval set from recent traffic       (nothing below is trustworthy
                                                   until this is done)
1. baseline: run current model + current prompt   -> the number to beat

2. change ONE thing:
     new model, same prompt        -> isolates the model
     same model, trimmed prompt    -> isolates the prompt
     removed stage, same rest      -> isolates the architecture

3. compare against the baseline, not against your memory of it

4. if two changes are both wanted, sequence them and keep the intermediate
   numbers -- otherwise a regression is unattributable and you will revert
   the wrong one

5. canary in production before default-on, and keep the previous
   configuration reachable by a flag rather than by a revert

Step zero is the one under time pressure that gets skipped, and skipping it makes every subsequent number meaningless. The production-side mechanics of steps four and five — percentage rollout with automatic rollback and replaying real traffic against a new model — are well covered elsewhere; what belongs here is the discipline of moving one layer at a time.

Refactoring towards less

The most valuable outcome of one of these reviews is usually subtraction, and it is worth going in with that expectation rather than with a list of improvements.

• A stage that can be deleted. Worth more than any prompt improvement: it removes a call, a failure mode, a latency contribution and a piece of code, permanently. Check every workaround for a limitation against the current capability before optimising it.
• A prompt that can be shortened. Shorter prompts cost less on every call, and instruction adherence degrades as instruction count rises, so removing dead clauses can improve behaviour as well as price. Only doable with cases; without them, nobody will dare.
• A model call that can become code. Six months of outputs frequently reveal that a step’s answer is nearly deterministic — the classifier that returns the same label for a recognisable pattern, the formatting step a template would do. Each one converted is a permanent reduction in cost and variance.
• A feature that can be retired. The review is the right moment to ask whether anyone uses it. A feature with usage that never grew, costing money and maintenance, is a candidate for deletion, and deciding that deliberately is far better than letting it decay unattended — which is the failure mode the post-mortem categories call working, but nobody used it.