Writing the Post-Mortem for an AI Feature That Failed

5 min read · updated August 3, 2026

This page is not a post-mortem. It is how to write one, which is a more useful thing to have on hand, because the single most common reason an AI feature’s post-mortem is worthless is that the evidence it needed was never recorded — and that is a decision taken months before anything goes wrong.

Why this one is harder to write

A classical post-mortem has a timeline, a trigger and a fix. It works because the system is deterministic: given the same inputs and the same code, the same thing happens, so a root cause is discoverable by reconstruction.

An AI feature breaks three of those assumptions at once. The output is non-deterministic, so “it produced this” is not reproducible — the same request run again may produce something fine, which is not evidence that the problem is gone. The dependency changes without your involvement, so a behaviour change may have no corresponding change on your side at all. And the failure is often not an incident with a start time: quality degrades, or was never good, and the “event” is a decision to stop rather than an outage.

That last one matters most, because the most valuable post-mortems in this space are about features that did not break — they were built, shipped, maintained, and never justified their cost. That has no page in an incident tracker and no alert, and it is the failure most worth understanding.

A taxonomy of how they fail

Six categories. The value of naming them is that the remedies are entirely different, and a post-mortem that lands in the wrong category produces a fix for a problem the team did not have.

Category	Description
Never worked	The accuracy required was never achieved, and the demo concealed it. Symptom: no evaluation existed, or it was built after the decision to ship. The finding is about the process that let it ship, not about the model.
Worked, then drifted	Quality declined over time — inputs shifted, an index changed, a provider updated a model. Symptom: no canary suite, no logged prompt or model version, and a decline nobody can date. The remedy is instrumentation, not a better prompt.
Worked, cost too much	Accuracy was fine and the unit economics never closed. Symptom: cost per successful outcome was never computed, only cost per call. The remedy is a decision made earlier, not an optimisation made later.
Worked, nobody used it	Shipped, accurate, ignored. Symptom: no baseline for what people did before and no measure of the problem's actual frequency. The remedy is almost always to have asked the not-using-ai questions first.
Worked, then was unmaintainable	Every model migration, prompt fix or provider change required disproportionate work, and the team stopped keeping up. Symptom: prompts with no owner, no eval to migrate against, one hard-coded provider.
Caused harm	Wrong output reached someone and had a consequence — a customer misled, data exposed, an action taken that should not have been. The only category where the post-mortem is urgent, and the only one where the root cause is usually a missing rail rather than a model shortcoming.

Two of these — cost and non-use — are almost never written up, because neither produces an incident. They should be, and the review is worth scheduling deliberately at six and twelve months rather than waiting for something to break.

The evidence you must already have

This is the section to act on today rather than after the failure. Every field below is one that cannot be reconstructed later; if it was not recorded at the time, the corresponding question is permanently unanswerable.

PER REQUEST                     ANSWERS THE QUESTION
  model id + version            "did the provider change something?"
  prompt id + version           "was it our change or theirs?"
  input hash + token counts     "did the inputs shift?"
  validation outcome            "was it detectably wrong at the time?"
  degradation rung / fallback   "how much of this traffic was already degraded?"
  cost                          "what did a successful outcome cost?"
  latency (ttft and total)      "was it slow, or was it wrong?"

PER OUTCOME
  user action after the output  accepted, edited, ignored, undone
  review verdict + correction   the only ground truth you will ever get free

PERIODICALLY
  a fixed canary set, run on a schedule, results retained
                                "when did behaviour change?" -- unanswerable
                                without a time series, and no log of live
                                traffic substitutes for it, because live
                                traffic changes too

The canary series is the one people regret. Live-traffic metrics confound two variables — the system changed and the inputs changed — and a fixed set run weekly separates them, which is the difference between a post-mortem that says “behaviour changed on or around this date” and one that says “quality seems to have declined at some point”. The mechanics belong to catching provider-side changes, and the reason to run one is largely that it is what makes the later investigation possible at all.

The template

Structure it so that the honest answers are easy to give and the evasive ones are visibly missing.

What it was meant to do — the original claim, in the words used at the time, with the metric it was supposed to move. Quoting the original framing rather than paraphrasing it is what makes the rest of the document hard to soften.
What actually happened — the observed numbers, with their source and their sample size. If a number cannot be produced, say that it cannot, and say which missing instrumentation is the reason. That sentence is often the most actionable line in the document.
Which category — from the taxonomy above, chosen explicitly. Arguments about the category are arguments about the cause, and they are better had here than in the remedies section.
The decision points — not a timeline of events but a list of moments where a different choice was available: when it was scoped, when it shipped without an eval, when the cost model was not built, when the metric went unread for a month. This is the section that produces transferable lessons; a timeline of symptoms does not.
What was true and remains true — the parts that worked. Post-mortems that condemn the whole feature lose the extraction pipeline that was fine, and the next team rebuilds it.
What changes — process changes, not resolutions. “We will evaluate before shipping” is a resolution. “A feature cannot pass review without a case set in CI” is a change.

One discipline is worth importing wholesale from classical incident practice: separate what happened from who did it, and write the document so that a reasonable person with the information available at the time would have made the same call. In this field that is usually literally true — the information available at the time was a demo, and the whole point of the exercise is to change what information is available next time.

Reading someone else’s

Published write-ups of AI features that did not work are rarer than the successes and worth more, but they need reading with two adjustments.

First, check the date against the model generation, and check what was actually attributed. A conclusion of the form “the model could not do X” may not survive two generations of models; a conclusion of the form “we had no way to tell whether the output was right” is about system design and does not expire. The second kind transfers; the first often does not, and most disagreement about published failures comes from treating one as the other.

Second, look for the denominator. “It got things wrong” without a rate, a sample size and a comparison to whatever preceded it is an impression. A write-up that states its evaluation method is worth ten that describe an experience, and the same standard applies to the one you are about to write — which is the most practical reason to build an evaluation set early: it is what lets you say something true about your own feature afterwards, in either direction.