AI Feature Launch Checklist

Thirty things to settle before an AI feature ships, each with the reason it is cheap now and expensive later, exportable as markdown.

Complete

0 of 30

20 blockers outstanding — the first is "An eval set exists, with examples you did not write while building".

View

Before you can call it done

An eval set exists, with examples you did not write while building — blockerAn eval assembled from the cases you already fixed measures your memory, not the feature.You know the score of the thing you are replacing — blockerWithout a baseline, 82% is a number rather than an improvement.The eval runs in CI, on every prompt changePrompts are code with no type system. CI is the only thing standing between you and a silent regression.The eval set is big enough to detect the difference you care aboutForty hand-written examples will show a ten-point swing that is entirely noise.

Cost

You have computed the cost of one average request, input and output separately — blockerOutput tokens usually cost several times input. A cost model that averages them is wrong in the direction that hurts.You have computed the cost of the worst plausible request, not the average one — blockerThe bill is set by the long tail: the user who pastes a whole document into a box you sized for a sentence.A hard spend ceiling exists, per user and in total, and something enforces it — blockerAn unbounded loop against a metered API is the most expensive bug in this category, and it happens on a weekend.Spend is alerted on before it is exceeded, not afterA monthly invoice is not a monitoring system.

Limits and failure

Input length is capped, and the cap is enforced server-side — blockerA client-side limit is a suggestion. The context window is not.Every call has a timeout, and the timeout is shorter than your user's patience — blockerA request with no timeout holds a connection until something else breaks first.Retries are bounded, backed off, and only fire on retryable errors — blockerRetrying a 400 is a way to pay four times for the same failure.There is a defined behaviour when the model is unavailable — blocker"The feature is unavailable right now" is a design. A spinner that never resolves is not.The rest of the product still works when this feature is downAn AI feature in the critical path turns a vendor's incident into your outage.

Safety and abuse

Untrusted input is not treated as instructions — blockerAnything retrieved, uploaded or pasted is data. If your prompt concatenates it next to your rules, it is now a rule.Model output is escaped and validated before it is rendered or executed — blockerOutput is untrusted input to the next system. Markdown that renders, SQL that runs, links that resolve.Any tool the model can call is scoped to what it needs, and irreversible actions need a human — blockerThe blast radius of a bad generation is exactly the permissions you handed it.The endpoint is rate-limited per user and per IP — blockerAn unauthenticated, unlimited LLM endpoint is a free API someone else will resell.You know what you will do when someone uses this to generate something awfulDeciding during the incident is how you end up deciding badly.

Privacy and data

You can say exactly what user data leaves your infrastructure — blockerSomeone will ask, in writing, and the answer needs to have been true all along.The provider's data-retention and training terms have been read for the plan you are actually on — blockerThese differ by tier and by endpoint, and the default is rarely the one you assumed.PII is redacted or minimised before it is sent, where it can beThe cheapest data breach is the data you never transmitted.Prompt and response logs have a retention period and an access boundary — blockerDebug logs of an AI feature are a full transcript of what your users typed.

What the user sees

It is clear to the user that they are reading generated output — blockerNot a legal checkbox — an expectation setting. Users forgive a wrong answer they were warned about.The user can check the answer: a citation, a source, a link, a way to see the inputAn unverifiable answer is only useful to someone who did not need it.You are not displaying a confidence number you have not calibrated"87% confident" from an uncalibrated model is a fabricated statistic on your own UI.There is a way to report a bad answer, and someone reads itThis is the only eval set that grows on its own.

Operating it

Prompts are versioned in source control, and the deployed version is identifiable from a log line — blocker"Which prompt produced this?" is unanswerable six weeks later unless you decided it would be.The model version is pinned, and you know what happens when it is deprecated — blockerA silent model upgrade is a silent regression across every prompt you have.Latency is measured at p95 and p99, not as an averageAverages hide the streaming stall that makes the feature feel broken.Someone owns this feature after launch, by name — blockerAI features rot faster than the rest of the product, because the ground under them moves.

Items ticked: 0 of 30
Blockers ticked: 0 of 20
Sections: 7

Export — copy this into your PR description or your runbook

# AI feature launch checklist

## Before you can call it done

- [ ] An eval set exists, with examples you did not write while building **(blocker)**
      An eval assembled from the cases you already fixed measures your memory, not the feature.
- [ ] You know the score of the thing you are replacing **(blocker)**
      Without a baseline, 82% is a number rather than an improvement.
- [ ] The eval runs in CI, on every prompt change
      Prompts are code with no type system. CI is the only thing standing between you and a silent regression.
- [ ] The eval set is big enough to detect the difference you care about
      Forty hand-written examples will show a ten-point swing that is entirely noise.

## Cost

- [ ] You have computed the cost of one average request, input and output separately **(blocker)**
      Output tokens usually cost several times input. A cost model that averages them is wrong in the direction that hurts.
- [ ] You have computed the cost of the worst plausible request, not the average one **(blocker)**
      The bill is set by the long tail: the user who pastes a whole document into a box you sized for a sentence.
- [ ] A hard spend ceiling exists, per user and in total, and something enforces it **(blocker)**
      An unbounded loop against a metered API is the most expensive bug in this category, and it happens on a weekend.
- [ ] Spend is alerted on before it is exceeded, not after
      A monthly invoice is not a monitoring system.

## Limits and failure

- [ ] Input length is capped, and the cap is enforced server-side **(blocker)**
      A client-side limit is a suggestion. The context window is not.
- [ ] Every call has a timeout, and the timeout is shorter than your user's patience **(blocker)**
      A request with no timeout holds a connection until something else breaks first.
- [ ] Retries are bounded, backed off, and only fire on retryable errors **(blocker)**
      Retrying a 400 is a way to pay four times for the same failure.
- [ ] There is a defined behaviour when the model is unavailable **(blocker)**
      "The feature is unavailable right now" is a design. A spinner that never resolves is not.
- [ ] The rest of the product still works when this feature is down
      An AI feature in the critical path turns a vendor's incident into your outage.

## Safety and abuse

- [ ] Untrusted input is not treated as instructions **(blocker)**
      Anything retrieved, uploaded or pasted is data. If your prompt concatenates it next to your rules, it is now a rule.
- [ ] Model output is escaped and validated before it is rendered or executed **(blocker)**
      Output is untrusted input to the next system. Markdown that renders, SQL that runs, links that resolve.
- [ ] Any tool the model can call is scoped to what it needs, and irreversible actions need a human **(blocker)**
      The blast radius of a bad generation is exactly the permissions you handed it.
- [ ] The endpoint is rate-limited per user and per IP **(blocker)**
      An unauthenticated, unlimited LLM endpoint is a free API someone else will resell.
- [ ] You know what you will do when someone uses this to generate something awful
      Deciding during the incident is how you end up deciding badly.

## Privacy and data

- [ ] You can say exactly what user data leaves your infrastructure **(blocker)**
      Someone will ask, in writing, and the answer needs to have been true all along.
- [ ] The provider's data-retention and training terms have been read for the plan you are actually on **(blocker)**
      These differ by tier and by endpoint, and the default is rarely the one you assumed.
- [ ] PII is redacted or minimised before it is sent, where it can be
      The cheapest data breach is the data you never transmitted.
- [ ] Prompt and response logs have a retention period and an access boundary **(blocker)**
      Debug logs of an AI feature are a full transcript of what your users typed.

## What the user sees

- [ ] It is clear to the user that they are reading generated output **(blocker)**
      Not a legal checkbox — an expectation setting. Users forgive a wrong answer they were warned about.
- [ ] The user can check the answer: a citation, a source, a link, a way to see the input
      An unverifiable answer is only useful to someone who did not need it.
- [ ] You are not displaying a confidence number you have not calibrated
      "87% confident" from an uncalibrated model is a fabricated statistic on your own UI.
- [ ] There is a way to report a bad answer, and someone reads it
      This is the only eval set that grows on its own.

## Operating it

- [ ] Prompts are versioned in source control, and the deployed version is identifiable from a log line **(blocker)**
      "Which prompt produced this?" is unanswerable six weeks later unless you decided it would be.
- [ ] The model version is pinned, and you know what happens when it is deprecated **(blocker)**
      A silent model upgrade is a silent regression across every prompt you have.
- [ ] Latency is measured at p95 and p99, not as an average
      Averages hide the streaming stall that makes the feature feel broken.
- [ ] Someone owns this feature after launch, by name **(blocker)**
      AI features rot faster than the rest of the product, because the ground under them moves.

_0 of 30 complete, 20 blockers outstanding._

What this checked: this is a checklist and a state machine, nothing more: it tracks what you have ticked, keeps that state in the URL so the link you send a colleague opens with your ticks intact, and exports the result as markdown. It does not inspect your code, your prompts or your infrastructure, and ticking every box here does not mean the feature is safe — it means you have considered thirty things that commonly go wrong. The items marked "blocker" are the ones that, in our reading, are hard to fix after launch rather than during it; that judgement is ours and you should disagree with it where your situation differs.

What this assumes: the list is deliberately generic and errs towards features that call a hosted model over an API. A self-hosted model changes the cost and privacy sections substantially. The URL stores one character per item in list order, so a link stays valid as long as the list only grows at the end — which is how it will be maintained.

Everything on this page runs in your browser. Nothing you paste is uploaded, logged, or put in the URL — only the settings above the input are, so a configured tool can be linked to.

Why these thirty and not others

Every item here is one that is cheap before launch and expensive afterwards. Pinning a model version costs one line today and costs a weekend when the provider deprecates it. Deciding what happens when the API is down costs a paragraph today and costs an incident later. Nothing on this list is about model quality, because model quality is the part teams already worry about — the failures that actually take AI features down are cost, availability, unescaped output and a prompt nobody can find the source of.

The three that get skipped most

The spend ceiling, because the demo cost eleven cents. The server-side input cap, because the textarea has a maxlength. And treating retrieved content as data rather than instructions, because the retrieved content is "your own documents" — right up until a customer uploads one. Each of these is a single afternoon before launch.

Use the export box to paste this into the pull request that ships the feature. A checklist in a document is a reminder; a checklist in the PR is a conversation, and the items someone argues with are usually the ones worth the argument.