Retiring an AI Feature Nobody Uses

Retiring a feature is a comparison between what it returns and what it costs to keep alive. The first number is usually available. The second is almost never computed, because most of it is not on the invoice.

What is not in the calculation

What it cost to build is not in the calculation. That money is spent whatever you decide, and including it can only produce the wrong answer. This is elementary and it is also the argument the room will actually have, so it is worth naming out loud before the numbers go up: the question is not whether the project was worth doing, it is whether keeping it running is worth doing from today.

Two adjacent things are also not in it. The reputation of whoever sponsored it — retiring a feature that did not find a use is a normal outcome of trying things, and treating it as a failure is how you get an organisation that never retires anything. And the sunk effort of the people who built it, which is real and deserves acknowledgement in a different conversation from this one.

What is in the calculation is everything that will be spent next year if you do nothing.

What a live feature costs to keep

Six terms. The inference bill is usually the smallest of them, which is why a feature that costs “almost nothing to run” is frequently expensive to keep.

annual carry =
    inference        metered spend at current volume
  + licences         any tool bought for this feature alone
  + on-call          the fraction of incidents this feature
                     causes, times what an incident costs
  + migration        model deprecations and API changes per
                     year, times the work each one takes
  + assurance        re-running evals, the annual review, the
                     re-tiering when something changes,
                     answering it on every security questionnaire
  + attention        the fixed cost of one more thing existing:
                     it appears in every audit, every dependency
                     review, every onboarding document

The migration and assurance terms are the ones that make this different from retiring an ordinary feature. A model-backed feature sits on a dependency that changes on someone else’s schedule, and it is a live data flow that has to be re-justified every time a customer sends a security questionnaire. Both are small per event and neither is optional.

Worked, with every figure an assumption: $80 a month of inference; no dedicated licences; one incident a year at roughly two engineer-days; two provider changes a year at a day each; three days of assurance work across the year.

inference   12 * 80                    = $  960
on-call      2 days  * $800/day        = $1,600
migration    2 days  * $800/day        = $1,600
assurance    3 days  * $800/day        = $2,400
                                  carry = $6,560 / year

inference is 15% of the cost of keeping it.

That ratio is the point. Judging a feature by its inference bill will keep alive a great many things whose real cost is seven or eight times larger and lands on engineers rather than on the budget line anybody is watching.

The comparison

Against the carry, put what the feature returns, measured the same way you would have measured it at launch.

return = A * u * v

  A   users who used it in the last 30 days — active, not
      entitled, and not counting the team that built it
  u   uses per active user per year
  v   value of one use, in the same units as the carry:
      minutes saved x loaded rate, or a revenue effect you
      can name

Keep it if   return > carry, comfortably.
If it is close, it is a retirement: a feature that is
marginal on paper is a feature nobody will maintain.

Two honesty rules for A. Exclude the team that built it, whose usage is not evidence. And use a rolling 30-day active count rather than a cumulative one, because “400 people have tried it” and “nine people use it” are frequently the same feature. If you have per-feature cost attribution, this whole comparison is a query rather than a project — and if you do not, the reconstruction is itself a reason to add it.

Three endings, not one

Retirement is the default reading of a failed comparison, and it is often the third-best option.

• Narrow it. Very often the aggregate looks poor because the feature works for one segment and not the rest. Keeping it for the segment where v is high, and removing it elsewhere, can flip the comparison while cutting the carry — fewer edge cases, fewer questionnaires, a smaller blast radius.
• Absorb it. If the capability is worth having but not worth maintaining separately, fold it into a platform component or replace it with a bought equivalent. The carry moves to somewhere that is already paying it, which is the honest version of “consolidation”.
• Retire it. Remove the feature, delete the data, close the vendor line. The cleanest outcome and the one that actually returns the carry to the budget, which the other two only partly do.

Whichever you choose, write down the condition under which it would be worth revisiting — a cheaper model tier, a data source that did not exist, a segment that grew. That sentence is what stops the decision being re-argued every quarter on no new information.

Two failure modes are worth naming before the meeting, because both are arguments that sound like analysis. The first is the rescue plan: one more model, one more prompt revision, one more month. It is occasionally right, and the test is whether anyone can say what specifically would be different — a named failure mode with a named fix — rather than that the new model is better in general. The second is the strategic exemption: the feature stays because it demonstrates that the company does AI. That may be a real reason, but it should be stated as one and funded from a budget that expects no return, rather than smuggled into the comparison as value.

The runbook

T-60  Decision recorded, with the numbers. Named owner for
      the retirement. Check for contractual notice periods
      before announcing anything externally.

T-45  Tell the people who use it, individually if there are
      few enough to name. Offer the alternative, if there is
      one, before they ask for it.

T-30  Disable new activation. Existing users keep working.
      In-product notice with the date on it.

T-14  Second notice. Export path published for anything the
      feature accumulated that users would want.

T-0   Off. Feature flag, not a deploy, so it can be
      reinstated for a day if something was missed.

T+14  Remove the code path. Revoke the API keys. Cancel or
      downgrade the vendor line — this is the step most
      often forgotten, and it is where the money is.

T+30  Delete the data on the retention schedule, including
      logs and any vector index built from it. Record the
      deletion.

T+30  Close the risk-register rows and remove the entry from
      the AI register, so the next audit does not ask about
      a system that no longer exists.

KEEP  The evaluation set and the write-up. They cost nothing
      to store and they are the most reusable thing the
      project produced.

The T+14 line deserves emphasis because it is where retirements leak. A feature can be switched off in the interface and still be costing money in a vendor subscription, still be holding an API key, and still be listed as a live data flow on a security questionnaire somebody answers next quarter. Retirement is finished when the vendor line is closed and the register entry is gone, not when the button disappears.

If the feature was visible to customers, the announcement is a separate piece of work with its own constraints — notice periods, wording, and the question of what to say about why. That is communicating an AI change to customers, and the lead times in the runbook above should be checked against whatever your contracts require before T-45 is set.