Measuring AI Feature Adoption
6 min read · updated August 3, 2026
A feature that works perfectly and a feature that fails twice before succeeding produce the same graph, except the failing one produces a taller graph. Counting calls is not a weak metric; it is a metric that points the wrong way.
Why invocations overstate a bad feature
The confound is structural rather than statistical. Every recovery affordance in this cluster generates an invocation: a regenerate is a call, a rephrase is a call, an automatic retry after a malformed response is a call. So a user who got what they wanted immediately contributes one, and a user who fought the feature through four attempts and gave up contributes four.
Which means the metric moves in the wrong direction under exactly the conditions you most need to detect. Ship a prompt regression and invocations rise. Ship a fix and invocations fall. Any dashboard whose headline number is calls per user will report a quality regression as growth, and it will do so confidently.
The fix is not a better denominator. It is to count outcomes, and to deduplicate by the unit the user cares about — the task — rather than by the unit your infrastructure emits.
The outcome ladder
Six stages, each observable, each dropping some users. The shape of the drop tells you where the problem is, which no single number does.
invoked user asked for something | drop here = latency or abandonment delivered output completed and was displayed | drop here = the answer was visibly not useful viewed output was on screen long enough to read | drop here = quality, format, or trust accepted copied, inserted, applied, sent | drop here = it looked right and wasn't kept still present at T+7d, not reverted | unedited kept with little or no modification Dedupe rule: collapse the retry chain first. Four attempts on one task is ONE task, with attempts=4 as an attribute. Every rate below is per task, not per call.
The two most informative transitions are the ones teams instrument last. Delivered to viewed is a latency signal: users who left before reading were not commenting on quality, they were commenting on the wait. Accepted to kept is the only transition that catches an output that looked right and was not, because it is the only one measured after the user found out.
Read as a shape rather than as six numbers, the ladder is diagnostic in a way no single rate is. A large drop at the top with healthy rates below means the feature works and nobody is waiting long enough to find out — a latency problem, and prompt tuning will not touch it. Healthy rates at the top with a drop at accepted means people are reading the output and rejecting it, which is quality. A clean run all the way to accepted followed by a collapse at kept is the worst shape on the list: it means the output is persuasive and wrong, users are finding out later, and every metric above the last one was reporting success.
Not every product can measure the whole ladder, and it is worth being honest about where yours stops. A copy-to-clipboard flow ends at accepted — you never see what happened to the text — which is a real argument for keeping AI output inside a surface you own rather than handing it off, quite apart from the interaction benefits. Where the ladder is truncated, say so on the dashboard rather than letting the last measurable rung be read as the outcome.
What each metric can and cannot tell you
| Metric | Description |
|---|---|
| Task completion rate | Tasks that reached 'accepted' over tasks started. Can tell you whether the feature works end to end. Cannot tell you whether the accepted output was correct — acceptance is a decision made before the consequences are known. |
| Attempts per task | The retry chain length. The single most sensitive quality signal available and it needs no widget: a rise means outputs are not good enough on the first pass. Cannot distinguish a bad model from a bad prompt from an unclear request. |
| Retention of output | Accepted output still present some days later. The strongest available proxy for actual usefulness, because reverting takes deliberate effort. Only measurable where you own the destination — an editor, yes; a copy button, no. |
| Edit distance on accepted output | How much the user changed before shipping. Gives a continuous quality measure and a free corrected reference. Cannot separate 'wrong' from 'right but not in my voice', which is why the distribution is more useful than the mean. |
| Abandonment mid-stream | Navigated away while generating. A latency metric wearing quality clothes. Keep it in a separate pool; treating it as dissatisfaction sends teams to tune prompts when they needed a faster model. |
| Repeat use at 7 and 28 days | Whether anyone came back. Slow, coarse, and the only one of these that is about the feature rather than about individual outputs. Cannot attribute a change to any specific cause. |
What none of them establish is correctness. An accepted, kept, unedited output can be wrong — a user accepted it, nobody checked, and the error is now in a document. Only evaluation on production traffic measures that, and these metrics are not a substitute for it. They measure whether the feature is used and whether its output survives, which are different questions from whether it is right.
Cost per accepted output
One number decides whether an AI feature can exist at its current price, and it is not cost per call:
cost_per_accepted = total_spend / accepted_count where total_spend includes EVERY attempt, including the ones that were regenerated away, plus reasoning tokens, plus any verification passes (attribution checks, guardrails, judges). Worked, with assumed inputs: 1,000 tasks 1.8 attempts per task on average -> 1,800 generations $0.004 per generation -> $7.20 620 tasks reached 'accepted' cost per call = $0.0040 cost per accepted = $7.20 / 620 = $0.0116 Nearly 3x the headline figure, and it is the 3x that has to fit inside what the feature earns.
The numbers above are assumptions to be substituted, not observations. What is not an assumption is the structure: attempts multiply spend and acceptance divides it, so a quality improvement shows up twice in this ratio. That is the argument for spending on quality that a cost-per-call view never produces, and it is why this metric belongs next to the adoption ones rather than in a finance dashboard. See unit economics and attributing spend to features.
The event schema
type AiTaskEvent = {
taskId: string; // stable across the retry chain — the key
// that makes every rate per-task
attempt: number; // 1, 2, 3...
feature: string;
promptVersion: string;
model: string;
stage: "invoked" | "delivered" | "viewed" | "accepted"
| "kept" | "reverted" | "abandoned";
ttftMs?: number;
totalMs?: number;
costMicros?: number; // this attempt only; sum over the chain
editDistance?: number; // on 'accepted'
survivedDays?: number; // on 'kept'
};The whole schema turns on taskId. Without a key that is stable across retries there is no way to collapse the chain, and every rate you compute is a rate per call — which is the metric this page opened by rejecting. Generate it where the user starts the task, not where the request is made.
The promptVersion and model fields do the second most work, for a reason specific to this domain: the thing being measured changes underneath you. A prompt edit, a model upgrade, or a provider-side update you were not told about can move every rate on this page, and without those fields the before and after are pooled into one undifferentiated series in which nothing is attributable. Carry them on every event even when nothing is changing, because the moment you want them is always retrospective.
Two habits make the resulting numbers usable rather than merely available. Segment by feature and never report a company-wide AI adoption number — different features have different ladders, and the aggregate is a weighted average of unrelated things that moves when traffic mix moves. And annotate the series with deploys: prompt versions, model changes, UI changes. Almost every question anyone asks of this data is “did that change help”, and a chart without the change markers on it cannot answer it.
A closing caution about how these numbers get used. Every metric here measures whether output survives contact with a user, which is a reasonable proxy for value and a poor one for correctness — and it is also, uncomfortably, a set of numbers that improves if the feature becomes more agreeable rather than more accurate. Optimising them without an accuracy measure alongside is the exact mechanism described in dark patterns: a reasonable proxy, promoted to an objective, selecting for something nobody chose.