Proving ROI on an AI Project

ROI on an AI project is not hard to calculate. It is hard to calculate credibly, and the difference is entirely decided by things you must do before anyone gets access to the tool. Afterwards there is nothing to do but argue.

Why ROI computed afterwards is unfalsifiable

The usual sequence is: roll out the tool, wait a quarter, then ask finance to work out what it saved. At that point the only available inputs are a usage figure, a bill, and people’s recollections. From those you can construct any number you like, and the number you construct will be the one whoever sponsored the project was hoping for — not through dishonesty, but because every free parameter gets set in the direction of the expected answer.

The specific failure is that nothing in the after-the-fact calculation can come out negative. If handle time fell, the tool did it. If handle time rose, the ticket mix got harder. If nothing changed, adoption was too low to show up yet. A quantity that cannot come out against you is not a measurement, and everyone in the room senses that even when they cannot say why, which is why these presentations produce polite nods and no budget.

Everything below is about restoring the possibility of a negative result. That is the only thing that makes a positive one worth anything.

One primary metric, defined operationally

One. Not a dashboard. A primary metric chosen in advance, with secondary metrics allowed but explicitly labelled as secondary, because five metrics with a threshold each is five chances to find a winner and report it as the result.

“Operationally defined” means somebody who has never met you could compute it from the same data and get the same number. Write the definition as a query, not as a phrase.

• Bad: “agent productivity”.
• Better: “median minutes from first agent view to first outbound reply, for tickets in queue support-tier-1, excluding tickets reopened within 24 hours”.
• Also acceptable: a count with a clean denominator — tickets resolved per agent-hour logged in, contracts reviewed per reviewer-day, invoices matched per run.

Two properties make a metric survivable. It should be hard to improve by doing the wrong thing — reply time falls beautifully if agents send useless replies faster, so pair it with a guardrail metric such as reopen rate, and pre-commit that a guardrail breach voids the result. And it should already be collected, because a metric introduced at the same time as the tool has no history and therefore no baseline.

The baseline window

A baseline is not last month. It is a window long enough to contain the variation the metric normally has, measured before anyone knows who is getting the tool.

Pick the window from the seasonality of the work rather than from convenience: at least one full cycle of whatever periodicity the metric obviously has — weekly for anything staffed on a rota, quarterly for anything tied to billing or reporting deadlines. Then compute the metric’s normal week-to-week spread in that window. That spread is the number that tells you what a detectable effect looks like. If the metric routinely swings by 15% between quiet and busy weeks, an 8% improvement is not a finding, and no amount of enthusiasm converts it into one.

This is also where you discover the awkward thing early: for a lot of candidate use cases the baseline does not exist because nobody was measuring, and you cannot construct it retroactively. That is a real finding. It usually means picking a different first use case rather than proceeding and hoping — which is why measurability carries the most weight in the use-case rubric.

Four comparison designs, ranked

The comparison is what turns a change in a number into a claim about a cause. Ranked from strongest to weakest, with what each can and cannot rule out.

The practical recommendation is the second row. A staggered rollout costs nothing extra, is easy to justify to the people who go last (“we are switching teams on in waves” is true and normal), and produces a comparison you can defend in a room containing a sceptic. It is also the design that makes a staged rollout with exit criteria and a measurement plan the same piece of work rather than two.

The pre-registration

One page, circulated before access is granted, so that the analysis cannot be selected after the data arrives.

MEASUREMENT PLAN — written before rollout

PRIMARY METRIC   exact definition, as a query
GUARDRAILS       metrics that void the result if they move the wrong way
BASELINE         window, and the metric's normal spread inside it
DESIGN           randomised / staggered / matched / before-after, and
                 how units are assigned
UNITS            what is randomised: person, team, ticket, document
DURATION         start and end dates, fixed now
EFFECT OF INTEREST
                 the smallest change that would change the decision
ANALYSIS         the comparison to be run, named in advance
EXCLUSIONS       what data is dropped and why, decided now
DECISION         what result leads to what action, written as if-then

Two lines do most of the work. Effect of interestforces the conversation about whether the study can detect what you care about at all — the arithmetic for that is the same sample-size reasoning used for comparing two non-deterministic systems, and it regularly reveals that a four-week pilot with eleven people could not have detected the effect even if it were real. Exclusions matters because the single most common way a result gets manufactured is by dropping the awkward weeks after seeing them.

The cost side is not the invoice

Return is the numerator and it gets all the attention. The denominator is where the number quietly becomes wrong, because the model bill is the smallest term in it.

• Inference spend, attributed to this feature rather than to a shared key — which requires tagging requests before the pilot, not after, because retrospective attribution of a shared bill is guesswork.
• Licences and platform fees for anything bought to support it, including the observability and evaluation tooling.
• Engineering time at a loaded rate, including the people who reviewed, integrated and secured it, not only the ones who built it.
• Review time — if a human checks the output, that time is part of the cost of the new process and it is often the largest single term. A workflow that halves drafting time and adds a mandatory review step may have moved cost rather than removed it.
• Ongoing operation at the rate it will actually run at, not the pilot rate. Pilots are staffed by people who care.

Put those five into the denominator and a lot of enthusiastic ROI figures come back down to something modest and defensible — which is the better outcome, because a modest defensible number survives the finance review and an implausible one does not survive contact with a sceptic. The gross-margin view of the same arithmetic, for a feature you sell rather than one you use internally, is unit economics, and it uses the same denominator.