Measuring Productivity Gains Honestly

Almost every productivity figure attached to an AI rollout comes from asking people how much time they saved. That method has a predictable direction of error, and the fix is not a better survey — it is a design in which the answer is not supplied by the people being measured.

Narrowing “productivity” to something countable

Productivity is output per unit of input. Before anything can be measured, both halves have to be fixed to something a query can count, and the narrowing is where most of the intellectual work is.

Pick a task, not a role. “Engineering productivity” is not measurable in a quarter; “median time from pull request opened to first review comment” is. “Legal team efficiency” is not measurable; “contracts reviewed per reviewer-day, for contracts of type X” is. The narrowing feels like a retreat and it is the opposite: a narrow metric with a real baseline supports a conclusion, and a broad one supports a slide.

Then fix the input side, because this is where the arithmetic quietly breaks. Time saved is only a gain if the time went somewhere countable. If drafting takes twenty minutes less and the person spends those twenty minutes on the next ticket, output rises and you can see it. If they spend it in a meeting that was going to expand anyway, nothing observable changed. Both are real outcomes; only one is a productivity gain, and a design that cannot tell them apart will report the first while measuring the second.

Why asking people cannot settle it

Not because people are dishonest. Because the question asks for something nobody has access to, and the ways it goes wrong all push the same direction.

• The comparison is counterfactual. “How long would this have taken without the tool” asks about a task that did not happen. The respondent substitutes a remembered similar task, and remembered tasks are not a random sample — the memorable ones are the long ones.
• The saving is salient and the cost is not. The twenty minutes of drafting that vanished is a distinct experience. The four minutes spent reading the draft, the two spent re-prompting, and the one wasted following a wrong suggestion are diffuse and attributed to normal work.
• The question carries its own expectation. It is usually asked by the team that introduced the tool, in a rollout everyone knows is being evaluated. That is a demand characteristic, and it does not require anybody to intend anything.
• Aggregation multiplies it. A modest per-task over-estimate becomes an annual hours figure by multiplying by task counts and headcount, so a small bias arrives at the executive summary as a large number.

Note what is not claimed here: any particular magnitude. The claim is directional and it follows from the structure of the question, which is why it holds without a study behind it. Self-report is still worth collecting — it is the best available instrument for whether people find the thing useful, which is a genuine question — but it cannot be the numerator of a productivity claim.

Three designs, ranked

Whichever design you use, the sample question comes first: is the study large enough to detect the effect you care about? The arithmetic is the same reasoning as detecting a difference between two non-deterministic systems, and it regularly shows that a four-week trial with a dozen people could not have detected a realistic effect even if it were there — a result which is itself worth knowing before you run it.

The metric that survives

A metric is survivable when improving it the wrong way is visibly punished by something else you are watching. Three properties, and a shape.

PRIMARY     a count with a clean denominator
            e.g. contracts reviewed per reviewer-day
                 tickets resolved per agent-hour logged in
                 PRs merged per engineer-week

GUARDRAIL   the quality that could be traded away for it
            e.g. reopen rate, escalation rate, revert rate,
                 defect rate, customer complaint rate

VOLUME      the amount of work available
            because a metric per person collapses when the
            inbound queue changes, and it always changes

Pre-commit that a guardrail moving in the wrong direction voids the primary result rather than being reported alongside it as a trade-off to be weighed later. Weighing it later is how a quality regression becomes a footnote under a headline number.

Report an interval rather than a point. “Between 4% and 14%, most likely around 9%” is a more useful and more defensible sentence than “9%”, and it prevents the number being multiplied by headcount three slides later as though it were exact.

Five confounds to pre-register

• Novelty. Usage and effort both spike in the first fortnight. Define the measurement window to start after it — a common choice is to discard the first two to four weeks — and write that down before you see the data, because deciding afterwards where the novelty ended is the same as choosing your result.
• Selection. If people opted in, the treated group is the group that expected to benefit. Randomised or staggered assignment removes this; a matched comparison does not, and should say so.
• Task mix. A tool that speeds up the easy work may cause people to take on more of it, so throughput rises while the hard work queues. Track the composition of completed work, not only its count.
• Substitution. Freed time going into other work is fine and it is a different claim from throughput. Decide in advance which claim you are making.
• Reactivity. People who know their throughput is being watched change it. Applies to both groups, which is another reason to have both groups.

A last suggestion about how to present the result. The most credible version of this work is usually not a headline percentage at all; it is a narrow, well-defined, modest number with its interval, its guardrails intact, and an explicit statement of what the design could not rule out. It survives the sceptic in the room, and it is the only kind of result that is still worth anything when someone re-reads it a year later — which is also why the measurement plan has to exist before the rollout rather than being assembled from whatever data survived.