Misleading AI Statistics: Six Recurring Errors

Numbers about AI go wrong in a small number of repeating ways. None of the six below requires anyone to have acted in bad faith, which is exactly why they survive editing: each one is a plausible reading of a real figure. All the numbers in this page are illustrative, chosen to make the arithmetic visible.

1 · The moving denominator

A quantity is computed per one thing and reported per another. This is the single most common source of order-of-magnitude errors in AI coverage, and it happens in the gap between a technical source and a summary of it.

The pattern: a figure for annual consumption at a facility is divided by an estimate of requests to produce a per-request number, that number is rounded, and the rounded number is then multiplied back up by a different request estimate in a later article. Each step is arithmetically valid; the compounding of two independent estimates is not disclosed, and the final figure carries none of the uncertainty of either.

The check: ask what was measured and what was computed. A measured facility total divided by an estimated request count is an estimate, and its error bars are those of the divisor, which is usually the number nobody has. The water-per-query figure is the textbook case.

2 · Relative change without the base

“Errors fell by 50%” is compatible with falling from two in a hundred to one in a hundred, and with falling from sixty in a hundred to thirty. The two support entirely different decisions and the sentence does not distinguish them.

The reverse form is equally common and more persuasive: a small absolute change in a rare event produces a large relative change. If an event occurs three times in a period and four times in the next, that is a 33% increase, and it is also noise.

The check: demand both numbers. A percentage change without the base and the count is not a finding.

3 · Aggregation across incomparable units

“AI uses X energy per query” treats a one-line completion from a small model and a long answer from a large reasoning model as the same event. The underlying quantities differ by orders of magnitude, driven by model size, active parameters, output length and whether the model spends reasoning tokens before answering.

An average over a mixture is a real number about the mixture and tells you almost nothing about any member of it. It becomes actively misleading the moment it is used to reason about a specific case, which is what an average in a headline invites.

The check: ask what the unit was and how much it varies within the aggregate. Where the spread is orders of magnitude, the mean is not a summary — a distribution or a range is.

4 · Self-report treated as behaviour

Survey questions of the form “do you use AI at work” measure what people say when asked, which is influenced by how the question is worded, whether the tool is labelled as AI, whether the respondent thinks the answer reflects well on them, and who is doing the asking. Adoption figures derived this way are not measurements of usage.

The specific distortion worth knowing: features that are not marketed as AI are under-reported, and features that are prominently marketed as AI are over-reported. Both effects push the same measured quantity in opposite directions depending on the year, which makes trends across surveys with different wording unreliable even when each survey is well conducted.

The check: find the question wording and the sampling frame. A survey without both published is not evidence, and a survey with both is often evidence of something narrower than the headline.

5 · Extrapolating a short curve

Three points on a rising line support many curves. Fitting an exponential to them and projecting five years out produces a number with the visual authority of a measurement and the epistemic status of a guess.

This error has been made in both directions in this field, which is why it is worth naming rather than using as ammunition. Extrapolations of capability growth and extrapolations of energy consumption have both been produced from short series and both have missed, because both assumed the constraint that eventually bound would be the one operating at the time of the fit.

The check: ask how many points the fit uses, and what would have to keep being true for it to hold. Projections that name their binding constraint — fabrication capacity, power availability, data — are doing something defensible; projections that just continue a line are not.

6 · Base-rate neglect

The most consequential of the six, because it produces conclusions that are not merely imprecise but backwards. A test with a low false-positive rate, applied to a population where the thing being tested for is rare, produces mostly false positives.

Illustrative, not measured.

  population           10,000 items
  true positives            200  (2% base rate)
  detector sensitivity      90%
  detector false-positive    5%

  flagged and true      200 x 0.90  = 180
  flagged and false   9,800 x 0.05  = 490

  of 670 flags, 490 are wrong — 73%

Both stated rates sound good. The result is that most flags are wrong, and no improvement in sensitivity fixes it, because the false positives come from the size of the negative population rather than from the test’s quality on positives. This arithmetic is why AI-detection tools produce so many false accusations even when their published accuracy is high.

The check: ask for the base rate. If it is not given, the accuracy figure cannot be converted into anything you can act on.