How Often Models Hallucinate, According to the Studies

Published hallucination rates range from a couple of per cent to well over half, and the studies are not contradicting each other. They are measuring different tasks with different protocols, and two of the protocol choices move the number by more than any difference between models.

It is not one quantity

“Hallucination” covers at least four failure modes that are measured in different ways and have different remedies. A rate that does not say which is being counted is not usable.

A rate of three per cent from a grounded-summarisation benchmark and a rate of forty per cent from an adversarial closed-book set are both correct and describe different things. Putting them in the same table is the single most common error in writing about this. The taxonomy is developed further in types of hallucination and the mechanism in why models hallucinate.

What each named benchmark measures

• TruthfulQA (Lin, Hilton and Evans, 2021). Adversarially constructed questions where a common human misconception is the tempting answer. It was built to elicit imitative falsehoods, so its rate is a measure of susceptibility to a specific trap and by design not a base rate for anything.
• SimpleQA (OpenAI, 2024). Short fact-seeking questions with a single verifiable answer, selected to be hard. Importantly, it scores three outcomes rather than two — correct, incorrect, and not attempted — which makes it one of the few public benchmarks that lets you separate being wrong from declining to answer.
• FActScore (Min et al., 2023). Decomposes long-form generated text into atomic factual claims and checks each against a knowledge source. It produces a precision-style score over claims rather than a per-answer rate, which is a different and often more informative quantity.
• HaluEval and similar constructed sets. Pair each question with a known hallucinated answer to test whether models can recognise them. Measures detection, not production.
• Summarisation-faithfulness leaderboards. Vectara publishes an ongoing evaluation using its own Hughes Hallucination Evaluation Model, scoring whether a summary of a supplied document contains unsupported content. Narrow by construction — one task, one document set, one judge model — which is exactly why its numbers are low and why they should not be generalised.
• FreshQA and time-sensitive sets. Questions whose correct answer changes over time, which separates a knowledge-cutoff failure from a fabrication. See knowledge cutoffs.

Abstention changes the number threefold

The most under-reported variable in this whole area is whether a model is permitted to decline, and whether declines count as errors. It moves the headline more than any model difference.

Take 100 questions and one model's behaviour on them:

  answered correctly     55
  answered incorrectly   15
  declined to answer     30

Three defensible ways to report a "hallucination rate":

  1. Errors as a share of all questions:
       15 / 100  =  15%

  2. Errors as a share of attempted answers:
       15 / (55 + 15)  =  15 / 70  =  21.4%

  3. Anything not correct counts as failure:
       (15 + 30) / 100  =  45%

15%, 21%, and 45%, from identical behaviour.

Now compare two models with the same conditional accuracy but
different willingness to answer:

  Model A: 55 correct, 15 wrong, 30 declined  ->  15% by measure 1
  Model B: 79 correct, 21 wrong,  0 declined  ->  21% by measure 1

Both are right on roughly 79% of what they attempt. Model A looks 29%
better on the headline rate purely because it refuses more, and Model B
answers 24 more questions correctly.

Which measure is right depends entirely on your application. For a medical or legal reference tool, a decline is a good outcome and measure 1 is correct. For a search-replacement product, a decline is a failure and measure 3 is correct. Report all three, or state which one you chose and why. This is why abstention is a design decision rather than a bug, and why a benchmark that separates the three outcomes is worth more than one that does not.

Adversarial sets are not base rates

Most public hallucination benchmarks are deliberately hard. Questions are selected precisely because models get them wrong, often by filtering out questions that current models answer correctly. That construction is right for a benchmark, because an easy benchmark stops discriminating, and it makes the resulting rate useless as an estimate of what your users will experience.

Adversarial benchmark:  questions selected *because* models fail them.
Your traffic:            questions your users happen to ask.

If a benchmark keeps only questions that a reference model got wrong,
its reported error rate for that model approaches 100% by construction,
and for other models it measures overlap in failure modes.

The relationship between a benchmark rate and a traffic rate depends on
the difficulty distribution of your traffic, which nobody has measured
and which differs per application.

There is no conversion factor. Quoting an adversarial benchmark rate as
"models are wrong X% of the time" is a category error, not a rounding
error.

The same construction logic applies in reverse to grounded tasks. A faithfulness benchmark on clean, well-formed documents will report a low rate that does not survive contact with scanned PDFs, tables and contradictory sources — which is what retrieval-augmented generation failures are made of.

Measuring your own rate

1. Define the failure before sampling anything. Write down what counts as a hallucination for your task, with three or four worked examples of things that do and do not qualify. Without this, two annotators will disagree on a third of the cases.
2. Sample from real traffic, not from imagined queries. Stratify by query type if your traffic is heterogeneous, and report per stratum — an average over a mixed population hides the segment that is failing.
3. Have two people label independently and measure their agreement. If agreement is poor, the definition is the problem and no rate computed from it means anything.
4. Compute the confidence interval. It is the same binomial arithmetic as in the benchmark-scores page: at 100 samples and a true rate near 10 per cent, the 95 per cent interval is roughly plus or minus 6 points. A 100-item sample cannot distinguish a 5 per cent rate from a 15 per cent one.
5. Report all three abstention measures, so that a later change in how often the model declines does not silently move your series.
6. Re-run on the same items after any model, prompt or retrieval change. The absolute rate is less useful than the direction of change on a fixed set. The tooling for this is in measuring hallucination and regression testing.

Quoting somebody else’s figure

If you are going to repeat a published hallucination rate, the sentence needs to carry five things or it is misinformation with a citation attached: the benchmark, the task type, the model and its snapshot, the date, and how abstention was handled.

“Models hallucinate 20 per cent of the time” is not a statement about anything. “On a short fact-seeking QA benchmark published in 2024, this named model snapshot gave an incorrect answer to N per cent of questions it attempted, declining on M per cent” is a statement somebody can check. The first is what circulates; the second is what earns a citation. The general pattern of the first is the subject of AI statistics nobody can source.