Measuring Hallucination Rate in Your Own App

“Our hallucination rate is 3%” is meaningless without four more sentences: 3% of what unit, on which inputs, judged by whom, against what source. This page is those four sentences, made into a procedure you can run on a Friday and repeat next quarter.

Pick the unit before anything else

Every downstream number depends on this choice, and the choice is usually made accidentally.

• Response-level. One label per response: did it contain any unsupported claim? Cheap to grade, easy to explain to a stakeholder, and it saturates — a long answer almost always contains something, so a verbose model looks worse than a terse one that omits the same facts.
• Claim-level. Decompose each response into atomic claims and label each. This is the FActScore protocol (Min et al., 2023), built for biography generation, and it is the unit that actually tracks quality. It is roughly five to ten times the grading effort.
• Decision-level. Only claims that could change what a user does. Hardest to define, most useful to the business, and the one to reach for when the other two produce numbers nobody acts on.

Pick one, write it down, and never compare a rate computed one way to a rate computed another. Most of the disagreements about whether a model got worse are unit mismatches.

Building the evaluation set

Sample from real traffic, not from your imagination. Two hundred to three hundred inputs is enough to detect the size of change that matters in practice, and stratifying matters more than raw count: sample within categories you expect to differ — long-tail entities versus common ones, retrieval-hit versus retrieval-miss, first turn versus tenth turn — and keep the strata balanced so a shift in traffic mix does not masquerade as a quality change.

Freeze it. The set is a fixed asset with a version number, stored next to your code. Every input keeps the exact context it was served with, including retrieved documents, because grading a grounded answer without its source is not grading.

The rubric

Four labels, chosen so that a grader never has to decide how bad something is, only which box it falls in:

The hallucination rate is then (contradicted + unsupported) / (supported + contradicted + unsupported). Report contradicted separately as well; conflating the two hides the difference between a model that invents and a model that over-explains.

Grading, and checking the graders

Grade the first hundred claims with two humans independently and compute Cohen’s kappa on the four labels. If kappa is low the rubric is ambiguous, not the graders — go back and add examples to the label definitions until it is not. Doing this once is what makes every later number defensible.

Two disagreements come up every time and both are worth pre-deciding in writing. The first is scope: if a claim is half supported — the right entity, an embellished attribute — the decomposition was too coarse, and the fix is to split the claim rather than to invent a fifth label. The second is implicature: a response that is literally supported but strongly implies something unsupported. Decide once whether you grade the literal text or the reading a user will take, write it down, and note that the second is the one your users will complain about.

Distinguish, too, between reference-based grading — you hold a gold answer and check against it — and reference-free grading, where the grader checks the response against the provided source. The first is cheaper to run and can only cover inputs somebody already answered; the second scales to all of your traffic and inherits the source’s errors. Most teams need both: a small reference-based set that catches regressions precisely, and a larger reference-free sweep that catches the failures nobody anticipated.

Only then bring in a model grader, and treat it as an instrument that needs calibrating against the human labels: report the grader’s agreement with humans alongside the rate it produces. A model judge that agrees with humans 80% of the time cannot resolve a two-point difference between two systems, and quoting one that does is how evaluation theatre starts. Use a different model family for grading than for generation, and give the grader the source span rather than asking it to recall.

The harness

The loop, with the grading model left abstract because the shape is the point:

import json, math
from collections import Counter

LABELS = ("supported", "contradicted", "unsupported", "unverifiable")

def atomic_claims(response: str) -> list[str]:
    """One model call. Ask for a JSON array of standalone factual
    sentences, each independently checkable, pronouns resolved."""
    return json.loads(call_model(DECOMPOSE_PROMPT, response))

def grade(claim: str, source: str) -> str:
    """One model call per claim, with the source in context.
    Constrain the output to the four labels."""
    out = call_model(GRADE_PROMPT.format(source=source), claim).strip()
    return out if out in LABELS else "unverifiable"

def wilson(k: int, n: int, z: float = 1.96) -> tuple[float, float]:
    if n == 0:
        return (0.0, 1.0)
    p = k / n
    d = 1 + z * z / n
    c = p + z * z / (2 * n)
    h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return ((c - h) / d, (c + h) / d)

def run(eval_set) -> dict:
    counts = Counter()
    for item in eval_set:                       # frozen inputs + their sources
        response = call_system(item["input"])   # your app, not a raw model call
        for claim in atomic_claims(response):
            counts[grade(claim, item["source"])] += 1
    denom = counts["supported"] + counts["contradicted"] + counts["unsupported"]
    bad = counts["contradicted"] + counts["unsupported"]
    lo, hi = wilson(bad, denom)
    return {"rate": bad / denom, "ci95": (lo, hi), "n_claims": denom,
            "counts": dict(counts)}

Two details that are easy to skip and expensive to skip. Call your system, not the model — the rate you care about includes your retrieval, your prompt and your post-processing. And run the generation at the temperature you serve at, several times per input if that temperature is above zero, because a rate computed from one sample of a stochastic process is a point estimate of the wrong thing.

Reporting a rate honestly

Always with the interval. With 300 claims, a Wilson 95% interval on a small rate is several points wide, which means a change from 4% to 6% between two runs is not a change. Watching teams chase that noise for a quarter is a common way to waste a quarter.

Report per stratum as well as overall, because the aggregate hides the thing you can fix — the failure is usually concentrated in retrieval-miss cases or in long-tail entities, exactly as the monofact argument on why models hallucinate predicts. And record the model id, the provider and the date with every run, because in six months the first question about any number in this table will be which version produced it.