Grounding: The Only Reliable Hallucination Fix

Grounding replaces a question the model cannot answer — is this true of the world? — with one it can: is this supported by the text in front of me? That substitution is the entire mechanism, and it only works if something in your system actually checks.

What grounding actually claims

The precise version comes from Rashkin et al.’s Attributable to Identified Sources framework (2023). A statement is attributable to a source if a reasonable reader, having seen the source, would agree the statement follows from it. Note what this does not require: it does not require the statement to be true. Attribution is a relation between a sentence and a document, which is precisely why it is checkable by machine while truth is not.

So a grounded system is one where every claim in the output stands in that relation to a span you can point at. Retrieval-augmented generation (Lewis et al., 2020) is the plumbing; attribution is the property. You can have the plumbing without the property, and most systems that call themselves grounded do.

Why the prompt version does not hold

“Answer only from the context below. If the context does not contain the answer, say you do not know.” This is a good instruction and it does help. It is not grounding, for a structural reason: it changes the conditional distribution, and nothing verifies the outcome. Parametric knowledge is still in the weights and still influences every token. The model has no mechanism for separating “I read this in the context” from “I know this”, because both arrive at the same place — a shift in logits.

The failure this produces is the extrinsic one from the taxonomy: plausible additions that are not in the source and frequently are true, which is why they survive review. Gao et al.’s ALCE benchmark (2023) was built to measure exactly this — whether models asked to generate text with citations actually produce statements their citations support — and the general finding across systems was that citation-supported generation is much harder than citation-shaped generation.

The four commitments

Grounding as an architecture decision means committing to all four of these. Three out of four is a system that produces attributions nobody checks.

• Evidence is provided, not recalled. The context contains the material the answer must come from, retrieved by a component you control, with provenance attached to every chunk: source id, timestamp, trust tier.
• Output carries span-level attribution. Not a bibliography at the end — per claim, a pointer to the chunk id and ideally the character offsets. If the model can emit a sentence with no pointer, your schema is not enforcing anything.
• A verifier runs after generation. Every claim–span pair goes through an entailment check before the response reaches a user. This is the commitment that most systems skip and it is the one that makes the other three real.
• There is a defined path for “not in the evidence”. A refusal, a partial answer with the gap named, or an escalation. Without it, the model’s only way to satisfy your schema when retrieval fails is to invent an attribution — and it will, because a well-formed answer is the higher-probability continuation.

The verifier, concretely

A natural-language-inference model over (premise = retrieved span, hypothesis = generated claim), thresholded. Cross-encoder NLI models are small, fast and run locally; a second LLM works and costs more. The shape:

# Response schema the generator must satisfy:
# {"claims": [{"text": "...", "chunk_id": "doc7#3"}, ...]}

def verify(response, chunks, threshold=0.7):
    unsupported = []
    for claim in response["claims"]:
        span = chunks.get(claim["chunk_id"])
        if span is None:                       # cited a chunk we never sent
            unsupported.append((claim, "dangling-citation"))
            continue
        p = nli_entailment(premise=span.text, hypothesis=claim["text"])
        if p < threshold:
            unsupported.append((claim, f"entailment={p:.2f}"))
    return unsupported

flagged = verify(response, chunks)
if flagged:
    # Do NOT silently strip. Either regenerate with the flagged claims
    # named, degrade to an evidence-only answer, or escalate.
    handle_ungrounded(flagged)

The dangling-citation check on its own catches a surprising amount, and it costs nothing: a chunk id the model produced that you never sent is a fabrication with a machine-checkable signature. Log the entailment scores; their distribution over a week tells you where your threshold should be far better than a guess does.

Chunk granularity decides whether any of this works. A verifier can only check a claim against a span short enough that entailment is a meaningful question — a paragraph works, a 4,000-token document does not, because almost any claim is “not contradicted” by a long enough passage. This pushes back on retrieval design: chunks want to be small enough to quote and large enough to stand alone. Overlapping windows help, and keeping the character offsets of each chunk within its parent document lets you show a user the highlighted source, which is the feature that makes attribution worth its cost to anyone outside engineering.

Budget for it honestly. A verified answer costs one generation plus one decomposition plus one entailment call per claim, and the tail latency is the sum rather than the max unless you parallelise the checks. In practice the entailment step is short-context classification, so a small model or a local cross-encoder keeps the marginal cost near the noise floor; running it on the same frontier model you generated with is what makes teams conclude grounding is unaffordable.

Where the failure moves to

Grounding does not eliminate the failure. It relocates it, and the new location is one you can staff.

• Into retrieval. A confidently wrong answer sourced from a stale wiki page is now a corpus problem with an owner. This is a real improvement — corpora can be fixed — but it means retrieval quality is now your accuracy ceiling.
• Into the corpus’s write path. Anything that can be retrieved can be poisoned, deliberately or by neglect. That risk is the subject of context poisoning, and it is the direct cost of the substitution this page recommends.
• Into the verifier’s errors. NLI models have their own failure modes, especially on numbers, negation and quantifiers. Sample the flagged and the unflagged, and treat the verifier as an instrument you calibrate rather than an oracle.