Citations and Source Attribution Users Can Verify

“Sources: [1] [2] [3]” under a paragraph is not attribution. It is an assertion that some part of some document supports some part of the answer, and it is unfalsifiable, which is exactly the property a citation is supposed not to have.

Why footnote-style citations are theatre

Consider what a reader can do with a document-level citation. They can open a twelve-page PDF and search it. If the claim is wrong, they will not find the supporting sentence and cannot distinguish that from not looking hard enough. In practice, nobody checks, which means the citation performs trustworthiness without conferring it — the worst possible arrangement, because it increases confidence without increasing correctness.

A useful citation has three properties. It points at a span, not a document. It is verified before display, so a fabricated citation never reaches the user. And it is attached to a specific claim rather than to the answer as a whole, so a partially supported answer is visibly partially supported.

All three are achievable with a normal chat model and about eighty lines of code. Some providers now expose this as a first-class API feature that returns character ranges into the documents you supplied — Anthropic’s Citations API is one example — which removes the verification burden if you are on a provider that offers it. The approach below is what to do when you are not.

The bookkeeping starts at ingest

You cannot map a quote back to a location in a source document if you did not record where each chunk came from. This is the step people skip, and it cannot be retrofitted without a reindex.

@dataclass
class Chunk:
    id: str          # doc_id + ":" + str(index), stable across reindex
    doc_id: str
    text: str
    start: int       # character offset into the ORIGINAL document
    end: int
    heading: str     # "Billing > Refunds > Timing"
    url: str         # deep link target, if the source has one

Two traps. First, offsets must be into the document as the user will see it, not into your cleaned-up intermediate — if you stripped HTML, normalised whitespace or de-hyphenated a PDF before chunking, your offsets refer to a string that no longer exists anywhere. Either keep the cleaned text as the canonical rendering, or maintain an offset map through the transformation. The first option is much easier and is usually fine.

Second, chunk ids must be stable. If a reindex renumbers chunks, every citation you stored in a conversation log now points somewhere else, and your historical answers quietly become misattributed. Derive ids from content or from a durable position, never from enumeration order over a directory listing.

Ask for a quote, then verify it

The prompt change is small. Instead of asking for source numbers, ask for the exact sentence that supports each claim.

Return JSON:

{
  "claims": [
    {
      "text": "Refunds arrive within 14 business days.",
      "source_id": "billing-policy:7",
      "quote": "Refunds are issued to the original payment method
                within 14 business days of approval."
    }
  ],
  "unsupported": ["Anything you asserted that no source backs."]
}

The "quote" must be copied character-for-character from the source.
Do not paraphrase it, shorten it, or fix its punctuation.

The model will still sometimes get this wrong — that is the entire point. Asking for a verbatim quote converts an unfalsifiable claim into a falsifiable one, and falsifiable claims can be checked by a program.

The verifier

import re
from rapidfuzz import fuzz

def norm(s):
    s = s.replace("\u2019", "'").replace("\u201c", '"').replace("\u201d", '"')
    return re.sub(r"\s+", " ", s).strip().lower()

def verify(claim, chunks_by_id):
    chunk = chunks_by_id.get(claim["source_id"])
    if chunk is None:
        return None, "cited a source that was not retrieved"

    hay, needle = norm(chunk.text), norm(claim["quote"])
    at = hay.find(needle)
    if at >= 0:
        return (chunk.start + at, chunk.start + at + len(needle)), "exact"

    # Tolerate whitespace and ligature damage from PDF extraction,
    # but not rewriting. 92 is strict enough to reject paraphrase.
    if fuzz.partial_ratio(needle, hay) >= 92:
        return None, "fuzzy"

    return None, "quote not present in cited source"

Three outcomes, three different meanings. An exact match gives you real character offsets in the original document, which is what a highlight-on-click UI needs. A fuzzy match means the quote is substantively there but the text was mangled somewhere in extraction — display it, and log it, because a rising fuzzy rate is an ingest regression. A miss means the model cited something it did not read.

Track that third rate. It is one of the few genuinely objective quality metrics available in a RAG pipeline: no judge model, no human labelling, no ambiguity. A citation either appears in the cited source or it does not.

What to do with an unverified claim

Deleting the sentence is tempting and usually wrong — you would be silently removing content the user asked for on the basis of a citation failure that may be a formatting artefact. Better options, in order of how much engineering they cost:

• Render it without a link. Verified claims get a clickable marker that jumps to the highlighted span; unverified ones get nothing. The asymmetry is legible without any explanation, and users learn it in about two answers.
• Surface the model’s own unsupported list. Asking the model to name what it asserted without support is cheap and turns out to be reasonably honest, because it is a classification task rather than a generation one.
• Retry once with the failures named. Feed back “this quote was not found in that source” and regenerate. Doubles the cost of the affected answers only.

What matters is that the guarantee you make to the user is one you can actually keep. “Every linked claim was checked against the source text by a program” is a strong, true, defensible statement. “Sources: [1] [2]” is a decoration.

Two design questions remain, and both are worth deciding deliberately. The first is granularity. Citing per sentence is the natural unit for a reader and a poor fit for how answers are written, because a single sentence often fuses a fact from one source with a qualifier from another. Citing per claim, as the schema above does, handles that correctly but requires the renderer to map claims back onto the rendered prose — which is why the schema asks for the claim text verbatim, so the mapping is a substring search rather than a guess.

The second is what happens to answers that are mostly unsupported. There is a temptation to keep generating until everything verifies, and it is worth resisting: an answer where two of five claims fail verification is telling you something true about your retrieval, and suppressing it converts a diagnosable retrieval gap into an invisible one. Emit the verification rate as a metric per request, chart it, and alert on it. It is a leading indicator for index staleness, a bad chunking deploy and a model change, and it costs nothing beyond the verifier you already wrote.

Two failure modes of the scheme itself are worth knowing before you rely on the numbers it produces. Models sometimes cite the correct source but quote a neighbouring sentence — the claim is supported, the quote is not the supporting sentence — which verifies as a miss and looks like a hallucination in your metrics. And a quote that appears verbatim in several chunks (boilerplate, a repeated definition, a standard clause) will verify against whichever one the model happened to name, which may not be the one that actually supports the claim in context. Neither is fatal, but both argue for reading a sample of verification failures by hand before treating the rate as ground truth.