Evaluating Summarisation Without Reference Summaries

Almost nobody evaluating summarisation in production has reference summaries, and writing them is expensive and arbitrary — two competent people summarise the same document differently, and neither is wrong. The way out is to stop comparing to a reference and start checking properties against the source.

Why references are the wrong tool here

A reference summary encodes one person’s choice of what mattered. Scoring overlap against it, as ROUGE does, penalises a summary that made a different but equally defensible choice, and — worse for production use — rewards a summary that copies the reference’s phrasing while inventing a fact. The metric is blind to the failure mode you actually care about.

The SummEval study (Fabbri et al., 2021) is the standard reference for how weakly the automatic metrics of that era tracked human judgements of factual consistency on news summarisation, and it is why the field’s attention moved to reference-free faithfulness checks.

Two axes, measured separately

Almost every real summarisation complaint is one of two things, and collapsing them into one score makes both unfixable.

• Faithfulness — is everything in the summary supported by the source? Failures here are fabrications, subtle number changes, dropped hedges and flipped negations. This axis is checked from the summary towards the source.
• Coverage — is everything important from the source in the summary? Failures here are omissions, and they are invisible to any check that only reads the summary. This axis is checked from the source towards the summary.

The direction of the check is the whole design. A perfectly faithful summary can be one useless sentence; a perfectly covering one can be the document itself. You need both numbers and they trade off, which is a fact about summarisation rather than a flaw in the evaluation.

Faithfulness by claim decomposition

The pipeline that has held up: split the summary into atomic claims, check each one against the source independently, and report the proportion supported. Checking claims one at a time rather than judging the whole summary is what makes the result stable and attributable — a judge asked “is this summary faithful” gives you a vibe, a judge asked “is this single sentence entailed by this passage” gives you an answer.

DECOMPOSE = """Split the summary below into atomic factual claims.
An atomic claim states exactly one fact and is understandable on its own
(resolve every pronoun). Ignore opinions and transitions.
Return one claim per line, no numbering.

SUMMARY:
{summary}"""

VERIFY = """SOURCE PASSAGE:
{source}

CLAIM:
{claim}

Is the claim fully supported by the source passage? Consider numbers,
dates, quantifiers ("all", "some"), negations, and attributed versus
asserted statements. A claim that is plausible but not stated is NOT
supported.

Answer with one line: SUPPORTED, CONTRADICTED, or NOT_STATED."""

# faithfulness = supported / total_claims
# Report CONTRADICTED and NOT_STATED separately -- a contradiction is a
# different bug from an unsupported addition, and they get fixed
# differently.

This is the shape used by the reference-free faithfulness literature: QAGS (Wang et al., 2020) generates questions from the summary and compares answers derived from summary and source; FactCC (Kryscinski et al., 2020) trains a classifier on synthetically corrupted summaries; SummaC (Laban et al., 2022) applies sentence-level natural language inference between source and summary sentences and aggregates the matrix, which is a good option when you want a local model rather than API calls. Any of these can replace the VERIFY step; the decomposition step is the part that carries most of the benefit.

For long sources, retrieve before verifying — pass only the passages most relevant to the claim rather than the whole document. Verification accuracy degrades when the evidence is buried, and this is also what makes the cost manageable.

The cheap checks to run first

Before any model call, three programmatic checks catch a disproportionate share of real faithfulness failures, and they are free and deterministic:

• Every number in the summary must appear in the source. Normalise first — strip thousands separators, unify decimal marks, expand “3.2 million” to digits, handle percentages — then require an exact match. Summarisers transpose digits, round silently, and confuse two figures from adjacent sentences, and this check catches all three.
• Every named entity and date in the summary must appear in the source. Same principle. An invented name is a serious failure and a trivial one to detect.
• Hedges must survive. Where the source says “may”, “is expected to”, “alleged” or “according to X”, and the summary states the same proposition without the hedge, that is a faithfulness failure with real consequences in regulated domains. Flag summary sentences that assert something whose source sentence contained a hedging term.

Run these on every summary in production. They are cheap enough to be a runtime guard rather than an offline metric, and a guard that refuses to ship a summary containing a number not present in the source prevents a whole class of incident.

Coverage from the source, not the summary

Coverage requires a notion of what was important, which has to come from the source. Three approaches, in increasing cost:

• Structural salience. For documents with known shape, the important units are known in advance: a support thread has an issue, a resolution and a next step; an earnings release has a period, a headline figure and a guidance statement. Enumerate the slots, check each is filled. This is by far the most reliable approach and it is available more often than people assume.
• Question generation from the source. Generate questions a reader should be able to answer after reading, from the source only, then attempt to answer each from the summary alone. Coverage is the proportion answerable and correct. The critical detail is that the questions are generated before the summary is seen, so the summary cannot influence what counts as important.
• Claim recall. Decompose the source into claims, weight them for importance, and check which appear in the summary. Thorough and expensive; usually only worth it as a periodic deep evaluation rather than a routine metric.

Report the two axes side by side and never average them. A pair like “faithfulness 0.98, coverage 0.61” tells you the summary is careful and too short, which is an actionable prompt change. The average of those two numbers tells you nothing at all.