Summarisation Prompts That Keep the Numbers Right

Summarisation has exactly one interesting failure mode: the prose is right and a number is wrong. The fix is to stop asking one call to both select and rephrase. Extract the facts verbatim in one call, compress against that extract in a second, and check the numbers deterministically in between.

The failure this prevents

A single-call summariser is doing two jobs at once: deciding what matters, and writing fluent sentences about it. Fluency is the job it is best at, so when a figure does not fit the sentence, the sentence usually wins. What comes out is not a hallucination in the dramatic sense — it is a rounding, an annualisation, a merged period, a percent that was a percentage point.

source:  "Revenue was 4.2m EUR in Q3, up from 3.9m in Q2. Margin fell
          120 basis points to 31.4%."

one-call summary:
  "Revenue grew to roughly 4 million euros, with margins down 1.2% to 31%."
                        ^ rounded      ^ percent vs basis points   ^ rounded

Every one of those is a small, plausible edit and the paragraph reads well. That is the problem: nothing about the output signals that it drifted. Splitting the call removes the pressure — the extraction call is not writing prose, so it has no reason to reshape a number, and the compression call is copying from a list rather than recalling from a document.

Call one: extract, do not summarise

You will be given a document. Do not summarise it. Extract only.

Return JSON:
{
  "claims":   [{"text": "<the claim in the document's own words>", "quote": "<verbatim substring>"}],
  "figures":  [{"value": "<exactly as printed>", "unit": "<or null>",
                "subject": "<what it measures, or null>",
                "period": "<or null>", "quote": "<verbatim substring>"}],
  "dates":    [{"date": "<exactly as printed>", "event": "...", "quote": "<verbatim substring>"}],
  "entities": [{"name": "...", "role": "..."}]
}

Rules:
- Every item carries a "quote" that is an exact substring of the document,
  character for character. If you cannot produce the exact substring, drop
  the item rather than paraphrasing it.
- Copy each figure as printed: the sign, the currency symbol, the thousands
  separators, the number of decimal places, the unit word. Do not convert,
  do not round, do not annualise, do not turn basis points into percent.
- If a figure has no subject or no period stated in the document, leave the
  field null. Do not infer it from a neighbouring sentence.
- Extract nothing that is not in the document. No totals you computed, no
  implications, no context you know from elsewhere.
- Order the arrays by first appearance in the document.

<document>
{{document}}
</document>

Why each rule is there

• The quote requirement is the whole mechanism. It converts “be faithful” — which nothing can check — into “produce a substring”, which a two-line function checks. An item that cannot produce its span gets dropped by the model or caught by your verifier.
• “Do not summarise” up front. Without it, the model produces one long claim per section. The instruction is at the top rather than buried in the rules because it contradicts what the model expects the task to be.
• Null rather than inference. A figure with an invented period is worse than one with no period, because the second one is visibly incomplete and the first one is quietly wrong.
• Order by appearance. Makes two runs of the same document diffable, which is how you notice the extraction changed when nothing else did.

Call two: compress against the extract

Write a summary of at most {{words}} words.

You may state only facts that appear in <extract>. <document> is provided so
you can match its phrasing and ordering; it is not a source of facts. If
something is in <document> and not in <extract>, it does not go in the summary.

Every figure and every date you write must appear character for character in
<extract>. If a number does not fit the sentence you want to write, change the
sentence, not the number.

Order: the outcome first; then the two or three things that caused it; then the
one caveat that would change a reader's decision. No background, no restatement
of the question, no closing sentence about implications.

If <extract> does not support a summary of the requested length, write a
shorter one and say why in "shortfall".

Return JSON: {"summary": "...", "used": ["<quote from extract>", ...],
              "shortfall": "<or null>"}

<extract>
{{extract_json}}
</extract>

<document>
{{document}}
</document>

The used array is not decoration. It gives you a per-sentence trail from summary back to source without a second grading call, and it makes the check in the next section stricter: you can assert that every quote in used is genuinely in the extract, which catches the case where the model has quietly gone back to the document.

Including document in the second call is a deliberate trade-off. It roughly doubles the input tokens and it improves the phrasing, because an extract alone produces a list read aloud. The rules are written to make it clear which role each block plays. If cost matters more than register, drop it — the summary gets flatter and no less accurate.

The numeric fidelity check

This runs between the two calls, or after the second, and it needs no model. Every numeric token in the summary must appear in the extract.

import re

NUM = re.compile(r"[-+]?[£$€]?\s?\d[\d,.]*\s*(?:%|bn|m|k|bps)?", re.I)
SEP = re.compile(r"[\s,  ]")   # space, comma, nbsp, narrow nbsp

def norm(s: str) -> str:
    "Ignore separators and case; keep digits, sign, decimal point and unit."
    return SEP.sub("", s.lower()).rstrip(".")

def numeric_fidelity(summary: str, extract_text: str) -> list[str]:
    """Return every numeric token in the summary that is not in the extract."""
    source = {norm(m.group()) for m in NUM.finditer(extract_text)}
    return [m.group() for m in NUM.finditer(summary)
            if norm(m.group()) not in source]

# extract_text is the concatenation of every "quote" in the extract JSON,
# not the extract's own prose fields — quotes are the part guaranteed verbatim.

Two things about this check are worth stating plainly. It is conservative in one direction: a summary that says “about four million” where the source says 4.2m passes the digit test and is still a rounding, so the compression prompt bans the rounding explicitly rather than relying on the checker. And it produces false positives on legitimately reformatted numbers, which is the correct behaviour here — reformatting is the thing you are trying to prevent.

Wire it as a gate, not a report. On failure, retry the compression call once with the failing tokens appended as These numbers are not in the extract: [...]. Rewrite without them. A single retry fixes most of them; a second failure should fall back to returning the extract itself as a bulleted list, which is uglier and correct.

The recipe on one paragraph

The whole loop on a short source, so the shape of each stage is visible. The document is the two sentences from the first section.

-- call one, extract --------------------------------------------------------
{"claims": [
   {"text": "Revenue rose quarter on quarter",
    "quote": "Revenue was 4.2m EUR in Q3, up from 3.9m in Q2."},
   {"text": "Margin fell",
    "quote": "Margin fell 120 basis points to 31.4%."}],
 "figures": [
   {"value": "4.2m", "unit": "EUR", "subject": "Revenue", "period": "Q3",
    "quote": "Revenue was 4.2m EUR in Q3"},
   {"value": "3.9m", "unit": "EUR", "subject": "Revenue", "period": "Q2",
    "quote": "up from 3.9m in Q2"},
   {"value": "120", "unit": "basis points", "subject": "Margin", "period": null,
    "quote": "Margin fell 120 basis points"},
   {"value": "31.4%", "unit": null, "subject": "Margin", "period": null,
    "quote": "to 31.4%"}],
 "dates": [], "entities": []}

-- call two, compress to 20 words -------------------------------------------
{"summary": "Revenue was 4.2m EUR in Q3, up from 3.9m in Q2, while margin
             fell 120 basis points to 31.4%.",
 "used": ["Revenue was 4.2m EUR in Q3", "up from 3.9m in Q2",
          "Margin fell 120 basis points", "to 31.4%"],
 "shortfall": null}

-- the check ----------------------------------------------------------------
numeric_fidelity(summary, quotes) -> []          # pass

-- what the single-call version produced, checked the same way ---------------
summary: "Revenue grew to roughly 4 million euros, with margins down 1.2% to 31%."
numeric_fidelity(summary, quotes) -> ['4', '1.2%', '31%']

Three things are worth noticing. The period on the margin figures is null rather than “Q3”, because the source does not say — the model is not allowed to carry it over from the neighbouring sentence, and that restraint is what makes the field trustworthy elsewhere. The two-call summary is longer and duller than the one-call version, which is the trade being made. And the checker catches all three drifted numbers in the single-call output without knowing anything about finance.

The failing case is also instructive: 4 is flagged because “roughly 4 million” is not 4.2m, which is exactly the rounding the compression prompt bans. The checker cannot tell you the rounding is acceptable in some contexts; that is a policy decision, and the right place for it is your retry rule rather than the regex.

What the second call costs

Two calls is more expensive than one, and by roughly how much is calculable rather than guessable. Take a 3,000-token document, a 150-word summary and an extract that comes to about 600 tokens:

one call    in 3,000 + 250 (prompt)        out ~200        = 3,250 in / 200 out
two calls   in 3,000 + 300  out ~600        (extract)
            in 3,600 + 250  out ~220        (compress, document included)
                                            = 6,900 in / 820 out

ratio       input  x2.1     output x4.1

Whether that is worth paying depends entirely on what a wrong number costs you. For an internal reading list, it is not. For anything a person acts on — a financial digest, a clinical note summary, a diligence pack — it is, and the alternative is a human checking every figure by hand. Dropping document from the second call takes the input ratio to about 1.3×.

When it stops working

• Quotes stop being substrings. Run the substring assertion on the extract itself, not just on the summary. When the failure rate on that assertion climbs, the model has started normalising whitespace or quotation marks, and the fix is usually to normalise both sides before comparing rather than to change the prompt.
• The shortfall field starts firing on ordinary documents. That means the extraction call is returning too little — check its output length against the document length, because a truncated extract is invisible downstream.
• The fidelity checker goes quiet. Zero failures for weeks is not a triumph; it usually means the summaries stopped containing numbers. Count numeric tokens per summary as well as failures.

For grading the summaries themselves rather than their arithmetic — coverage, faithfulness, whether the right thing was chosen — the reference-free approach in evaluating summarisation without reference summaries is the companion to this, and the rubric that makes such a grade reproducible is in this cluster.