Task-Specific Metrics: BLEU, ROUGE, and Their Limits
4 min read · updated August 3, 2026
BLEU is from 2002 and ROUGE from 2004. They are routinely dismissed as obsolete by people who then spend forty dollars of judge calls detecting a regression that a character-level F-score would have caught in eight milliseconds.
What BLEU computes
BLEU (Papineni et al., 2002) was built for machine translation. It is a modified n-gram precision: of the n-grams in the candidate, what fraction appear in the reference — with each reference n-gram usable only as many times as it occurs, so repeating a common word does not inflate the score. Precisions for n = 1 to 4 are combined as a geometric mean, and a brevity penalty punishes candidates shorter than the reference, since precision alone rewards saying almost nothing.
Two structural facts follow. Because the mean is geometric, a zero at any n zeroes the whole score — which is why sentence-level BLEU needs smoothing and is unreliable regardless. And because it is a precision against a fixed reference, a correct translation that uses different words scores badly. BLEU was designed and validated as a corpus-level, system-level statistic, and its authors said so.
What ROUGE computes
ROUGE (Lin, 2004) inverts the emphasis for summarisation: recall-oriented overlap with a reference summary. ROUGE-N is n-gram recall; ROUGE-L uses the longest common subsequence, which rewards in-order overlap without requiring contiguity; ROUGE-Lsum applies that per sentence and aggregates, and is the variant most summarisation papers actually report even when they write “ROUGE-L”.
The recall orientation is the point: a summary that omits the main finding should score badly even if every word it does contain is impeccable. The cost is the mirror image of BLEU’s — a summary that copies large spans of the source scores well whether or not it summarised anything, which is why extractive baselines are stubbornly competitive on ROUGE and why ROUGE alone cannot detect a fluent fabrication.
The reporting traps
- Tokenisation changes the number. This is the big one. BLEU is computed over tokens, so a different tokeniser gives a different score for identical translations — which made published BLEU scores non-comparable across papers for years. Post’s 2018 paper, “A Call for Clarity in Reporting BLEU Scores”, is the standard citation and sacreBLEU is the standard fix: it takes detokenised text and applies one canonical tokenisation, and it emits a signature string identifying the exact configuration. If you report BLEU without that signature, you have reported a number nobody can reproduce, including you in six months.
- Reference count changes the number. More references means more ways to be right, so multi-reference scores are systematically higher. Comparing across datasets with different reference counts is meaningless.
- Sentence-level use. Both metrics are noisy on single instances. Use them on a corpus, or use them as a regression tripwire on a fixed corpus, never as a per-item grade.
- Correlation with human judgement is task-dependent. Reiter’s 2018 structured review of BLEU validity concluded that it is reasonable at the system level in machine translation and should not be relied on for evaluating natural language generation more broadly, or for individual outputs. That distinction — system level in MT, not general NLG — is the honest summary of the evidence.
Where classic metrics beat a judge
Not on judgement. On the properties a measurement instrument needs:
| Property | Description |
|---|---|
| deterministic | The same inputs give the same score forever. A judge gives a distribution, and comparing today's judged score with last quarter's requires that nothing about the judge changed — which you cannot guarantee for a hosted model. |
| free and instant | Microseconds per item and no API call. You can score every request in production rather than a 1% sample, which changes what questions you can ask. |
| no bias to correct | n-gram overlap has no position bias, no verbosity preference and no self-preference. It has different problems, but they are stationary, which means you can characterise them once. |
| no vendor | Nothing to deprecate, rate-limit, or reprice. An eval pipeline whose baseline metric depends on a hosted model has a dependency that can invalidate its own history. |
Which makes them the right tool for a specific set of jobs. Constrained transformation — translating into a fixed house style, normalising addresses, converting one structured format to another — has a reference and a narrow space of correct outputs, which is exactly BLEU’s home ground, and chrF (character n-gram F-score) usually behaves better than BLEU on morphologically rich languages and on short strings. High-volume regression detection — running a metric over every production output daily to catch the day the score drops — works because you only need the change to be detectable, not the absolute level to be meaningful. Extraction and structured output should mostly use exact match, normalised edit distance or set F1, and reaching for a judge there is paying money to make a deterministic check stochastic.
The rule of thumb: if there is a reference and the space of correct answers is narrow, use a classic metric. If the space of correct answers is wide, no reference metric can work and you need a judge with a rubric or a reference-free check.
The learned metrics in between
There is a middle tier worth knowing, between n-gram overlap and a general-purpose judge. BERTScore (Zhang et al., 2020) matches contextual embeddings instead of exact tokens, so a correct paraphrase is no longer punished. COMET (Rei et al., 2020) is trained on human quality ratings and has become the standard reference-based metric in machine translation evaluation, largely displacing BLEU for system comparison at the major shared tasks.
These keep most of the operational advantages — a pinned model checkpoint you run locally, so the score is reproducible and free at the margin — while behaving far better on paraphrase. They cost you the ability to explain a score in one sentence, and they have their own training-distribution biases. For anything with a reference, this tier deserves consideration before a general judge does.
However you use them, the operational pattern is the same and it is worth stating plainly: pick the metric, freeze the corpus, record the score every day, and set the alert on the change rather than the level. You never have to defend what a chrF of 0.62 means. You only have to notice the morning it becomes 0.54, and for that job a deterministic metric on a fixed corpus is not merely adequate, it is strictly better than a judge — because a judge that drifts gives you the same alert for a reason that has nothing to do with your system.