MMMU, DocVQA and ChartQA

MMMU asks whether a model can answer a university exam question that includes a figure. DocVQA asks whether it can read a scanned document and find a value. ChartQA asks whether it can read a number off a plot. These are three unrelated capabilities, and two of the three are scored with metrics that grant partial credit in ways an exact-match reader will misjudge.

Three benchmarks, three demands

A single “multimodal score” that averages these is a category error. A model can be strong at chart reading and poor at scanned documents, because one demands numeric estimation from a clean vector-rendered image and the other demands robust character recognition on a degraded scan. How the pixels reach the model at all is the subject of how a multimodal model sees an image.

MMMU: exam questions with figures

MMMU (Yue and colleagues, 2024) assembles around eleven thousand questions from university exams, quizzes and textbooks across six disciplines — art and design, business, science, health and medicine, humanities and social science, and technology and engineering — split into a small development set, a validation set of several hundred, and a large held-out test set whose answers are not published.

Questions are multiple choice or open-ended, and each carries one or more images interleaved with the text. The construction goal was breadth of image type: a model that has learned to read natural photographs and nothing else should not do well.

MMMU-Pro is the follow-up and its three changes are instructive because each fixes a specific leak.

1. Questions answerable without looking at the image were filtered out — identified by testing text-only models on them. This is the most important fix and it is discussed below.
2. The number of candidate options was raised, dropping the guessing floor and making elimination strategies less productive.
3. A vision-only setting was added, in which the question text itself is rendered into the image rather than supplied separately, so the model must read the question from pixels. This is much closer to a screenshot workflow and much harder.

If you are reading an MMMU figure, find out whether it is the original, the Pro standard setting or the Pro vision-only setting. They are three numbers with one name.

DocVQA and the ANLS metric

DocVQA (Mathew and colleagues, 2021) is tens of thousands of questions over roughly twelve thousand scanned document images taken from a public industry document archive — mid-twentieth-century typed letters, forms, memos, tables and reports, many of them poorly scanned. Answers are short strings that appear in the document.

Grading by exact match would be brutal here, because a single OCR-level error on a long answer would score zero. So DocVQA uses average normalised Levenshtein similarity, and it is worth computing once by hand.

For one question with predicted string p and gold string g:

  NL(p, g) = 1 - editdistance(p, g) / max(len(p), len(g))

  score = NL(p, g)   if NL(p, g) >= tau
        = 0          otherwise            (tau is conventionally 0.5)

ANLS = mean of the per-question scores.

Worked:

  gold "Nestle", predicted "Nestlé"
    edit distance 1, max len 6 -> NL = 1 - 1/6 = 0.833 >= 0.5 -> score 0.833

  gold "November 12, 1987", predicted "November 12 1987"
    edit distance 1, max len 17 -> NL = 1 - 1/17 = 0.941 -> score 0.941

  gold "4,500", predicted "4500"
    edit distance 1, max len 5 -> NL = 1 - 1/5 = 0.800 -> score 0.800

  gold "4,500", predicted "4,600"
    edit distance 1, max len 5 -> NL = 0.800 -> score 0.800  <-- wrong number,
                                                                 80% credit

Read that last case again. ANLS is a string-similarity metric with no notion of semantics, so a numeric answer that is wrong by a hundred scores 0.8 because it differs by one character. On questions whose answers are numbers this is a real distortion, and it is why an ANLS figure is not an accuracy and should never be described as one.

The threshold matters too. Below tau the score drops to zero rather than degrading, so a prediction at NL = 0.49 scores nothing and one at 0.51 scores half. Two implementations using different thresholds produce different numbers on identical predictions.

ChartQA and relaxed accuracy

ChartQA (Masry and colleagues, 2022) pairs around twenty thousand charts from public statistics and data-journalism sources with two kinds of question: several thousand written by humans, and a larger set generated automatically from chart summaries. The human-written half is harder and more interesting; the generated half is templated and inflates the aggregate.

Its metric is relaxed accuracy, and the relaxation is a tolerance band.

Numeric answers:   correct if |predicted - gold| <= 0.05 * |gold|
Non-numeric:       exact string match

Worked, gold = 42.0:
  predicted 42.0  -> exact                          correct
  predicted 43.0  -> |1.0| <= 2.10                  correct
  predicted 44.0  -> |2.0| <= 2.10                  correct
  predicted 44.5  -> |2.5| >  2.10                  wrong

Gold = 1000:
  anything from 950 to 1050 is correct.

Gold = 2:
  only 1.9 to 2.1 is correct — the same 5% is a much
  tighter band on a small number.

The five per cent tolerance is defensible: reading a value off an axis is inherently approximate, and a human doing the same task would also land near rather than on the value. But it has two consequences worth holding.

• It is not comparable to exact match. A relaxed accuracy and an exact accuracy on the same predictions differ, and the gap is largest exactly where chart reading is hardest.
• The band scales with the magnitude. On a chart with large values the tolerance is wide in absolute terms; on small values it is tight. So the metric is easier on some charts than others in a way unrelated to how hard those charts are to read.

The practical failures of chart reading — misreading a legend, attributing a value to the wrong series, missing a log axis — are catalogued in what models get wrong about charts and diagrams.

The text-only shortcut

The most important weakness in multimodal benchmarking, and the easiest to check: some questions can be answered without the image.

It happens in several ways. The question text contains enough information on its own. The options are implausible except one. The question is a well-known exam item a text model has memorised. The image is decorative. In each case a vision-language model can score without doing any vision, and the benchmark records it as multimodal performance.

1. Run the benchmark with the images removed, giving only the question text and options to the same model or to a strong text-only one.
2. Whatever that scores is your blind baseline. It is not the guessing floor — it is usually well above it.
3. The informative quantity is the gap between the with-image score and the blind baseline, not the with-image score itself.

MMMU-Pro’s filtering step is precisely this check applied at construction time, which is why its numbers are lower and more meaningful. Any multimodal result published without a blind baseline is missing the control condition, and it is a cheap control to run.

What these are bad proxies for

• Your documents. DocVQA is a specific archive of mid-century typed and printed documents. Modern invoices, phone photographs of receipts, multi-column PDFs and handwriting are different problems with different failure rates — see document understanding and vision models against a real OCR engine.
• Multi-page reasoning. Nearly all DocVQA items are a single page and a single answer span. Finding a figure on page 40 and reconciling it with a table on page 3 is the actual document task and no widely quoted benchmark covers it well.
• Hallucination on images. Being right on a benchmark question does not tell you how often the model describes something not present. That is measured differently — vision model hallucinations.
• Screenshots and interfaces. A different distribution again: dense text, small fonts, coordinates that matter. See screenshot understanding for UI automation.
• Cost. Images are expensive in tokens and the benchmark does not report it. A high score obtained by processing images at maximum resolution has a very different bill from the same score at a lower one — the arithmetic is here.