Chart and Diagram Reading: What Models Get Wrong

A table states its values. A chart encodes them — as a length, an angle, a position along an axis, a colour — and reading it means inverting that encoding from pixels. Models are good at describing charts and much less good at inverting them, and the gap is predictable from which encoding the chart used.

Why a chart is harder than a table

Recovering a value from a bar chart requires locating the top of the bar, locating two labelled ticks, and interpolating linearly between them. That is a measurement over a patch grid whose spatial precision is bounded by the patch size — a 14-pixel patch on a downscaled chart can easily span several percent of the plot height. There is nothing in the architecture that measures; attention pools, it does not rule off distances.

So the model does what it does everywhere else: produces a plausible continuation. It will round to a pretty number. It will report the value the axis label suggests. It will get the ordering of the bars right — which is genuinely easy from a gist representation — and the magnitudes approximately, and it will state both with the same confidence.

Failure by encoding

Two structural traps sit underneath most of that table. The legend is spatially far from the marks it explains, so binding a colour to a series is a long-range association across a two-dimensional layout. And unlabelled axes make the whole task ill-posed — if the tick labels are illegible at the resolution you sent, no amount of reasoning recovers the scale.

Colour deserves its own warning, because it is the encoding most likely to fail silently. Two series in similar hues, a palette chosen for print, or a legend using small colour swatches at low resolution all make the binding ambiguous — and the model will still answer, picking one. If you generate the charts you feed to a model, you can remove this failure class entirely by labelling series directly on the marks rather than in a legend, which is also better design for human readers. If you do not control the charts, expect series misattribution and test for it specifically: ask which series is highest at a given x, where you know the answer, and see how often the reply names the wrong line.

What the published benchmarks measure

Three are worth knowing, and knowing what each is for stops you reading a headline number as a general capability:

• ChartQA (Masry et al., 2022) — question answering over real-world charts, split into questions that need a lookup and questions that need arithmetic over several extracted values. The split is the useful part: performance on the two halves is not the same skill.
• PlotQA — synthetically generated plots at large scale, which makes it cheap and makes it less representative of the visual mess of published figures.
• CharXiv (Wang et al., 2024) — figures taken from arXiv papers, deliberately including multi-panel and unconventional plots, and explicitly separating descriptive questions (“what is the title of the y-axis”) from reasoning questions. Its stated motivation is precisely that the descriptive half was being saturated while the reasoning half was not, so a single aggregate score hides the thing you care about.

The generalisable lesson from all three: when you evaluate, keep “read a label” and “compute over several read values” in separate buckets, because a model can be excellent at the first and unusable at the second.

Building an eval set for your charts

You have an advantage the benchmark authors did not: for your own dashboards, you have the underlying data. That makes ground truth free and makes a properly grounded eval a couple of hours’ work.

# render charts from known data, so ground truth is exact
import matplotlib.pyplot as plt, random, json

cases = []
for i in range(60):
    vals = [round(random.uniform(0, 100), 1) for _ in range(6)]
    labels = list("ABCDEF")
    fig, ax = plt.subplots(figsize=(4, 3), dpi=110)
    ax.bar(labels, vals)
    if i % 3 == 0:                    # a third with a truncated axis
        ax.set_ylim(min(vals) * 0.9, max(vals) * 1.02)
    fig.savefig(f"chart_{i}.png", bbox_inches="tight")
    plt.close(fig)
    cases.append({"file": f"chart_{i}.png",
                  "truth": dict(zip(labels, vals)),
                  "truncated": i % 3 == 0})

json.dump(cases, open("truth.json", "w"))

Score absolute error per bar rather than exact match, and report the truncated-axis subset separately. If the error on that subset is materially worse, you have learned something specific and actionable about your own dashboards: stop truncating axes on the charts you feed to a model.

What helps

• Send the data, not the picture. The best chart question-answering system is a table. If you control the source, attach the underlying CSV and the chart becomes a formatting detail.
• Ask for the extraction and the answer separately. Have the model list the values it read before computing over them. You can then check the extraction, and the arithmetic step stops being entangled with the perception step.
• Send it larger. Chart reading is resolution-bound in a way that photo description is not. A chart is the canonical case for high detail, or for cropping to the plot area.
• State the axis explicitly in the prompt if you know it. “The y-axis is logarithmic and starts at 10” removes an entire failure class for the cost of one clause.