HELM and Multi-Metric Evaluation
10 min read · updated August 4, 2026
HELM’s central claim is that reporting one accuracy number per model is a choice, not a necessity, and a bad one. It evaluates every model on every core scenario and reports seven categories of metric for each: accuracy, calibration, robustness, fairness, bias, toxicity and efficiency. Six of those are things nobody else measures and everybody claims to care about.
What HELM is
The Holistic Evaluation of Language Models was published by Liang and colleagues at Stanford’s Center for Research on Foundation Models in 2022. It is less a dataset than a protocol plus an implementation: a taxonomy of scenarios, a set of metrics, a standardised way of prompting, and a commitment to publishing every raw prompt and completion so results can be inspected rather than trusted.
That last commitment is underrated. Almost no other benchmark publishes the actual model outputs. When a HELM number looks strange you can go and read what the model said, which is how you distinguish a capability failure from a formatting one.
HELM has since become a family rather than a single run: lighter variants for cost, an instruction-following variant, domain versions for medicine and other fields, safety-focused versions, and a text-to-image sibling. The protocol is the constant; the scenario list is not.
Scenarios times metrics, densely filled
The structural idea is a matrix. Rows are scenarios — a task, a domain, and a user population, such as question answering on consumer-facing medical text. Columns are metrics. The complaint HELM was written against is that the literature filled that matrix sparsely and unevenly: each paper picked its own scenarios and reported accuracy on them, so no two models were measured the same way.
HELM’s answer is density. Pick a core set of scenarios, run every model on all of them under identical prompting, and compute all seven metric categories on every cell. The result is comparable in a way that a table assembled from different papers is not — a point made more generally in reading a benchmark paper critically.
The seven metrics, as procedures
| Metric | Description |
|---|---|
| Accuracy | Whatever the scenario's own correctness measure is: exact match, F1, ROUGE, or a task-specific score. This is the only one most other benchmarks report. |
| Calibration | Whether the model's stated or implied confidence matches how often it is right. Computed from the model's probability on its chosen answer, binned and compared to observed accuracy in each bin. Worked below. |
| Robustness | Perturb the input in ways that should not change the answer — typos, casing, spacing, contractions, synonym swaps — and measure how much accuracy drops. Reported as worst-case accuracy over the perturbations rather than average, because worst-case is what a user hits. |
| Fairness | Two procedures. Counterfactual: swap demographic terms in the input (names, pronouns, dialect) and check whether the answer changes when it should not. Performance disparity: compare accuracy across subgroups where the data supports it. |
| Bias | Measured in generated text rather than in correctness: the demographic composition of entities the model produces, and the strength of stereotypical associations between demographic and descriptive words in its output. A distributional statistic over completions, not a per-item score. |
| Toxicity | The fraction of completions a toxicity classifier flags, over the scenario's generations. Inherits the classifier's own error profile, which is documented to misfire on reclaimed language and on dialects. |
| Efficiency | Two figures. Idealised inference runtime, normalised to a fixed reference hardware configuration so models served on different machines are comparable, and denoted (actually observed) runtime. Training compute and energy are reported where known. |
The robustness definition deserves a second look, because it is the one most directly useful to somebody shipping a product. Reporting worst-case accuracy across perturbations answers a question the average hides: if a user types with a typo, or in lower case, or pastes text with odd whitespace, how bad does it get? That is a production question and almost nothing else measures it.
Calibration, worked
Calibration is the metric people find hardest to picture, so here is the arithmetic. The usual summary statistic is expected calibration error: bucket predictions by stated confidence, compare each bucket’s average confidence to its actual accuracy, and average the gaps weighted by bucket size.
Take 1000 predictions, bucketed by the model's confidence in its answer.
bucket n mean confidence actual accuracy |gap|
-------------------------------------------------------------
0.5 - 0.6 120 0.55 0.52 0.03
0.6 - 0.7 180 0.65 0.60 0.05
0.7 - 0.8 250 0.75 0.66 0.09
0.8 - 0.9 300 0.85 0.71 0.14
0.9 - 1.0 150 0.95 0.79 0.16
ECE = SUM over buckets of (n/N) * |gap|
= (120/1000)*0.03 + (180/1000)*0.05 + (250/1000)*0.09
+ (300/1000)*0.14 + (150/1000)*0.16
= 0.0036 + 0.0090 + 0.0225 + 0.0420 + 0.0240
= 0.1011
An ECE of 0.10 with every gap in the same direction is the classic
overconfidence signature: when this model says 90% it is right 79% of
the time. Accuracy here is 0.66 overall — the ECE is a separate fact
about it and the two can move independently.Why this matters operationally: if you route on confidence — escalate to a human below a threshold, accept above it — you are relying on calibration and not on accuracy. A well-calibrated 70-per-cent-accurate model is more useful for that design than a badly-calibrated 80-per-cent one, because you can act on its uncertainty. Abstention as a design goal is covered in making a model say I do not know, and the underlying signal in logprobs.
Why the other six get ignored
HELM has been available for years and the industry still quotes single accuracy numbers. The reasons are structural rather than ideological, and worth naming.
- Cost. Seven metric categories over dozens of scenarios means many multiples of the generations an accuracy-only run needs. Robustness alone multiplies the item count by the number of perturbations.
- No headline. A launch post needs one number that goes up. Seven numbers, three of which are lower-is-better, do not make a chart. This is the same incentive that produces the patterns in benchmark marketing.
- Some metrics are uncomfortable. Bias and toxicity numbers are reputational liabilities to publish and easy to omit. Efficiency invites cost comparisons vendors would rather not invite.
- Metric dependencies age. Toxicity scoring depends on an external classifier whose availability and behaviour change over time, which makes old runs hard to reproduce exactly.
How to use HELM if you are choosing a model
- Find the scenarios closest to your task rather than reading the aggregate. HELM is designed to be sliced.
- Read the robustness column before the accuracy column if your input comes from users rather than from a clean pipeline. Worst-case accuracy is closer to what you will experience.
- Read calibration if any part of your design branches on confidence. If nothing branches on confidence, skip it.
- Read toxicity and bias if you generate user-visible free text, and treat them as screening rather than clearance — safety benchmarks covers what these suites can and cannot establish.
- Then go and open the published prompts and completions for one scenario you care about and read twenty of them. Fifteen minutes of that will tell you more than the table, and HELM is one of the very few benchmarks that makes it possible.
The lasting contribution of HELM is not its leaderboard, which is always somewhat behind the frontier because running it is expensive. It is the demonstration that the other six axes are measurable, so “we only report accuracy because the rest is not quantifiable” stopped being a defensible position in 2022.