Reporting a Benchmark Result Honestly

A benchmark score on its own is not a result. It is a result plus eight pieces of context, and without them nobody — including you in six months — can reproduce it, compare it, or tell whether the difference it shows is real. This page is that list, in a form you can copy into your own repository.

What an unqualified number costs

The costs are specific rather than abstract, and each maps to a field below.

• You cannot reproduce it. Three months later nobody remembers whether it was five-shot, which harness, or which endpoint.
• You cannot compare it. Someone else’s number for the same benchmark was produced with different extraction and different prompting, so the comparison is between configurations, not models.
• You cannot tell whether a difference is real. Without the item count and an interval, a two-point gap and a twenty-point gap look identical on a slide.
• You cannot defend it. The first informed question — which split? which judge? — has no answer, and the number is discarded along with the rest of your work.

This is not pedantry about scientific form. It is the practical difference between a number that survives contact with a reviewer and one that does not.

The eight fields

1. Benchmark identity

Name, version, split, subset, and the number of items actually evaluated. “MMLU” is not an identity. “MMLU test, all 57 subjects, 14,042 items, dataset revision a1b2c3d” is one.

Prevents: comparing a validation-split number to a test-split one; comparing GPQA main to GPQA Diamond; comparing a subsampled run to a full one; comparing MMLU to MMLU-Pro; comparing a BEIR average over thirteen datasets to one over eighteen.

2. System identity

The exact model identifier including any date or version suffix, the provider or endpoint that served it, the weight precision or quantisation, and for a self-hosted run the inference engine and its version.

Prevents: attributing a number to a model family when an endpoint alias silently moved to different weights; comparing a full-precision local run to a quantised hosted one; comparing two providers serving the same open weights as though they were the same system.

3. Harness identity

The evaluation tool, its version or commit, and the task identifier within it. If you wrote your own, link the code at a commit.

Prevents: the 2023 MMLU episode described in running a public benchmark yourself — three implementations of one benchmark producing three different scores for the same weights.

4. Prompt configuration

The full prompt template or a link to it, the few-shot count, whether exemplars were fixed or sampled and with what seed, the system prompt if any, and whether the model’s chat template was applied.

Prevents: comparing zero-shot to five-shot; comparing a direct-answer BBH run to a chain-of-thought one; comparing an instruct model evaluated as a completion model to one evaluated properly.

5. Decoding configuration

Temperature, top-p, maximum tokens, stop sequences, seed, and the number of samples per item.

Prevents: a pass@k number with no temperature, which is undefined; a maths score depressed by truncation being read as a capability result; a comparison between a greedy run and a sampled one.

6. Scoring rule

The metric with all its parameters — pass@k with its n and k, nDCG@10, ANLS with its threshold, relaxed accuracy with its tolerance, WER with its normaliser, attack success rate with its judge. Plus the answer-extraction rule and, critically, what happened to outputs the extractor could not parse.

Prevents: comparing macro and micro averages; comparing a substring-matched attack success rate to a rubric-graded one; comparing an ANLS to an exact-match accuracy; silently scoring unparseable outputs as wrong in one run and excluding them in another, which alone can move a score several points.

7. Uncertainty

The item count, an interval over items (standard error or a bootstrap confidence interval), the number of repeats, and the spread across repeats if you sampled.

Prevents: the most common error in the field — treating a four-point gap on a 198-item set as a ranking. The arithmetic is on the GPQA page and the tests are in statistical significance with non-deterministic models.

8. Cost, compute and date

Total input and output tokens including reasoning tokens, money spent, wall-clock time, hardware if self-hosted, and the date of the run.

Prevents: comparing a score obtained with heavy test-time compute to one obtained in a single pass; presenting a result from a stale price list; and losing the date, which is what makes every other field unfalsifiable later. The composite metrics are on cost-aware leaderboards.

A machine-readable manifest

Write this next to the raw outputs, one per run, generated by the harness rather than by hand. If it is not generated it will not be accurate.

{
  "schema": "bench-report/1",

  "benchmark": {
    "name": "gpqa",
    "version": "2023-11",
    "split": "diamond",
    "subset": null,
    "n_items": 198,
    "source_revision": "hf:Idavidrein/gpqa@<commit-sha>"
  },

  "system": {
    "model_id": "vendor/model-name-2026-05-01",
    "endpoint": "https://api.example.com/v1",
    "served_by": "vendor-first-party",
    "precision": "unknown (hosted)",
    "engine": null
  },

  "harness": {
    "name": "lm-evaluation-harness",
    "version": "v0.x.y",
    "commit": "<commit-sha>",
    "task_id": "gpqa_diamond_zeroshot",
    "config_file": "configs/gpqa-diamond.yaml"
  },

  "prompt": {
    "template_path": "prompts/mcq-v3.txt",
    "n_fewshot": 0,
    "fewshot_source": null,
    "fewshot_seed": null,
    "system_prompt": null,
    "chat_template_applied": true
  },

  "decoding": {
    "temperature": 0.0,
    "top_p": 1.0,
    "max_tokens": 8192,
    "stop": [],
    "seed": 1234,
    "n_samples_per_item": 5
  },

  "scoring": {
    "metric": "accuracy",
    "metric_params": {},
    "extraction_rule": "last match of /(?:answer[: ]*)?\\(?([A-D])\\)?/i",
    "unparseable_policy": "scored_incorrect",
    "n_unparseable": 3,
    "judge": null
  },

  "uncertainty": {
    "n_items": 198,
    "n_repeats": 5,
    "point_estimate": 0.71,
    "ci_method": "bootstrap over items, 10000 resamples",
    "ci_95": [0.645, 0.771],
    "across_repeat_sd": 0.018
  },

  "cost": {
    "input_tokens": 241000,
    "output_tokens": 3980000,
    "reasoning_tokens_included": true,
    "usd_total": 61.4,
    "wall_clock_seconds": 5240,
    "hardware": null,
    "run_date": "2026-08-04"
  },

  "declarations": {
    "contamination_check": "none performed; hosted model, training data not disclosed",
    "conflict_of_interest": "none",
    "runs_discarded": 0,
    "raw_outputs_path": "runs/2026-08-04-gpqa-diamond/outputs.jsonl"
  }
}

Two fields in there deserve pointing at. n_unparseable and unparseable_policy are the pair that most often explains a surprising result, and almost no published report contains either. runs_discarded is the honesty field: if you ran the evaluation four times and are reporting the best, the number four belongs in the file.

The same thing as a paragraph

For a blog post or a slide, the manifest compresses to a footnote. This is the template; it is dense on purpose, because a reader who does not care skips it and a reader who does care needs all of it.

GPQA Diamond (198 items, split "diamond", dataset rev a1b2c3d),
evaluated with lm-evaluation-harness v0.x.y (commit deadbee), task
"gpqa_diamond_zeroshot". Model "vendor/model-name-2026-05-01" served by
the vendor's own API; precision not disclosed. Zero-shot, chat template
applied, no system prompt. Greedy decoding (temperature 0.0, top_p 1.0),
max_tokens 8192, seed 1234, 5 samples per item. Scored as accuracy;
answers extracted with the last A-D match in the reply; 3 outputs were
unparseable and scored incorrect. Result 0.71, 95% bootstrap CI over
items [0.645, 0.771], SD across the 5 repeats 0.018. Run on 2026-08-04:
241k input tokens, 3.98M output tokens including reasoning tokens,
$61.40, 87 minutes. No contamination check was possible (hosted model,
undisclosed training data). Raw outputs published at <link>.

That is eleven lines and it makes the result reproducible, comparable and checkable. Compare it with the usual form — a model name, a benchmark name and a percentage — which supports none of those three.

Two declarations that are not fields

Beyond the eight, two statements belong on any published result. They are not measurements, which is why they sit apart, and they are the two that separate an honest report from a marketing one.

Contamination

State what you did to check whether the test items were in the training data, or state plainly that you could not. For a hosted model whose training corpus is undisclosed, the honest declaration is “none performed and none possible” — which is informative, because it tells the reader the score has an unquantified upper-bound risk. For your own fine-tune, run an overlap check and say so; the methods are in benchmark contamination and decontaminating a dataset.

Interest

If you sell one of the systems being compared, say so in the same place as the number rather than in a disclosure page. If the evaluation was designed after seeing preliminary results, say that too. And if you ran the evaluation several times with different configurations and are reporting one, report how many you ran — selecting the best configuration per model is a legitimate methodology and a silently-selected best is not. The patterns to avoid are catalogued in how to read a launch post.

Round to the precision you earned

A small thing that signals whether a report was written by someone who understands their own number: the decimal places should be justified by the interval.

Result 0.7134, 95% CI [0.645, 0.771]  -> half-width ≈ 6.3 points

  Reporting "71.34%"  claims precision to a hundredth of a point
                      when the interval is ±6 points wide.
  Report "71%" or "0.71 [0.65, 0.77]".

Rule of thumb: report to one decimal place fewer than the interval's
leading digit. A ±6-point interval justifies whole numbers. A
±0.6-point interval justifies one decimal place.

The corollary matters more than the rule: if your presentation needs
two decimal places to show a difference, there is no difference.

The same logic applies to leaderboard tables. A column of numbers to two decimal places, with no interval anywhere on the page, is a presentation choice that manufactures apparent separation between systems that have not been separated — a habit noted in evaluating by vibes in its more general form.

Five questions for somebody else’s number

The reverse of the standard. When you meet a benchmark claim, these five questions resolve most of them, in the order that eliminates fastest.

1. How many items? If the answer is a few hundred, the interval is several points wide and any comparison inside that band is unsupported. This one question disposes of most published comparisons.
2. Which split and which version? Test or validation, full or subset, original or the harder successor. If the claim does not say, it is not comparable to any other claim.
3. What was the prompting and extraction? Shot count, template, and how the answer was pulled out of the reply. The largest source of unexplained disagreement between two numbers for one model.
4. Who ran it, and did they choose the configuration after seeing results? A vendor reporting its own number under a configuration tuned for its own model is not fraud, and it is also not a comparison. Ask whether the same tuning effort went into the baselines.
5. What did it cost? A score obtained with many samples and heavy test-time compute is a different claim from the same score in one pass. If the cost column is missing, assume the answer is unflattering — the general bullshit detector has more of this shape.

None of these require expertise in the benchmark. They require knowing that a number is a claim with a method behind it, and that a claim whose method is absent is not one you have to accept. Adopting the eight fields for your own results is what earns you the standing to ask — and it takes an afternoon to add to a harness, once.