Running a Public Benchmark Yourself

Run a public benchmark against a model whose score you already know and your number will differ. That is normal, and it is not usually a bug. It is the accumulated effect of nine configuration choices, each of which the harness makes on your behalf and none of which is in the headline. This page names them and gives you a way to measure how much each one is worth on your own setup.

Why your number will not match the published one

The most useful documented instance of this is public and worth reading in full: in June 2023 HuggingFace published an explanation of why the Open LLM Leaderboard’s MMLU numbers disagreed with the figures published for the same models. The cause was not a bug in anyone’s model. It was that three independent implementations of “MMLU” — the original paper’s code, the evaluation harness the leaderboard used, and HELM — made different choices about how to present the question and how to extract the answer, and the choices produced substantially different scores for the same weights.

That episode is the single best argument for the discipline this page describes. Three careful teams, one benchmark name, three numbers. The fix was not to declare one correct but to specify which implementation each number came from — which is the reporting standard in reporting a benchmark result honestly.

The harnesses, and what each is for

The nine settings that move a score

1. Task version. Benchmarks get corrected, respun and re-templated. “MMLU” in a harness is a specific task definition with a version, and two versions are two benchmarks. Record the task id and the harness commit.
2. Prompt template. How the question, the options and the instruction are laid out. Whether options are prefixed “A.” or “(A)”. Whether an instruction line appears at all. Small changes, real differences, especially on models sensitive to formatting.
3. Few-shot count and selection. Zero-shot and five-shot are different measurements. So are five-shot with a fixed exemplar set and five-shot with exemplars sampled per item — and if sampled, the seed is part of the configuration. The mechanism is in few-shot prompting.
4. Chat template and system prompt. Whether the model’s own chat template was applied, and what system prompt sat above the task. An instruct-tuned model evaluated as a raw completion model is being used outside its training distribution — see chat templates.
5. Answer extraction. Log-probability ranking over the options against generating text and parsing it, and if parsing, the exact rule. Discussed at length on the MMLU page because it is where that benchmark’s implementations diverge most.
6. Normalisation. For multiple choice, whether option log-probabilities are normalised by token length. For generation, what text normalisation runs before comparison. Both are scoring decisions dressed as implementation details.
7. Decoding parameters. Temperature, top-p, and the number of samples. Greedy decoding and temperature sampling produce different scores and different variances, and for pass@k the temperature is part of the metric.
8. Token limit and stop sequences. A limit that truncates a reasoning model before it reaches its answer scores that item zero. On maths and reasoning sets this is one of the largest and most avoidable effects, and it looks like a capability result.
9. Precision, quantisation and backend. The same weights served at a lower precision, on a different inference engine, with a different batching policy, are a different system for scoring purposes — the quality effects are catalogued in quantisation for inference.

Why a seed is not enough

Setting temperature to zero and fixing a seed feels like it should make a run reproducible. It usually does not, for reasons below the harness.

• Floating-point reductions are not associative. The order in which partial sums are combined on a GPU depends on how work was scheduled, so identical inputs can produce bit-different logits. Where two options are nearly tied, a bit-level difference flips the answer.
• Batch composition changes the arithmetic. Serving stacks batch requests dynamically, and a sequence computed in a batch of 4 can produce slightly different numbers from the same sequence in a batch of 32. You do not control what else was in flight.
• Temperature zero is argmax, not determinism. It removes the sampler’s randomness and nothing else. Everything above still applies.
• A hosted model is a moving target. An endpoint name can point at different weights over time, and serving configuration changes without announcement. Reproducibility against an API is aspirational; against local weights with a pinned engine version it is achievable.

The practical response is not to chase determinism but to measure variance: run the evaluation three or five times and report the spread. A run-to-run spread is a fact about your measurement that a single run cannot give you, and on small benchmarks it is frequently larger than the differences being argued about.

The shape of a harness run

Every harness invocation has the same anatomy regardless of which tool you use. Written out as a shape rather than as a literal command:

<harness-entrypoint> \
    --model            <backend: local weights, or an OpenAI-compatible endpoint> \
    --model-args       <precision, device, batch size, base URL, model name> \
    --tasks            <task id, at a pinned version> \
    --num-fewshot      <integer> \
    --batch-size       <integer> \
    --limit            <optional: subsample for a smoke run> \
    --seed             <integer> \
    --output-path      <a directory that will hold the per-item log>

The flag names above are placeholders. Every one of these tools spells
them differently and renames them between releases — read the harness's
own docs for the real ones, and paste the exact command you ran into
your results file.

Two rules that hold across all of them:

  1. Do a --limit run of 20 items first. It catches template, parsing
     and credential problems in 30 seconds instead of 3 hours.
  2. Keep the per-item output log. Without it you cannot tell a wrong
     answer from an unparsed one, and that distinction is most of what
     you learn from a first run.

A harness that measures your own delta

The claim worth making on your own results is not “this model scores N” but “changing this setting moved my score by M”. The script below does that: it runs the same items under two named configurations against any OpenAI-compatible endpoint, reports both accuracies with bootstrap confidence intervals, and reports the paired difference — which is the number you actually want, because the same items are used for both arms.

It has no dependencies beyond the Python standard library, so there is nothing to install and no API surface to get wrong.

# ab_eval.py — Python 3.11, standard library only.
#
# Runs one item set under two configurations and reports the PAIRED
# difference with a bootstrap interval. Works against any
# OpenAI-compatible /v1/chat/completions endpoint.
#
#   export EVAL_BASE_URL=https://your-endpoint/v1
#   export EVAL_API_KEY=...
#   python ab_eval.py items.jsonl
#
# items.jsonl: one JSON object per line, e.g.
#   {"id": "q1", "question": "...", "options": ["...","...","...","..."], "answer": "B"}

import json, os, random, re, sys, urllib.request

BASE  = os.environ["EVAL_BASE_URL"].rstrip("/")
KEY   = os.environ["EVAL_API_KEY"]
MODEL = os.environ.get("EVAL_MODEL", "gpt-4o-mini")

# The two arms. Change ONE thing between them, or you will not know
# which change moved the score.
ARMS = {
    "A_letter_only": {
        "instruction": "Answer with the letter of the correct option and nothing else.",
        "temperature": 0.0,
        "max_tokens": 4,
    },
    "B_reason_then_answer": {
        "instruction": "Think step by step, then end your reply with 'Answer: X'.",
        "temperature": 0.0,
        "max_tokens": 512,
    },
}

LETTERS = "ABCDEFGHIJ"

def render(item, arm):
    opts = "\n".join(f"{LETTERS[i]}. {o}" for i, o in enumerate(item["options"]))
    return f"{item['question']}\n\n{opts}\n\n{arm['instruction']}"

def call(prompt, arm, seed):
    body = json.dumps({
        "model": MODEL,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": arm["temperature"],
        "max_tokens": arm["max_tokens"],
        "seed": seed,
    }).encode()
    req = urllib.request.Request(
        f"{BASE}/chat/completions",
        data=body,
        headers={"Authorization": f"Bearer {KEY}", "Content-Type": "application/json"},
    )
    with urllib.request.urlopen(req, timeout=180) as r:
        payload = json.load(r)
    return payload["choices"][0]["message"]["content"]

# Answer extraction. This function IS a scoring decision — log every
# output it fails on, and count those separately from wrong answers.
ANSWER_RE = re.compile(r"(?:answer\s*[:\-]?\s*)?\(?([A-J])\)?\b", re.I)

def extract(text):
    tail = text.strip()[-200:]                # prefer the end of the reply
    matches = ANSWER_RE.findall(tail) or ANSWER_RE.findall(text)
    return matches[-1].upper() if matches else None

def run_arm(items, name, arm, seed=1234):
    results = []
    unparsed = 0
    for item in items:
        try:
            out = call(render(item, arm), arm, seed)
        except Exception as exc:                 # network, rate limit, refusal
            print(f"  [{name}] {item['id']}: request failed: {exc}", file=sys.stderr)
            results.append(0); unparsed += 1; continue
        got = extract(out)
        if got is None:
            print(f"  [{name}] {item['id']}: UNPARSED -> {out[:80]!r}", file=sys.stderr)
            unparsed += 1
        results.append(1 if got == item["answer"].upper() else 0)
    return results, unparsed

def bootstrap_ci(values, n=10_000, seed=0):
    rng = random.Random(seed)
    k = len(values)
    means = []
    for _ in range(n):
        means.append(sum(rng.choice(values) for _ in range(k)) / k)
    means.sort()
    return means[int(0.025 * n)], means[int(0.975 * n)]

def main(path):
    items = [json.loads(line) for line in open(path, encoding="utf-8") if line.strip()]
    print(f"{len(items)} items, model={MODEL}\n")

    scores = {}
    for name, arm in ARMS.items():
        vals, unparsed = run_arm(items, name, arm)
        scores[name] = vals
        lo, hi = bootstrap_ci(vals)
        acc = sum(vals) / len(vals)
        print(f"{name:24s} acc={acc:.3f}  95% CI [{lo:.3f}, {hi:.3f}]  unparsed={unparsed}")

    # Paired difference: same items, so pair them rather than comparing
    # two independent intervals. This is the correct test and it is
    # strictly more sensitive.
    a, b = list(ARMS)
    diffs = [x - y for x, y in zip(scores[b], scores[a])]
    d = sum(diffs) / len(diffs)
    lo, hi = bootstrap_ci(diffs)
    print(f"\npaired delta ({b} - {a}) = {d:+.3f}  95% CI [{lo:+.3f}, {hi:+.3f}]")
    if lo <= 0 <= hi:
        print("Interval contains zero: this configuration change is not distinguishable\n"
              "from no change on this item set. Add items or accept the null.")
    else:
        print("Interval excludes zero: the configuration change moved the score.")

if __name__ == "__main__":
    main(sys.argv[1] if len(sys.argv) > 1 else "items.jsonl")

Three things about this script are the point rather than incidental. The two arms differ in exactly one dimension, so the delta is attributable. Unparsed outputs are counted separately from wrong answers, because those are different problems with different fixes. And the comparison is paired — the same items in both arms — which is both correct and considerably more sensitive than comparing two independent confidence intervals. The statistics are developed in statistical significance with non-deterministic models.

The failures you will actually hit

When the run finishes, write down what you did before you write down what you got. Every field required for that is enumerated in the eight reporting fields, and filling them in afterwards from memory is how numbers become unreproducible a week later.