Why Public Benchmarks Don't Predict Your Results

The complaint that benchmarks are useless is too strong, and the habit of picking a model by leaderboard row is too weak. The useful question is how much the public ordering and your ordering actually disagree — which is a number you can compute in an afternoon and nobody else can compute for you.

The benchmarks are not lying to you

A benchmark is a sample from a distribution of tasks. MMLU samples multiple-choice academic knowledge; SWE-bench samples resolving real GitHub issues in Python repositories; GSM8K samples grade-school word problems. Each is a faithful measurement of the thing it samples.

The failure is not in the measurement, it is in the extrapolation. Your task is a different sample from a different distribution, and a ranking on one sample transfers to another only to the extent that the two are correlated. Nobody publishes that correlation for your task, because nobody has your task.

Four mechanisms that break transfer

1. Format dominance

Most classic benchmarks are multiple choice. Answering A/B/C/D is a constrained selection problem; writing a well-structured 300-word reply that obeys a tone guide and never invents a policy is a generation problem with a dozen simultaneous constraints. Models differ enormously in how well their selection ability predicts their generation ability, and multiple-choice scoring is additionally sensitive to implementation details — whether the harness scores by letter token or by the log-likelihood of the full option string can move a score, which is one reason two harnesses report different numbers for the same model.

2. The ceiling

When several models cluster within a couple of points near the top of a benchmark, the ordering among them is mostly noise plus whatever idiosyncrasies that particular test set has. Saturated benchmarks rank; they no longer discriminate. Treat any gap smaller than the benchmark’s own reported error bar as a tie, and note how few leaderboards show you that error bar at all.

3. Contamination

A benchmark published before a model’s training cutoff may be in its training data, in which case the score measures a mixture of capability and recall — and the mixture is different for each model, which is exactly what destroys a ranking. Detection is a page of its own.

4. Everything the benchmark holds constant

Benchmarks are usually run with a fixed short prompt, no tools, no retrieval, no system prompt of consequence, and often at temperature zero. Your system has a 2,000-token system prompt, a retrieval step that occasionally returns garbage, a tool schema with nine functions, and a latency budget. Models that are close on the naked task can be far apart once instruction-following under a long system prompt and reliable tool-call formatting are in play, and no public score covers that combination.

Measure the disagreement yourself

The honest version of “benchmarks do not transfer” is a rank correlation between the public ordering of the candidates you are considering and the ordering your own eval produces. Run your eval across six to ten candidates, write down the public ordering you would otherwise have trusted, and compute Spearman’s ρ. No dependencies:

from itertools import combinations

def _rank(values):
    """Ranks, averaging ties (1 = best)."""
    order = sorted(range(len(values)), key=lambda i: -values[i])
    ranks = [0.0] * len(values)
    i = 0
    while i < len(order):
        j = i
        while j + 1 < len(order) and values[order[j + 1]] == values[order[i]]:
            j += 1
        avg = (i + j) / 2 + 1
        for k in range(i, j + 1):
            ranks[order[k]] = avg
        i = j + 1
    return ranks

def spearman(a, b):
    ra, rb = _rank(a), _rank(b)
    n = len(a)
    ma, mb = sum(ra) / n, sum(rb) / n
    num = sum((x - ma) * (y - mb) for x, y in zip(ra, rb))
    da = sum((x - ma) ** 2 for x in ra) ** 0.5
    db = sum((y - mb) ** 2 for y in rb) ** 0.5
    return num / (da * db)

def kendall_tau_b(a, b):
    """Concordant minus discordant pairs; robust with few candidates."""
    con = dis = ta = tb = 0
    for i, j in combinations(range(len(a)), 2):
        da, db = a[i] - a[j], b[i] - b[j]
        if da == 0 and db == 0:
            continue
        if da == 0:
            ta += 1
        elif db == 0:
            tb += 1
        elif (da > 0) == (db > 0):
            con += 1
        else:
            dis += 1
    denom = ((con + dis + ta) * (con + dis + tb)) ** 0.5
    return (con - dis) / denom if denom else 0.0

MODELS    = ["a", "b", "c", "d", "e", "f", "g"]
PUBLIC    = [82.1, 79.4, 78.8, 76.0, 74.3, 71.9, 68.2]   # the leaderboard column
MINE      = [0.74, 0.80, 0.62, 0.78, 0.66, 0.70, 0.58]   # your eval pass rate

print("spearman", round(spearman(PUBLIC, MINE), 3))
print("kendall ", round(kendall_tau_b(PUBLIC, MINE), 3))
for m, p, q in sorted(zip(MODELS, _rank(PUBLIC), _rank(MINE)), key=lambda r: r[1]):
    print(f"{m:>4}  public #{p:<4.1f} mine #{q:<4.1f}  {'<-- moved' if abs(p-q) >= 2 else ''}")

The PUBLIC and MINE arrays above are placeholders to show the shape of the input; the point of the script is the two numbers it prints for your arrays. With seven candidates, Kendall’s τ-b is the more honest of the two — Spearman on a handful of points is jumpy, and τ has the direct interpretation of a rebalanced probability that a randomly chosen pair is ordered the same way by both.

Reading the coefficient you get

• High agreement. Your task is well covered by that benchmark’s distribution. You can use the leaderboard as a cheap prefilter and only run your eval on the top few. That is a genuine saving and worth knowing.
• Near zero. The public ordering carries no information about your task. Selecting on it is selecting at random while feeling informed, which is strictly worse than knowing you are guessing.
• Negative. Rare and worth investigating rather than celebrating. It usually means your eval is dominated by one characteristic — brevity, refusal behaviour, format compliance — that happens to be anti-correlated with what the benchmark rewards. You have learned something real about your eval.
• The individual movers. The most useful output is the last block, not the coefficient. A model that is mid-table publicly and first on your eval is the finding; the summary statistic just tells you whether to expect such findings.

Recompute this when you change task, not when a new leaderboard comes out. The coefficient is a property of the pair (your task, that benchmark), and it moves when your task moves.

Two caveats on interpretation. With seven or eight candidates the coefficient itself has a wide sampling distribution, so treat it as a rough band rather than a number — and if you want a p-value, a permutation test over the orderings is more honest than a table lookup at that sample size. And your own eval ordering has error too: if two of your candidates are separated by less than your eval’s confidence interval, their relative rank is arbitrary and they should be tied before the ranks are computed, which is why the helper above averages ties rather than breaking them.

What benchmarks are still good for

Three things, none of which is picking a winner. They generate a shortlist: a model that cannot do arithmetic at all will show it on GSM8K, and you can drop it without spending your own eval budget. They provide a floor: sustained catastrophic scores are real information even when high scores are not. And they support longitudinal reading of a single model family, where the confound of comparing different labs’ harnesses does not apply.