Pairwise Comparison vs Absolute Scoring

“Rate this answer out of ten” and “which of these two answers is better” feel like the same question asked two ways. They are not: one requires an absolute standard held in the judge’s head, the other requires only an ordering, and that difference decides which numbers you can trust.

The two questions are not equivalent

Absolute scoring asks for a cardinal judgement: a number on a scale whose meaning must be the same for item 1 and item 200, and the same today as last month. Pairwise comparison asks for an ordinal one: a direction, with no shared unit required.

Ordinal judgements are easier for humans and for models, and this is not a fact about language models — it is why psychophysics uses forced-choice designs, why wine is judged in flights, and why ranking systems in games are built on match outcomes rather than on referees scoring each player out of a hundred. A comparison needs no calibration; a rating is nothing but calibration.

Why absolute scores drift

Three mechanisms, all of which you can observe in a judge’s output without any statistics.

• No fixed referent. A 7 means “good, but not as good as the best thing I can imagine”, and what the judge can imagine depends on the item, the prompt and the phase of the run. Two identical answers to two different questions routinely get different scores because the implied ceiling moved.
• Range compression. Judges concentrate mass in the middle-upper part of any scale. A 1-10 rubric in practice yields almost everything between 6 and 9, so your ten-level instrument has four levels and most of your dynamic range is unused. That is why an improvement worth shipping often shows up as +0.2 on a ten-point scale — the compression, not the improvement, produced that number.
• Rubric coupling. Any edit to the rubric wording shifts the whole scale, so historical scores are not comparable across rubric versions. With pairwise comparison against a frozen baseline output, a rubric edit changes the sensitivity of the test but the quantity being estimated — is this better than the baseline — stays the same thing.

The standard aggregation for pairwise results is the Bradley-Terry model, which assigns each system a strength s such that the probability system i beats system j is exp(s_i) / (exp(s_i) + exp(s_j)), fitted by maximum likelihood over the observed wins. Chatbot Arena’s ratings are produced this way. The output is an interval scale recovered from ordinal observations, which is exactly the trade being made: you gave up asking for numbers and got numbers back anyway, with the calibration done by the model instead of by the judge’s intuition.

Ties are the detail that decides whether any of this works. A three-way verdict — A, B or tie — is much better behaved than a forced binary choice, because forcing a decision on genuinely equivalent outputs converts real indifference into coin flips that then look like signal. But it introduces a policy question: a tie can be counted as half a win, dropped, or reported as its own rate. Half a win is the conventional choice and the one Bradley-Terry fitting expects. Reporting the tie rate separately is the more informative one, because a candidate that ties 70% of the time against production is telling you the change did nothing on most items, which no win rate can say.

What pairwise costs

The cost is combinatorial and people underestimate it. With n systems and m eval items, a full round robin is n(n-1)/2 × m comparisons, and if you are correcting for position bias by running both orders, double it.

Concretely, with 8 candidate systems and 200 items: absolute scoring costs 8 × 200 = 1,600 judge calls. Round-robin pairwise costs 28 × 200 = 5,600, or 11,200 with order swaps. That is a factor of seven, and it grows — at 15 systems the round robin is 105 pairs per item.

Two ways out. Active sampling schedules comparisons where the current estimates are most uncertain, which is what large arenas do rather than running exhaustive round robins. Or use the anchored design below, which is what most application teams should actually do.

The anchored design

You almost never need a full ranking of eight systems. You need to know whether the candidate is better than what you ship today. So freeze the current production output for each eval item as a baseline artefact and compare everything against it:

• Cost becomes linear: n × m comparisons, the same order as absolute scoring, plus the doubling for order swaps.
• The output is directly the quantity you care about — a win rate against production, per item, which is a paired binary outcome and therefore feeds straight into a McNemar test with real power.
• It is stable across time as long as the baseline artefacts are frozen. Re-generating the baseline each run reintroduces exactly the drift you were avoiding: store the text, not the model id.
• Its blind spot is absolute level. If production is bad and the candidate is bad in the same way, the anchored comparison is a tie and reports nothing. Pair it with a small absolute or programmatic check to catch that.

Choosing between them

A practical composite that works for most teams: programmatic checks as a hard gate, an anchored pairwise win rate as the headline quality number, and a small absolute rubric-sum score reported alongside so that a slow decline in everything is still visible. Three numbers, each doing a job the others cannot.

One last piece of arithmetic, because it catches people out. A pairwise win rate is a proportion, so its precision follows the same rule as any other: with 200 items the 95% interval around a win rate near 0.55 is roughly seven points wide either side. A candidate that wins 55% of comparisons on a 200-item set has not been shown to be better — the interval comfortably contains 0.50. Pairwise comparison is more stable per judgement than absolute scoring; it is not exempt from needing a sample size, and the anchored design is worth adopting partly because it makes that sample size affordable.