How Arena Rankings and Elo Are Computed

An arena rating is not a score on a test. It is a parameter fitted to a pile of pairwise votes: people typed a prompt, saw two anonymous answers, picked one, and a statistical model was fitted to those outcomes. Understanding which model, and what its parameters mean, changes how you read the leaderboard.

Where the votes come from

The mechanism is the same across the public arenas. A visitor types their own prompt. Two models, chosen without telling the visitor which, answer side by side. The visitor votes: left, right, tie, or both bad. The pair identities are revealed afterwards. Millions of these accumulate.

Three properties of that data are worth fixing in mind before any arithmetic, because they bound everything the rating can mean.

• The prompts are whatever people type into an arena. Not a curated distribution, not your distribution. Short questions, creative requests, coding snippets, tests of the arena itself. There is no ground truth for any of them.
• The judgement is preference, not correctness. A voter picks the answer they liked. On a factual question they may not be able to tell which is right, so the vote measures persuasiveness as much as accuracy.
• The voters are self-selected and anonymous. Which makes the population unknowable and the data susceptible to organised voting, something the operators mitigate but cannot eliminate.

The formula, and what a 100-point gap means

The Bradley-Terry model gives each model a strength parameter and says the probability that model i beats model j is a logistic function of their difference. In the Elo-scaled form the arenas use:

                        1
P(i beats j)  =  ---------------------
                  1 + 10^((Rj - Ri)/400)

The 400 is a scale constant inherited from chess. It is arbitrary:
it fixes what "100 points" means and nothing else.

Worked, for the gaps you actually see on a leaderboard:

Gap (Ri - Rj)   10^(-gap/400)        P(i wins)     In words
-------------------------------------------------------------------
    0          10^0     = 1.000       0.500        a coin flip
   10          10^-0.025= 0.944       0.514        51 wins in 100
   25          10^-0.0625=0.865       0.536        54 in 100
   50          10^-0.125= 0.750       0.571        57 in 100
  100          10^-0.25 = 0.562       0.640        64 in 100
  200          10^-0.5  = 0.316       0.760        76 in 100
  400          10^-1    = 0.100       0.909        91 in 100

Check the 100-point row:  1 / (1 + 0.562) = 1 / 1.562 = 0.640.

Read the top of that table carefully. A ten-point rating gap is a 51-to-49 preference. A twenty-five point gap — which on a live leaderboard can be four or five places — means that if you ran the two models against each other a hundred times on arena-style prompts, one would be preferred 54 times. Presented as an ordered list, that looks like a ranking. Presented as a win rate, it looks like what it is.

Elo and Bradley-Terry are not the same thing

The leaderboards say “Elo” and mostly compute Bradley-Terry. The distinction is worth knowing because it explains why the numbers stopped moving around.

Classical Elo is an online update rule. After each game a player’s rating moves toward the outcome by a step proportional to how surprising the outcome was:

R_new = R_old + K * (S - E)

  S = actual result (1 win, 0.5 draw, 0 loss)
  E = expected result from the logistic formula above
  K = step size (32 is the traditional chess value)

Example: R_i = 1200, R_j = 1100, so E = 0.640 for i.
  i wins:   1200 + 32*(1   - 0.640) = 1200 + 11.5 = 1211.5
  i loses:  1200 + 32*(0   - 0.640) = 1200 - 20.5 = 1179.5
  draw:     1200 + 32*(0.5 - 0.640) = 1200 -  4.5 = 1195.5

That rule was designed for chess, where players improve over time and the rating should track a moving target. Applied to a fixed set of model checkpoints it has an unwanted property: the result depends on the order the games arrived in. Rerun the same votes in a different sequence and you get different ratings.

Bradley-Terry removes that. Instead of updating incrementally, it fits all the strength parameters at once by maximum likelihood over the whole vote history — equivalent to a logistic regression where each match is a row, the model identities are the features, and the winner is the label. The answer is order-independent and reproducible from the vote log. The arenas moved to this and kept the Elo scale so the numbers stayed familiar.

Ties, and how they are absorbed

Arena voters can declare a tie, and a large share of votes are ties. Plain Bradley-Terry has no tie outcome, so implementations handle them one of two ways.

• Split the tie. Count each tie as half a win for each side. Simple, and the standard choice. It treats “these were equally good” and “these were equally bad” as the same event, which they are not.
• Model the tie explicitly. The Rao-Kupper extension adds a threshold parameter: outcomes within a band of indifference are ties, and the band width is fitted. More faithful, less common, and it changes ratings slightly.

Either way, a rating computed with ties split is not comparable to one computed with ties dropped, and dropping them inflates the apparent separation between models. Ask which was done before comparing numbers from two sources.

Rank is an interval, not a position

Every rating is an estimate from a finite number of comparisons, so it has a standard error, usually obtained by bootstrapping over the votes. Reputable arenas publish it. Two models whose intervals overlap have not been separated, and the leaderboards say so — they assign both the same rank number, which is why you see three models all listed at rank 2.

The interval narrows with the number of votes involving that model, so a newly added model has a wide one and its position is unstable for days. The comparison that matters is not “which is higher” but “do the intervals overlap”. Reporting a rank without an interval is the arena equivalent of reporting an accuracy without n — see the eight reporting fields.

Style is a confound, and it is measurable

Human preference votes correlate with things that are not answer quality: length, markdown structure, headers, bullet points, a confident opening line. This is not speculation — it is why the arena operators added a style-controlled ranking, which fits the same Bradley-Terry model with additional covariates for response length and formatting so that the model strength parameter absorbs less of the style effect.

The style-controlled and raw boards differ, sometimes substantially, and which one a screenshot came from is rarely stated. If you are quoting an arena position, say which board. The same effect is why an LLM judge shows the same bias — it was trained on human preferences that contained it.

What an arena rating is a bad proxy for

• Correctness. The voter did not check. On tasks where verification is hard, preference and accuracy come apart, and the rating follows preference.
• Your prompt distribution. Arena prompts are short, open-ended and free of your system prompt, your tools, your documents and your constraints. This is the core of why benchmark rank does not transfer.
• Structured or long-form work. Nobody is voting on a 40-turn agent run or a JSON schema conformance rate. For those, see function-calling benchmarks and agentic benchmarks.
• Cost and latency. Voters are not shown either. A model that took twenty seconds and one that took two are voted on the same way.