Cost- and Latency-Aware Leaderboards

A quality score alone ranks models as though they cost the same, which none of them do. Adding cost and latency is right, but the obvious way to add them — dividing the score by the price — produces a metric maximised by a model too weak to deploy. The two formulations that work are a frontier and a cost-to-fixed-quality curve.

Why a second axis is needed at all

Serving prices across models that sit within a few points of each other on a quality benchmark span more than an order of magnitude. So the ranking that a single-axis leaderboard produces is systematically uninformative for a deployment decision: the difference it shows is often smaller than its own noise, while the difference it hides is a factor of thirty on the bill.

Several public leaderboards now report a second axis. Some publish cost per task alongside accuracy; some publish index scores against price and against throughput. The arithmetic behind all of them is the same and it is worth being able to reproduce, because the composite you want is usually not the composite they publish.

Computing cost per task

Per request:

  cost = (input_tokens  * input_price_per_million  / 1e6)
       + (output_tokens * output_price_per_million / 1e6)

Worked, one benchmark item:
  input  1,200 tokens at $3.00 per million  -> 1200  * 3.00 / 1e6 = $0.00360
  output   400 tokens at $15.00 per million ->  400 * 15.00 / 1e6 = $0.00600
  item cost                                                        = $0.00960

Over a 500-item benchmark: 500 * $0.00960 = $4.80

Now the same model in a reasoning mode that emits 4,000 hidden
reasoning tokens per item, billed as output:
  output 4,400 tokens at $15.00 per million -> $0.06600
  item cost = $0.00360 + $0.06600           = $0.06960
  over 500 items                            = $34.80

Same model. Same benchmark. 7.25x the cost.

Three things that this arithmetic makes visible and a leaderboard usually does not.

• Reasoning tokens dominate. They are billed as output, they are often not shown to you, and on a reasoning-heavy benchmark they can be most of the cost. A cost comparison that counts only visible output tokens is wrong by a large factor — see hidden reasoning tokens and who bills for them.
• Tokens are not portable across models. Two models tokenise the same prompt into different counts, so cost per task is comparable and cost per token is not. Always compare at the task level — the mechanism is in comparing tokenisers.
• Caching changes the answer. A benchmark with a long shared system prompt is much cheaper under prompt caching than the list price suggests, and the discount is not uniform across providers.

Why score-per-dollar is the wrong summary

The tempting composite is quality divided by cost. It fails, and the failure is easy to demonstrate.

Four models on the same 500-item benchmark.

model   accuracy   cost of full run   accuracy / dollar
-------------------------------------------------------
  A       0.91          $34.80              2.6
  B       0.86           $4.80             17.9
  C       0.71           $0.90             78.9
  D       0.42           $0.15            280.0

Ranked by "score per dollar":  D, C, B, A.

D is the winner and D is wrong on nearly six answers in ten.

The ratio's problem is structural: as cost tends to zero the ratio tends
to infinity for ANY non-zero accuracy. The metric is maximised at the
cheap end regardless of whether the quality is usable.

The ratio also assumes quality and cost are linearly exchangeable, which they are not. Below some accuracy your product does not work at any price, and the ratio has no way to express a threshold. Any leaderboard whose headline is a quality-over-price ratio is inviting this error, and the same objection applies to quality-over-latency ratios.

The Pareto frontier

The correct primitive is the frontier: the set of options that are not beaten on both axes at once. Everything else can be discarded without knowing your preferences, and among the frontier no choice is objectively better — you choose using a constraint you supply.

Same four models, plus a fifth:

model   accuracy   cost
------------------------
  A       0.91    $34.80
  B       0.86     $4.80
  C       0.71     $0.90
  D       0.42     $0.15
  E       0.68     $6.00

E is dominated: B is more accurate AND cheaper. Discard E.
A, B, C, D are all on the frontier — each is the best available
at its price point, and none is beaten on both axes.

Choosing among them needs one number from you:
  "accuracy must be at least 0.85"  -> B (cheapest that qualifies)
  "budget is $1.00 per run"         -> C (best that fits)

This is the presentation to demand, and the reason is that it separates the measurement from the decision. The frontier is a fact about the models; the constraint is a fact about you. A ratio jams the two together and hides the second inside the arithmetic. The three-axis version, adding latency, is developed in evaluating cost and latency alongside quality.

Cost to reach a fixed quality

The other honest composite inverts the question. Instead of asking what quality a fixed budget buys, ask what a fixed quality costs. This is usually the question a product actually poses, and it has a neat property: you can spend more on a weaker model to close the gap, and the arithmetic tells you whether that is worthwhile.

Self-consistency: sample k times, take the majority answer.
Cost scales linearly with k; accuracy rises with diminishing returns.

Model C, base accuracy 0.71 at $0.90 per run:

  k    cost      observed accuracy (you must measure this — it is
                 task-dependent and cannot be predicted)
  1    $0.90        0.71
  3    $2.70        ?
  5    $4.50        ?

If k=3 reaches 0.86 at $2.70, it beats model B (0.86 at $4.80).
If k=3 only reaches 0.78, it does not, and no k will — majority
voting converges to the model's own consistent answer, which is
wrong on items it is confidently wrong about.

The measurement is cheap and the answer is specific to your task.
This is the sweep worth running before choosing a model.

Note what the code block does not do: it does not tell you the accuracy at k = 3. That number depends on the task and on how the model’s errors are distributed, and nobody can supply it for you. The point of the layout is that it is a two-hour experiment with a decisive answer — the mechanism is described in self-consistency.

Latency numbers measure a serving stack

A latency figure on a leaderboard is not a property of a model. It is a measurement of one provider’s serving infrastructure, at one moment, from one place, under one load. Six things move it and none is the model’s weights.

Throughput and latency are different numbers

One more distinction that efficiency leaderboards routinely blur, and it changes which model is right for a job. Tokens per second measured on a single stream and tokens per second measured across a whole server are different quantities that move in opposite directions under batching.

Per-stream rate  = tokens a single request receives per second.
                   What one user experiences.

Aggregate rate   = tokens the server emits per second across all
                   concurrent requests. What your bill and your
                   capacity plan depend on.

Batching trades one for the other. Sketching the shape:

  concurrency   per-stream tok/s   aggregate tok/s
  ------------------------------------------------
       1              60                  60
       8              50                 400
      32              35               1,120
     128              18               2,304

The per-stream rate falls as the batch grows; the aggregate rises,
with diminishing returns, until memory or compute saturates.

Consequence: a provider optimising for cost runs large batches and
your individual requests feel slower. A provider optimising for
interactive latency runs small batches and charges more per token.
Both are 'the same model'.

The numbers above are illustrative of the shape rather than measured from any particular system — the exact curve depends on the model, the hardware, the sequence lengths and the serving engine, and the only way to know yours is to load-test it. What is reliable is the direction: per-stream rate falls and aggregate rate rises as concurrency increases, and no single “tokens per second” figure can describe both.

So decide which one your product needs before reading a throughput column. A chat interface lives on per-stream rate and on time to first token. A nightly batch job over a million documents lives on aggregate rate and cares about per-stream latency not at all. Picking a model using the wrong one of those is a common and expensive mistake — the mechanics of why the two behave differently are in why generation is sequential.

Reading an efficiency leaderboard

1. Check whether the cost figure counts reasoning tokens. If the benchmark involves reasoning and the cost column looks low, it probably does not.
2. Check the price source and its date. Serving prices change, and a leaderboard that caches them is quietly stale — the decay problem in reading a model release critically applies to leaderboards as much as to launch posts.
3. Reject any single ranking derived from a quality-over-price ratio. Ask for the two columns and plot them yourself; it is a five-minute job and it produces the frontier.
4. Check whether latency is p50 or a mean, and which latency. If it does not say, it is a mean of total time and it is not useful.
5. Apply your own constraint last. The frontier is the measurement; the threshold is your decision, and keeping them separate is the whole discipline.