Evaluating Cost and Latency Alongside Quality

Every real model decision is a three-way trade, and every leaderboard reports one axis of it. The fix is not a better composite score — it is refusing to compute one until you have written down what an extra hundred milliseconds is worth to you in quality points.

One number cannot hold three axes

Quality, cost per request and latency are not commensurable. Any single score that combines them has smuggled in an exchange rate — how many quality points is a cent worth — and hiding that rate inside a formula does not make it less of a business decision. It makes it an unreviewed one.

There is also a structural reason these must be measured together rather than looked up separately. All three depend on the prompt. A longer system prompt raises cost and time to first token; a reasoning-heavy configuration raises quality and both costs; a shorter output format lowers cost and latency and may or may not lower quality. A cost figure taken from a price list and a quality figure taken from an eval are figures for two different systems.

The third reason is that the axes are not independent of each other either. Quality improvements that come from generating more tokens — reasoning traces, self-critique passes, retries on a failed structural check — buy quality with cost and latency directly, at a rate you can only see if all three are recorded in the same run. A configuration that scores two points higher because it thinks for four hundred extra tokens has not beaten its rival; it has picked a different point on a curve, and the interesting question is whether the rival at the same token budget would land in the same place.

Measuring all three in the same run

Record, per eval item and per sample, at minimum:

Report latency as p50 and p95, never as a mean. Inference latency distributions have long right tails, and a mean is dragged past anything a user actually experienced by a handful of slow requests — while p95 is the number that describes your worst-served customers.

The Pareto front

A configuration is dominated if some other configuration is at least as good on all three axes and strictly better on one. Dominated configurations can be discarded without any judgement call, which is what makes the front useful: it is the part of the analysis that requires no opinion.

from dataclasses import dataclass

@dataclass
class Config:
    name: str
    quality: float      # higher better, e.g. mean item score
    cost: float         # lower better, dollars per 1k requests
    p95_ms: float       # lower better

def dominates(a, b):
    """a dominates b: at least as good everywhere, strictly better once."""
    ge = (a.quality >= b.quality and a.cost <= b.cost and a.p95_ms <= b.p95_ms)
    gt = (a.quality >  b.quality or  a.cost <  b.cost or  a.p95_ms <  b.p95_ms)
    return ge and gt

def pareto_front(configs):
    return [c for c in configs
            if not any(dominates(o, c) for o in configs if o is not c)]

def with_tolerance(configs, q_eps=0.01, c_eps=0.02, t_eps=0.05):
    """Same, but a difference smaller than measurement noise is not a win.
    q_eps in score points; c_eps and t_eps as relative fractions."""
    def dom(a, b):
        ge = (a.quality >= b.quality - q_eps
              and a.cost <= b.cost * (1 + c_eps)
              and a.p95_ms <= b.p95_ms * (1 + t_eps))
        gt = (a.quality > b.quality + q_eps
              or a.cost < b.cost * (1 - c_eps)
              or a.p95_ms < b.p95_ms * (1 - t_eps))
        return ge and gt
    return [c for c in configs
            if not any(dom(o, c) for o in configs if o is not c)]

The tolerance version is the one to use. A quality difference of 0.003 on a 300-item eval is not a difference — it is inside the confidence interval — and a strict dominance test will happily eliminate a configuration that is fifty percent cheaper on the strength of it. Set q_eps from the actual width of your quality interval, which is a number the bootstrap gives you.

Choosing among the survivors

The front usually has two to five members and you have to pick one. Two honest methods, and the difference between them is which constraint is real.

Constraint plus objective

Usually the right framing, because usually one axis is a hard product requirement. “p95 under 2.5 seconds, quality at least non-inferior to today, then minimise cost.” Filter, then sort. It requires no exchange rate and produces a decision anyone can audit.

Explicit exchange rate

When you genuinely need to trade quality against money, write the rate down rather than burying it in weights. Derive it from something real: if a failed request costs you a support contact, and a support contact costs a known amount, then one point of quality across a known request volume has a monetary value, and cost and quality become the same unit. Latency is harder and usually enters as a constraint rather than a term. The virtue of doing it this way is that the assumption is visible and someone can disagree with the number rather than with the conclusion.

Worked, with your own figures in place of these: if a failed response leads to a support contact in some fraction of cases, and a support contact has a known fully-loaded cost, then one percentage point of quality across a month’s request volume is worth volume × 0.01 × contact_rate × contact_cost. Set that against the monthly inference cost difference between two configurations and the comparison is in one unit. The arithmetic is not the point — the point is that the moment you write it down, someone who knows the support cost better than you will correct it, and the decision improves. A weighted score with hidden coefficients gets no such correction because nobody can see what to argue with.

What to avoid is a weighted sum of normalised scores. Normalisation depends on the range of candidates in the run, so adding a slow candidate silently rescales latency and can reorder the winners — which is a property of the arithmetic, not of anything you care about.

Reporting it so people can act

A three-axis result should be one table with a row per configuration, columns for quality with its interval, cost per thousand requests, p50 and p95 latency, and a flag for whether the row is on the front. Sort by cost. That table answers the question a decision-maker actually asks — “what do we give up by taking the cheap one” — in a form nobody has to interpret.

Add one line per front member saying what it is for: “cheapest that meets the latency budget”, “best quality regardless of cost”, “best quality under a cent per request”. Those labels are how a Pareto front turns into a routing policy, where different request classes take different rows.