Eval Run Cost and Time Estimator

Multiplies examples, models and repeats into calls, cost and wall clock — including the LLM judge, and including what it all costs per week in CI.

Examples in the eval set

Models under testRates below are the average across them; run this once per model if they differ a lot.

Repeats per exampleSamples per example. One is not enough to separate two close models.

Input tokens per call

Output tokens per call

Input rate per 1MExample figure — use your own.

$ / 1M

Output rate per 1MExample figure — output is normally the dearer half.

$ / 1M

Judge calls per output1 for a single LLM judge, 0 if you grade programmatically, 3 for a panel.

Judge input tokens per gradeThe rubric, the question and the answer being graded.

Judge output tokens per grade

Judge input rate per 1MExample figure.

$ / 1M

Judge output rate per 1MExample figure.

$ / 1M

Concurrent calls

Seconds per callMean wall clock, including the judge's own latency.

Runs per weekIf this sits in CI, that is the number that matters.

Cost of one full eval run

$17.91

3,600 API calls, about 45.0 min of wall clock at 8 concurrent. At 5 runs a week that is $89.55 weekly, $4,657 a year.

Calls — examples × models × repeats: 1,800
Judge calls: 1,800
Input tokens: 1,620,000
Output tokens: 630,000
Cost of the models under test: $14.31
Cost of the judge: $3.60
Judge share of the bill: 20.1%
Total per run: $17.91
Cost per example: $0.09
Wall clock: 45.0 min
Weekly, at the run rate above: $89.55

What this assumes: every example costs the same tokens, which is close enough for a set built to one template and wrong for a set with a few enormous cases. Rates are a single pair across all models under test; if one of them is an order of magnitude dearer, its share of the bill is larger than this suggests and the tool should be run per model. Wall clock assumes the concurrency is achievable — a provider rate limit will make it longer, and grading serially after generation rather than alongside it roughly doubles the elapsed time without changing the cost.

The repeat count is the expensive decision

Three numbers multiply into the call count, and only one of them feels expensive when you choose it. Adding a model is a visible decision. Adding examples is a visible decision. Raising repeats from one to three is a line in a config file, and it triples everything.

It is also usually the right call, which is what makes it worth costing rather than avoiding. A single sample per example measures the model plus a coin flip; models that differ by a few points cannot be separated that way, and an eval that reports a difference which disappears on the next run is worse than no eval, because someone will ship on it. Repeats buy you a variance estimate — the thing that tells you whether the difference is real.

The judge is the term that surprises people. A grader reading a rubric, a question and a full answer often consumes more input tokens than the call it is grading, so a cheap judge model can still be a third of the bill. The breakdown prints that share; if it is large, the fix is usually a shorter rubric or programmatic grading for the cases that admit it, not a cheaper judge.

The annualised figure is there because eval cost is a recurring cost the moment it enters CI. A run that costs a few dollars is nothing; the same run on every pull request, several times a day, is a budget line. Decide which of the three multipliers you can afford before it becomes one.

Eval Run Cost and Time Estimator

The repeat count is the expensive decision

The reasoning behind this

Related tools