Estimating the Monthly Bill From Your CI Eval Suite Before You Turn It On

The estimate people produce is cases times price and it is wrong by about an order of magnitude, because the trigger frequency and the reruns are where the money is. Here is the whole calculation with every input named, so you can substitute your own numbers rather than trusting anyone’s.

The formula

Five inputs, all of which you can read off your own repository before writing a line of eval code.

cost_per_case  = (in_tokens  x price_in)  + (out_tokens x price_out)
cost_per_run   = cases x cost_per_case x samples_per_case
monthly_cost   = cost_per_run x runs_per_day x active_days x reruns

• price_in, price_out — your provider’s published per-token prices, normally quoted per million tokens. These are the only inputs that come from outside your repository and the only ones that change without you doing anything.
• in_tokens — system prompt plus tool schemas plus the case. Measure it; do not estimate it. Tool schemas in particular are much larger than people expect.
• out_tokens — what the model actually returns, not your max_tokens ceiling.
• runs_per_day — count the triggers, not the commits. A push, a pull-request event and a merge-queue check on the same change are three runs.
• reruns — the multiplier from retried jobs and re-requested checks. It is never 1.0.

Two of the five are worth measuring rather than guessing, and both are measurable in an afternoon. Run the suite once against the real provider, sum the usage object from every response, and divide — that gives you real per-case token counts including the tool schemas and any hidden system content your framework adds. Then count the runs: the provider’s own request count over a week, divided by seven, is more honest than any count derived from your commit history, because it includes the triggers you forgot about.

A worked monthly figure

Every number below is an assumption, stated so you can replace it. For the price inputs this page uses a placeholder rate of $3.00 per million input tokens and $15.00 per million output tokens, which is a common shape for a mid-tier hosted model but is not a quotation of any current price. Substitute the figures from your provider’s pricing page before you act on the result — for example OpenAI’s pricing page or Anthropic’s pricing page.

Assumptions
  cases              120
  in_tokens/case   1,800   (system prompt + 4 tool schemas + the case)
  out_tokens/case    350
  samples_per_case     1
  price_in         $3.00 per 1M tokens   [placeholder — substitute]
  price_out       $15.00 per 1M tokens   [placeholder — substitute]

Per case
  input    1,800 / 1,000,000 x $3.00  = $0.00540
  output     350 / 1,000,000 x $15.00 = $0.00525
  total                                = $0.01065

Per run
  120 cases x $0.01065                 = $1.278

Per month
  14 runs/day x $1.278                 = $17.89 per day
  22 active days x $17.89              = $393.62 per month

Two things are worth noticing in that derivation before the multipliers. The first is that 350 output tokens cost almost exactly as much as 1,800 input tokens, because output is priced around five times higher. Any instinct to shorten the prompt before shortening the answer is attacking the smaller term.

The second is that runs_per_day is the input with the widest range and the least attention. Fourteen is a modest number for a team of six; a busy repository with a merge queue and a nightly can be at fifty, and the monthly figure scales linearly with it.

The multipliers estimates leave out

• The trigger matrix. Running on push and on pull_request without excluding branches means every commit on a branch with an open pull request runs the suite twice. Adding a merge queue makes it three times.
• The model matrix. Evaluating against three models multiplies everything by three. This is the single largest multiplier people add without recalculating.
• Reruns. A flaky assertion retried by the runner re-issues every model call in that job, not just the failing one, if the retry is at job level rather than at test level. A rerun rate of 15% is a factor of 1.15 at test level and can be far more at job level.
• Samples per case. Any quorum or majority-vote assertion multiplies its cases by n. Five samples on a quarter of the cases is a factor of 2 overall.
• Failed and abandoned runs. A job cancelled halfway has already paid for the calls it made. Cancellation refunds nothing.

Applying a realistic combination to the worked figure — three models, a 1.15 rerun factor, nothing else changed — takes $393.62 to roughly $1,358 a month. That is the number to put in front of whoever approves it, not the $394.

One term deliberately absent from the multipliers: developer machines. A suite that runs locally against the live API is billed identically to one that runs in CI, and it is invisible in any per-pipeline accounting. If the same key is in everyone’s environment, add the headcount as a multiplier on some fraction of the run count, or issue separate keys so the two can be told apart.

An LLM judge roughly doubles it

A rubric-scored eval calls a second model to grade the first, and the judge’s prompt is usually longer than the case prompt because it contains the rubric, the input and the candidate answer. Continuing the same assumptions, with a judge prompt of 2,200 input tokens returning 120 output tokens at the same placeholder prices:

Judge, per case
  input    2,200 / 1,000,000 x $3.00  = $0.00660
  output     120 / 1,000,000 x $15.00 = $0.00180
  total                                = $0.00840

Combined per case   $0.01065 + $0.00840 = $0.01905
Per run             120 x $0.01905      = $2.286
Per month           14 x 22 x $2.286     = $704.09

Judging costs 79% of what generating cost, on these assumptions. That is not an argument against a judge; it is an argument for deciding consciously whether every case needs one, and for judging on a schedule rather than on every commit. It is also a reason to pin the judge model separately, since a change to it invalidates the entire history of scores you paid for.

Which term to attack first

The formula tells you where the leverage is, and it is rarely where people look. The instinct is to cut cases, which is the term with the most coverage attached to it and usually not the largest multiplier. In order of typical effect:

• runs_per_day. Moving the eval from every commit to a nightly plus a pre-merge gate is often a factor of ten, and it is a configuration change rather than an engineering project.
• cases, on the blocking tier only. Not by deleting coverage — by moving most cases to the scheduled tier and keeping a small blocking subset, as in the two-tier split.
• in_tokens, via caching. A shared system prefix across cases is exactly the shape provider prompt caching rewards. The cache arithmetic works this through.
• price, via the batch endpoint. A published 50% discount for a 24-hour turnaround costs nothing on a nightly job. See batching eval calls in CI.
• out_tokens. Asking for a label rather than a paragraph attacks the expensive half directly, and for contract tests the paragraph was never the assertion.