What a Full Regression Suite Costs to Run on Every Commit

“Run the evals on every commit” is a reasonable-sounding policy that quietly multiplies your inference bill by the number of commits your team pushes. Here is the arithmetic, with every input named, so you can substitute your own numbers and find out whether your version of the policy is affordable before you adopt it.

The inputs, all assumptions

None of these are measurements. They are a plausible mid-size suite, chosen so the arithmetic is legible, and every one of them is a knob you will replace with your own figure:

• 240 cases in the suite. Assumption.
• 2,400 input tokens per case — assumed as a 1,800-token shared prefix (system prompt, tool schemas, few-shot examples) plus a 600-token case-specific tail.
• 400 output tokens per case. Assumption.
• 3 models per run — you gate on the model you serve plus two you would fail over to. Assumption.
• 45 pushed commits per working day across the repository, and 21 working days per month. Assumption.
• $3.00 per million input tokens and $15.00 per million output tokens — an assumed price, used only as a placeholder so the arithmetic has numbers in it. Substitute the current figure for the models you actually gate on from Anthropic’s pricing page or OpenAI’s pricing page.

Cost of one case

Input   2,400 tokens  x  $3.00 / 1,000,000  =  $0.00720
Output    400 tokens  x $15.00 / 1,000,000  =  $0.00600
                                    per case =  $0.01320

Per full run:  240 cases x 3 models x $0.01320  =  $9.504

Two things are already visible. Output is 14% of the tokens and 45% of the cost, because output is priced five times higher under the assumed prices — so a suite whose cases ask for long answers costs far more than its token count suggests, and capping max_tokens on eval cases is a real saving rather than a tidiness measure. And the model count multiplies everything linearly: gating on three models rather than one is a 3× bill for a benefit that is usually concentrated in a handful of cases.

The cadence multiplier

Now the part the row exists for. Per-commit and per-day are the same suite and differ by a factor equal to your commit rate.

Per commit   45 commits/day x $9.504   =  $427.68 / day
             21 days x $427.68        =  $8,981.28 / month

Nightly      21 runs x $9.504         =    $199.58 / month

Ratio                                 =  45x

The gap is not 45× because per-commit runs are wasteful in some subtle way. It is 45× because you are running the suite 945 times a month instead of 21. That is the whole finding, and it is worth stating in those terms when somebody proposes the policy, because the discussion usually happens in units of “is nine dollars a lot?” when the deciding number is the commit rate.

Two adjustments make the per-commit figure less bad and are worth folding in before you compare. Most teams gate on pull-request head commits rather than every pushed commit, which cuts the count to roughly the number of pull requests plus their updates — if that is 12 a day rather than 45, the monthly figure is 12 × $9.504 × 21 = $2,395.01. And CI concurrency cancellation, where a new push to the same branch cancels the in-flight run, removes the duplicate work from someone pushing four times in ten minutes.

What a shared prefix removes

The 1,800-token prefix is identical across all 240 cases, which is exactly the situation prompt caching exists for. Under Anthropic’s documented model, a cache breakpoint is marked with a cache_control block, cache reads and writes are reported separately in the response as cache_read_input_tokens and cache_creation_input_tokens, and the default cache lifetime is five minutes with a one-hour option available. Assume a cache read is charged at one tenth of the base input price — a further assumption, and one to check against the prompt-caching documentation for your model:

Cached prefix  1,800 tokens x $0.30 / 1,000,000  =  $0.00054
Fresh tail       600 tokens x $3.00 / 1,000,000  =  $0.00180
Output           400 tokens x $15.00 / 1,000,000 =  $0.00600
                                       per case  =  $0.00834

Per run    240 x 3 x $0.00834  =  $6.005
Per month  45 x 21 x $6.005    =  $5,674.53   (was $8,981.28)

A 37% reduction, and it depends entirely on the whole run finishing inside the cache lifetime and on the prefix being byte-identical across cases. It also has a failure mode specific to this use: a suite that shuffles case order or interleaves models can leave gaps longer than the five-minute window, at which point you pay the cache-write premium repeatedly and the total goes up. Assert on cache_read_input_tokens in the run summary rather than assuming the discount arrived.

Which lever to pull

Ranked by how much they remove per unit of effort, under the assumptions above:

• Change the cadence. Full suite nightly and on the release branch, a fast subset per commit. If the subset is the 30 cases that touch the changed prompt, the per-commit figure falls to 30/240 of $8,981.28, or about $1,123 — a 87.5% cut with no loss of coverage on the merge path.
• Drop to one model per commit. Two thirds of the per-commit cost is testing failover models on a change that usually does not touch routing. Keep the full matrix nightly.
• Cap output. 45% of the per-case cost under these prices. Most eval cases are graded on a classification, a schema or a tool call, none of which need 400 tokens.
• Cache the prefix — 37% here, and free once configured, but only within the TTL.
• Watch the judge. If cases are graded by another model call, this whole page is roughly half your real bill. Count judge tokens the same way and add them.

Whatever you settle on, put the estimate in the run itself: sum the reported input, cached and output tokens per run and print a cost line with the price constant that produced it. That makes drift visible on the day the suite grows rather than on the invoice instead of a month later, and it gives a threshold alert something to fire on. The other cost line that behaves this way is the one nobody budgets for at all — retries, which are billed as ordinary requests and are invisible in a per-case estimate. Read what retries cost alongside this, and add the retry multiplier to the per-run figure above rather than treating it as noise.