Setting a Per-Branch Budget for Eval Spend in CI

Eval spend is not large until it is, and the transition happens in a single afternoon on a single branch. The arithmetic that predicts it is four numbers you already have.

The unit cost of one run

Everything below is derived from inputs you should replace with your own; the numbers here are stated as suppositions so the method is visible, not as measurements of anything.

Suppose a suite of 200 cases, each sending about 1,200 input tokens (a system prompt, a few-shot block and the case input) and receiving about 300 output tokens. That is 240,000 input and 60,000 output tokens per full run. Suppose further that your provider charges $3.00 per million input tokens and $15.00 per million output tokens — a plausible shape for a mid-tier model, and a figure you must read off your provider’s current price page rather than from this page.

Inputs (all supposed, replace with your own):
  cases                200
  input tokens/case  1,200
  output tokens/case   300
  price in          $3.00 per 1M tokens
  price out        $15.00 per 1M tokens

Per full run:
  input   200 × 1,200 =   240,000 tok → 0.240 M × $3.00  = $0.72
  output  200 ×   300 =    60,000 tok → 0.060 M × $15.00 = $0.90
  total per run                                          = $1.62

One dollar sixty-two is why nobody budgets for this at the start. The number that matters is not the run, it is the run count, and that is set by developer behaviour rather than by anything in the suite.

Where the month actually goes

Suppose thirty pull requests a week, each receiving four pushes that trigger the gate. That is 120 runs a week, or $194 — a real number but not an alarming one. Now add a nightly full run at $1.62 a night, and it is barely visible.

The distribution is what bites. Suppose one branch where somebody is tuning a prompt and pushing forty times in a day, because that is what prompt tuning looks like when the eval suite is the feedback loop.

  40 pushes × $1.62                 = $64.80 in one day, one branch
  and if a judged metric is used, the judge model is a second call:
  200 cases × (900 in + 150 out) at the same supposed prices
    input   0.180 M × $3.00  = $0.54
    output  0.030 M × $15.00 = $0.45
  judged run total                  = $1.62 + $0.99 = $2.61
  40 pushes with judging            = $104.40 in one day, one branch

The judge is the term people leave out, and it roughly doubles the unit cost of every run. If the judge is a larger model than the system under test — a common and defensible choice — it dominates. Include it in the model before deciding the suite is cheap.

The second term people leave out is retries. A run that retries 15% of its cases twice is a run that costs about 30% more than the model above predicts, and retries rise exactly when things are going wrong. See bounding retries for why a total retry budget is a cost control as much as a correctness one.

Why the cap lives in the harness

The instinct is to cap spend at the provider. It cannot work at this granularity, for a structural reason: a provider key has no idea what a branch is. The finest control most providers expose is a limit per key or per project, and the enforcement is asynchronous — usage is aggregated and the block arrives after the spend, not during it.

So the cap has to be somewhere that knows both the branch and the running total, which means your own harness. Three layers, cheapest first:

• A per-run token budget. The harness counts tokens as it goes and aborts the run when it crosses the budget, reporting a distinct failure — “budget exceeded after 143 of 200 cases” — rather than a generic red. This alone stops the catastrophic case, which is not many runs but one run that loops.
• A per-branch daily counter. Keyed on branch name, stored wherever you keep eval history, incremented at the end of every run. When today’s total for this branch crosses the allowance, the gate degrades to the sampled tier rather than failing — failing teaches people to bypass the gate, degrading does not.
• A concurrency group per branch. concurrency: { group: evals-${{ github.ref }}, cancel-in-progress: true } means a rapid series of pushes runs the suite once rather than six times. For the forty-push branch this is the single highest-leverage line in the workflow, and it costs nothing.

Tiering the suite instead of capping it

A hard cap is a blunt instrument, and the better structure is to decide how much evidence each event deserves. The full suite on every push is over-provisioned; the full suite before a merge is not.

• Pre-commit: a smoke set of about ten cases, no judge, cheapest model that exercises the code path. Cost per run in the region of a few cents on the assumptions above. This is what a pre-commit hook can afford.
• Pull request push: a stratified sample — say 40 of the 200, chosen to cover every category rather than at random, plus every case that has failed in the last thirty runs. About a fifth of the full cost, and it catches the large regressions, which are the ones a fast loop needs to catch.
• Merge queue: the full suite, once per merge group. This is the tier that must not be sampled, because it is the last point at which the combination of changes is tested. Batching means it runs once for several pull requests rather than once each — see the merge queue page.
• Nightly: the full suite plus the expensive judged metrics plus any long-context cases you excluded elsewhere. Once a day is thirty runs a month, which is affordable at almost any unit cost.

Under this structure the forty-push branch costs roughly a fifth of what it did, the merge path is more thorough than it was, and nobody has been told they may not run the tests.

The costs that are not tokens

Two other lines belong in the model, and one of them is frequently the larger.

Runner minutes. An eval job is mostly idle, waiting on the network, so it is cheap in compute and expensive in wall clock — which is exactly the billing shape hosted runners charge for. Read the per-minute rate off your own plan and multiply by the run count; do not assume it is negligible on a private repository with a large matrix, because a matrix multiplies minutes even when it does not multiply tokens. Sharding, in particular, buys wall-clock time and buys nothing on either cost line.

The cost of a gate people route around. Harder to put a number on and it dominates everything above. A suite that takes twenty minutes and $2.61 per push will be disabled on some branch by somebody in a hurry, and the regression it would have caught costs more than a year of running it. Tier the suite so it stays fast enough to keep.