Setting Per-PR Token Budgets So One Branch Can't Blow the CI Bill

A per-run cap stops one job. It cannot see the pull request that triggered forty jobs over three days, each one obediently under the cap, together costing more than the rest of the month. Catching that needs a counter with a longer memory than a CI job has.

The gap a per-run cap leaves

The in-process spend cap is per process. It resets when the job starts, which is exactly right for the runaway-loop incident it exists to stop and exactly wrong for the pattern that costs more in aggregate: an ordinary branch, pushed repeatedly, with the eval suite running each time.

The shape is familiar. Someone is iterating on a prompt, so they push small changes and read the eval results. Each run is $1.28 and well under any sane per-run cap. Forty pushes across a few days is $51 on one branch, and three people doing that simultaneously is most of the month’s budget spent on iteration rather than on verification. Nothing here is misuse — it is the workflow the suite was built to support — which is why the answer is a budget with feedback rather than a prohibition.

A per-pull-request budget also gives the person iterating something a per-run cap cannot: a running total attributable to their own work. In practice that visibility changes behaviour more than the limit does.

Where the counter has to live

Outside the job, and it has to tolerate concurrent writers. Both constraints follow from the same fact: two jobs on the same pull request can run at once, so a read-modify-write from each of them will lose an update.

• A key-value store with an atomic increment. The straightforward answer — an atomic add returns the new total, so there is no race and no read step. Key it on repository plus pull request number, with an expiry so abandoned branches clean themselves up.
• A database row updated in a single statement. Same property, using an update that returns the new value. Right if you already have a database and want the history for reporting.
• The CI cache. Tempting, and wrong. Cache entries are read at job start and written at job end, so two concurrent jobs both read the same starting value and one overwrites the other. It fails precisely when the branch is busy, which is when the budget matters.
• A comment or label on the pull request. Workable for visibility, poor as the source of truth for the same read-modify-write reason, and it consumes API rate limit on every run.

Reruns must not double count

CI jobs are rerun constantly — a transient failure, a re-requested check, someone clicking the button. If a rerun adds its usage again, the budget is consumed by retries and the developer is blocked for a reason that has nothing to do with what they spent.

The fix is to make the increment idempotent by keying it on the attempt rather than accumulating blindly. GitHub Actions exposes a run identifier and an attempt number in the environment, and the pair is unique per execution; the same information exists under different names in other CI systems. Record usage against that key and sum the keys, rather than incrementing a single scalar, and a replayed report overwrites its own row instead of adding a second one.

# report_usage.py — run at the end of every job that calls a model
import os, requests

payload = {
    "repo":     os.environ["GITHUB_REPOSITORY"],
    "pr":       int(os.environ["PR_NUMBER"]),
    # (run_id, attempt) makes the write idempotent: a rerun replaces its
    # own row rather than adding a second one.
    "run_id":   os.environ["GITHUB_RUN_ID"],
    "attempt":  os.environ["GITHUB_RUN_ATTEMPT"],
    "tokens_in":  TOTALS.input_tokens,
    "tokens_out": TOTALS.output_tokens,
    "est_usd":    TOTALS.est_cost,
}

resp = requests.post(BUDGET_URL, json=payload, timeout=10)
resp.raise_for_status()
remaining = resp.json()["remaining_usd"]

print(f"PR spend: ${resp.json()['spent_usd']:.2f}, remaining ${remaining:.2f}")
if remaining < 0:
    raise SystemExit(f"per-PR budget exhausted for PR #{payload['pr']}")

Two details in that snippet earn their place. The report happens at the end of the job, so the budget is checked after the spend rather than before — which is the honest ordering, because the per-run cap is what prevents a single job overrunning and this mechanism is about the next job, not this one. And the failure is a process exit with a message naming the pull request, not an assertion, so it cannot be mistaken for a test result.

Wiring it up

1. Accumulate usage in-process during the run, using the same wrapper as the per-run cap. One meter, two consumers.
2. Derive the pull request number from the CI environment. On a pull-request event GitHub Actions gives it in the event payload, and the ref takes the form of a pull-request merge ref, so both routes exist; pick one and handle the case where neither is present, which is a push to the default branch.
3. Post the report at job end, unconditionally — including on failure, since a failed job still spent money. Put it in an always-run step, not after the test command.
4. Have the budget service return the new total and the remainder, and print both. The number in the log is most of the value.
5. Fail the job when the remainder goes negative, with a message that names the total, the cap and where to ask for more.
6. Post the running total as a pull-request comment that updates in place rather than appending. A thread of forty comments is worse than no comment.

Resetting, exempting, escalating

The mechanism is the easy half. The policy around it decides whether the check is still enabled in three months.

• Reset on merge or close. The budget belongs to the pull request, not the branch. Reopening should start from zero; expire keys after a few weeks so abandoned branches do not accumulate.
• Provide an exemption, with an owner. A label that raises the cap, applied by someone accountable. Without an exemption path, the first legitimate large change — a prompt rewrite that genuinely needs many eval runs — ends with the check being removed for everyone.
• Warn before you block. A message at 70% is what actually changes behaviour; a hard stop at 100% with no warning reads as an outage. The warning is also where you suggest running the mocked tier locally instead.
• Fail open on infrastructure error. If the budget service is unreachable, log loudly and continue. A budget tracker that can block every pipeline in the organisation when it has a bad day will be deleted after its first bad day.
• Report weekly by repository. The per-pull-request number is for the developer; the aggregate is what tells you whether the cap is set at a sensible level or is simply annoying everyone.