Running Evals Nightly Instead of on Every Commit

Moving an eval suite from per-commit to nightly is usually proposed as a cost decision and decided as a vibe. It is worth ten minutes of arithmetic, because for a small suite the saving is negligible and for a large one it is the difference between a hobby and a line item.

The question is not cost alone

Three quantities decide this and they pull in different directions. The money per run, which scales with cases and tokens. The wall-clock time per run, which decides whether the gate can sit in a pull request at all — a check that takes forty minutes is not a gate, it is a delay people learn to start and walk away from. And the resolution of the signal: how many changes are between one run and the next, because that is how many candidates you have when it goes red.

Per-commit optimises the third at the expense of the first two. Nightly does the reverse. The reason this is not a straight preference is that the first two are continuous and the third is not: doubling the spend buys you a proportional amount of something, whereas going from one candidate commit to forty changes the nature of the investigation rather than its size.

Working the cost

Do it with your own numbers. The shape of the calculation, with every input named, using figures that are assumptions rather than measurements:

A small suite. Assume 120 cases, and assume each case sends 1,500 input tokens and receives 400 output tokens. That is 180,000 input and 48,000 output tokens per run. Assume a price of $0.40 per million input tokens and $1.60 per million output tokens. Input costs 0.18 × $0.40 = $0.072 and output costs 0.048 × $1.60 = $0.077, so a run is about $0.15. At an assumed 40 pushes a day, per-commit costs about $6 a day; nightly costs $0.15. The saving is real and it is six dollars, which is less than the time spent discussing it.

A large suite. Now assume 2,000 cases against a reasoning-heavy configuration at 6,000 input and 1,500 output tokens each: 12 million input and 3 million output tokens. Assume $3.00 per million input and $15.00 per million output. That is $36 + $45 = $81 a run. At the same 40 pushes a day, per-commit is roughly $3,240 a day and nightly is $81. That is the case where the cadence question is genuinely a budget question.

The wall-clock version of the same calculation is worth doing too. At an assumed 8 seconds per case with 10 concurrent requests, 2,000 cases take about 1,600 seconds, or 27 minutes. That number, not the cost, is often what actually rules out per-commit.

What a nightly failure costs to attribute

This is the part that gets left out. A per-commit failure names its cause: the change that turned it red is the change that turned it red. A nightly failure covering 40 merged commits gives you 40 candidates, and finding the culprit means re-running the suite on intermediate commits. A bisect over 40 candidates is about six probes, and at the $81 per run assumed above that is roughly $486 — six times the nightly saving of one day, spent in one investigation, plus a morning of somebody’s attention.

So the honest framing is that nightly does not remove the cost, it moves it from a predictable daily charge to an unpredictable per-incident one, and adds a lag between writing a regression and learning about it. Whether that trade is good depends almost entirely on how often the suite goes red. If it fails once a quarter, nightly is clearly right. If it fails twice a week, you are paying the bisect tax repeatedly and per-commit on a subset would have been cheaper.

The hybrid most teams end up with

1. A fast subset on every pull request — the 30 to 60 cases that cover the properties most likely to break, sized to run in two or three minutes. This is the merge gate.
2. The full suite nightly on the default branch, against a pinned model version, with the result stored as a time series rather than only as a pass or fail.
3. The full suite on demand, triggered by a label on a pull request, for changes that touch the prompt or the model configuration. Detecting those is changed-file detection for prompt tests.
4. The full suite before a release, gated on the same thresholds, so no deploy leaves on nightly evidence that is up to a day old.

The design work is entirely in choosing the subset, because a subset that misses the property that breaks turns the merge gate back into decoration. Pick cases that have historically caught something, cover each distinct output shape at least once, and include every case that covers a hard safety or redaction requirement regardless of speed.

Who owns a red nightly

A merge gate has an owner by construction: the person whose pull request is blocked. A nightly has none unless you assign one, and an unowned failing scheduled job is the most reliably ignored artefact in software. Within a month it is red for three unrelated reasons and nobody can say which of them is new.

Two things prevent that. Route the failure to a place with a name attached — a rotation, an issue assigned to a person, not a channel — and make the nightly’s red state block something, even if it is only the next release rather than the next merge. A signal that blocks nothing and belongs to nobody is not a test, it is a log line with a schedule.