Skip to content
Multigrid

Determinism and the cost of testing

Temperature, seeds, fixtures and tiers — making tests repeatable, and not paying twice for the same assertion.

A test suite that calls a language model breaks two assumptions that every other test suite gets for free. The first is that the same input produces the same output, which is not true here even at temperature 0 and even with a seed — providers document best-effort sampling, not reproducibility. The second is that running the suite is free. Neither assumption fails loudly. They fail as a build that goes red on an unrelated pull request, and as an invoice that is twenty times what it was last month.

These pages treat the two problems as one problem, because the fixes overlap: the decision about which tests genuinely need a reproducible answer is also the decision about which tests are allowed to spend money. What follows is the parameter-level detail (temperature, seed, pinned model strings, system_fingerprint), the assertions that survive a non-deterministic answer, and the arithmetic that tells you what a suite costs before you turn it on rather than after.

Setting Temperature to Zero for Reproducible Tests

Where to set temperature so every test inherits it, which layers quietly default it elsewhere, and the precise thing it buys a suite.

9 min read

Using the seed Parameter in Tests Against an OpenAI-Compatible API

Which APIs document a seed, exactly what each one promises, and how to prove your endpoint honours it rather than ignoring it.

9 min read

What to Do When a Provider Doesn't Support the seed Parameter

How to tell a rejected seed from a silently ignored one, and the four fallbacks in the order they are worth trying.

9 min read

Pinning Model Version in Tests to Avoid Silent Drift

Why an alias makes your suite test a different model next month, where to put the pin, and the assertion that proves it took.

9 min read

Combining Low Temperature With Property Assertions for Stable Tests

Why narrowing the output distribution and narrowing what you demand of it solve different halves of the same problem.

9 min read

Testing Against a Fixed system_fingerprint to Catch Silent Model Changes

What the field identifies, why asserting equality on it in pull-request CI is the wrong use, and where it belongs instead.

9 min read

Deciding Which Tests Need Determinism and Which Don't

Five classes of model test, what each one is really asserting, and why forcing determinism on two of them hides the bug.

9 min read

Estimating the Monthly Bill From Your CI Eval Suite Before You Turn It On

The arithmetic from case count, token counts, trigger frequency and a stated per-token price, with the multipliers most estimates leave out.

10 min read

Using a Cheap Model in CI and the Expensive One in Production

What transfers between model tiers, what does not, and the two opposite failure modes a tier substitution creates.

9 min read

Setting a Hard Spend Cap on a CI Pipeline That Calls an LLM

Five layers of guard, the in-process token counter that aborts the job, and the reason streaming hides the number you need.

10 min read

Recording Fixtures Once and Never Paying for the Same Test Twice

The amortisation arithmetic for recorded responses, what a recording can and cannot prove, and the maintenance cost that makes the saving honest.

9 min read

Why Your CI Bill Jumped After Adding a Prompt Test Suite

Attributing the spend first, then the six causes in the order they occur, then the enforcement that stops it recurring.

10 min read

Splitting Tests Into a Mocked Tier and a Paid Live Tier

What each tier can prove that the other cannot, how to express the split so it cannot rot, and where a live failure has to go.

10 min read

Batching Eval Calls in CI to Reduce Per-Request Overhead

Where a 24-hour batch endpoint fits in a pipeline, why custom_id is the whole design, and the partial-failure trap that silently passes.

10 min read

Reusing Prompt Caching to Cut the Cost of a Regression Suite

Why a regression suite is the ideal shape for provider prompt caching, the arithmetic with published multipliers, and four ways a suite destroys its own cache.

10 min read

Setting Per-PR Token Budgets So One Branch Can't Blow the CI Bill

Accounting that spans every run on a pull request, why the counter has to live outside the job, and how to stop a rerun double-counting.

10 min read

Other topics