Caching Eval Results Between CI Runs to Save Cost

A pull request that edits one prompt of nine re-scores all nine, and pays for all nine, every push. Almost all of that spend is buying an answer somebody already has. The fix is a result cache, and the whole difficulty of a result cache is the key.

Cache the case, not the run

The unit has to be one model call for one case. Caching a whole run keyed on the commit gives you a hit rate of zero, because the commit changes every push. Caching per case means a push that touched one prompt pays for the cases that use that prompt and reuses the rest, and the saving scales with how narrow your changes are — which for prompt work is very narrow indeed.

Store the model’s raw response, not the score. Scoring is cheap and local, and scorers change more often than prompts do; if you cache the score, then improving a rubric invalidates everything, whereas caching the response lets a rubric change be re-scored for free against responses you already paid for. That single choice is usually worth more than the cache itself.

What goes in the key

The key is a hash over everything that could change the response. Miss one component and the cache returns a stale answer that looks authoritative, which is worse than no cache, because a wrong cache hit silently reports a passing score for a prompt that was never run.

• The rendered prompt, not the template. Hash the final message array after variable substitution. A template with a changed variable produces a different request, and hashing the template alone misses it.
• The model identifier, at the most specific version you can get. A floating alias points at different weights over time, so a key containing only the alias will happily serve last month’s model as this month’s. Where the provider returns a resolved version in the response, key on that.
• The decoding parameters. Temperature, top-p, max_tokens, seed, stop sequences. Changing temperature changes the distribution being sampled, so a cached response from a different temperature is a different experiment.
• The tool schemas, if any are sent. Tool definitions are part of the request and they change the output. A renamed tool or a reordered parameter list belongs in the hash, and this is the component people forget most often.
• A manual cache version prefix. One string you bump by hand when you need to discard everything — after a scorer bug, or when you no longer trust what is in there.

import hashlib, json

CACHE_VERSION = "v3"

def cache_key(request: dict) -> str:
    material = {
        "messages": request["messages"],
        "model": request["model"],
        "tools": request.get("tools"),
        "params": {
            k: request.get(k)
            for k in ("temperature", "top_p", "max_tokens", "seed", "stop")
        },
    }
    blob = json.dumps(material, sort_keys=True, separators=(",", ":"))
    digest = hashlib.sha256(blob.encode("utf-8")).hexdigest()
    return f"{CACHE_VERSION}-{digest}"

sort_keys=True matters more than it looks: without it, two identical requests whose dictionaries were built in a different order hash differently and never hit. The same argument applies to floating point — a temperature written as 0.0 in one place and 0 in another must normalise to one form before hashing.

The store

Keep it dull. A directory where each file is named for the key and contains the response JSON plus the token usage is enough, and it makes the cache inspectable, which you will want the first time a hit looks wrong. Writing the usage alongside the response also lets you report what the run would have cost without the cache, which is the number that justifies keeping it.

Two operational rules. Never write a cache entry for a failed or truncated response — a cached response with a finish_reason of length will be replayed forever as a genuine result. And expire on age as well as on key, because a cache entry for a floating model alias is a claim about a model that may have been replaced since; a maximum age of a couple of weeks bounds how wrong that can get.

Why CI caches miss

The cache directory then has to survive between runs, and this is where most implementations quietly stop working. On GitHub Actions, cache entries are immutable: once a key is written it cannot be overwritten, so a key that includes anything run-specific gets a fresh miss every time and saves a new entry that is never read. The pattern that works is a stable prefix plus a varying suffix in the save key, with restore-keys naming the prefix so a restore falls back to the most recent matching entry.

      - uses: actions/cache@v4
        with:
          path: .eval-cache
          key: eval-cache-v3-${{ github.sha }}
          restore-keys: |
            eval-cache-v3-

The second rule to know is branch scoping. A cache saved on a branch is visible to that branch and to branches created from it, but not to unrelated sibling branches; caches from the default branch are visible everywhere. The practical consequence is that a cache built up only by pull request runs is far less useful than it appears, because each new branch starts nearly cold. Populate the cache from a scheduled or post-merge run on the default branch and every branch inherits it.

Other products express the same idea differently — CircleCI uses an ordered list of keys in restore_cache with prefix matching, and Buildkite’s cache plugin takes a restore level, as shown in the Buildkite eval gate. The common mechanism is prefix fallback; only the spelling differs.

Caching changes what the eval measures

Be clear-eyed about this. A cached suite measures “the responses we once got”, not “the responses this model gives today”. At temperature 0 those are close and the substitution is reasonable. Above 0 they are not: the point of sampling is that repeated calls differ, and a cache freezes one draw from the distribution and reports it as the answer forever.

The workable arrangement is to cache aggressively on pull requests, where the question is “did my change break something” and a fixed comparison point is actually an advantage, and to run the full suite uncached on a schedule, where the question is “is the model still behaving”. That division falls out naturally from the cadence argument in running evals nightly instead of on every commit, and it means the cache never hides a provider-side change for longer than a day.