Caching Model Weights Between CI Runs

Re-downloading fifteen gigabytes of weights on every CI run is not just slow; on a metered GPU runner it is the largest single line in the job’s cost. The cache itself is three lines of YAML. Getting the key right is where this goes wrong.

Where the weights actually land

You cannot cache a directory you have not located. For anything using the Hugging Face Hub client — which is most of the ecosystem, including transformers, diffusers and sentence-transformers — Hugging Face documents the default cache as ~/.cache/huggingface/hub, overridable with HF_HUB_CACHE (a direct path to the cache) or HF_HOME (whose hub subdirectory becomes the cache). HF_HUB_CACHE takes precedence over HF_HOME. The Hugging Face cache guide is the reference.

The layout inside matters for a reason that bites later. Each repository becomes a models--<org>--<repo> directory, files are stored once in a content-addressed blobs/ directory, and snapshots/<revision>/ holds symlinks pointing into blobs/. Two revisions of a model that share a tokeniser share one blob. That is excellent for disk and mildly hazardous for a cache action, because whatever archives the directory has to preserve symlinks rather than dereference them — dereferencing turns a deduplicated tree back into its full expanded size.

Set the location explicitly in the job rather than relying on the default, because the default is relative to the home directory and the home directory inside a container is not the one on the runner host.

env:
  HF_HOME: ${{ github.workspace }}/.hf
  HF_HUB_DISABLE_PROGRESS_BARS: "1"

A key on the model revision

The instinct carried over from dependency caching is to key on a lockfile hash. That is correct for node_modules and wrong here, because the model is not in your lockfile. A repo pinned to a moving tag — main, or a bare model id with no revision — can change under a key that never changes, and you will serve stale weights indefinitely.

Resolve the revision first and key on that. The Hub API returns the commit sha for any repo and ref without downloading anything.

- name: Resolve model revision
  id: model
  run: |
    SHA=$(python - <<'PY'
    from huggingface_hub import HfApi
    print(HfApi().model_info("org/model-name", revision="main").sha)
    PY
    )
    echo "sha=$SHA" >> "$GITHUB_OUTPUT"

- name: Cache weights
  uses: actions/cache@v4
  with:
    path: ${{ github.workspace }}/.hf/hub
    key: hf-${{ runner.os }}-org-model-name-${{ steps.model.outputs.sha }}

The resolve step costs one HTTPS round trip. In exchange the key is exactly as specific as the artifact it names: a new upstream commit produces a new key and a cold miss, which is correct, and an unchanged upstream commit hits forever, which is also correct. If you already pin an immutable revision in code — and for anything reproducible you should — you can skip the resolve and interpolate the pin directly.

Why restore-keys is wrong here

Almost every cache tutorial adds restore-keys with a truncated prefix so a near-miss still restores something useful. For a package cache that is right: a partially-populated node_modules saves the installer most of its work, and the installer reconciles the rest against the lockfile.

A weights cache has no reconciler. A prefix restore hands you the previous revision’s blobs under a key that claims to be the current one, and because the Hub client verifies what it has against what it needs, the best case is that it silently downloads the delta anyway and you saved nothing. The worse case is a job that runs against last week’s weights and reports a score for them. If the exact revision is not cached, you want a clean download.

There is one prefix that is safe, and it is not a prefix of the revision: a separate cache entry for the tokeniser and config files, which are small, change rarely, and are genuinely reusable across revisions. Cache them under their own key if the download of the small files is measurably hurting you. Usually it is not.

Size limits and eviction

GitHub documents a default cache allowance of 10 GB per repository with a seven-day retention window on last access, evicting least-recently-used entries when a repository exceeds its limit. Since a November 2025 change, administrators can raise the size eviction limit and the retention limit above those defaults, with storage beyond the plan’s included allowance billed. GitHub’s dependency caching reference carries the current numbers.

Two consequences follow. A single model larger than the repository allowance will never cache — the save step fails and the job keeps working, silently, at full download cost every run, which is exactly the kind of failure that goes unnoticed for months. And a weekly-cadence pipeline will miss the seven-day retention window every time, so the cache pays for itself only on repositories with daily traffic. Check the cache list on the Actions tab after a week; if the entry is not there, the cache is not working.

When a cache is the wrong tool

Past roughly the repository allowance, the CI cache stops being the right mechanism and three alternatives take over.

• Bake the weights into the runner image. If the model changes on a release cadence rather than a commit cadence, putting it in the container image moves the download to image build time and gets you content-addressed layer reuse for free. The cost is a fat image and a rebuild whenever the model moves.
• Pull from your own object storage over a private path. An S3 bucket in the same region reached through a gateway VPC endpoint, or a GCS bucket in the same region, is faster than the public Hub and avoids egress charges on the pull. This is the usual answer for self-hosted runners.
• Keep a warm directory on a persistent runner. A long-lived self-hosted runner can simply keep HF_HOME on a mounted volume outside the workspace and never evict it. This is the fastest option and the least reproducible, so pair it with a job step that asserts the resolved revision matches what is on disk.

Whichever you pick, keep the revision assertion. The failure this page exists to prevent is not a slow job — it is a fast job that scored the wrong weights.