Model Weights in CI/CD
11 min read · updated August 4, 2026
Weights are a build artefact that happens to be a thousand times larger than your application. Everything unusual about deploying them follows from the size: they cannot live in the repository, should not live in the image, must be referenced by something immutable, and have to be on the machine before the machine is asked to serve.
Why not git, and why not the image
Git stores whole objects, and weights files are binary blobs that change entirely between versions. Ten checkpoints of a 40 GB model is 400 GB of history that every clone pays for. Large-file extensions make this survivable rather than good — they move the blobs to a separate store while leaving you with git’s semantics for something that has no diff.
The container image is the more tempting mistake, and containerising an AI service lists the four reasons it goes wrong. The shortest version: it couples the release cadence of your code, which changes hourly, to an artefact that changes monthly, and it puts the entire transfer on the critical path of every scale-up.
Object storage is the right home. It is cheap, it handles multi-gigabyte objects, it supports ranged and parallel reads, and it has a permissions model that can be narrower than your image registry’s.
Content addressing and immutable releases
A path like s3://models/my-model/latest/ is a mutable pointer and it will eventually serve two different sets of bytes to two replicas of the same deployment. That failure is nearly impossible to diagnose from the outside: half your traffic gets one model, half another, and every trace looks normal.
So make the object path immutable and put the mutability in a separately versioned pointer.
# Immutable object layout — the digest is over the manifest, not one file.
s3://models/my-model/sha256-3f9a1c.../
manifest.json # names, sizes and per-file sha256
config.json
tokenizer.json
model-00001-of-00008.safetensors
...
# The mutable part, versioned and reviewable, lives in your config repo:
# deploy/models.yaml
serving:
chat-default:
digest: sha256-3f9a1c...
uri: s3://models/my-model/sha256-3f9a1c.../
params: 8030261248
files: 8
bytes: 16060522496Now “which model is production running?” is answered by a git log rather than by an object-storage timestamp, a rollback is a revert, and a digest mismatch at load time is a loud failure instead of a quiet one. Verify the digest on the node after download and refuse to start on a mismatch — that check has caught truncated transfers, partially replicated buckets and, once in a while, the wrong model entirely.
The release pipeline
Weights deserve the same pipeline discipline as code, with two extra stages: the eval gate and the format conversion.
- Produce. A training or fine-tuning job writes a checkpoint to a staging prefix, along with the run metadata — code commit, dataset version, hyperparameters.
- Convert and verify. Convert to the serving format your runtime wants, then load it once in a test container and run a handful of prompts. A file that loads is not the same as a file that answers; check both.
- Evaluate. Run your eval set. This is the gate: below threshold, the artefact never gets a digest in the config repo. See building an eval harness for what goes in the set and evaluating a fine-tuned model for the comparison against the base.
- Publish. Compute the manifest digest, copy to the immutable prefix, and make the objects write-once if your storage supports it.
- Propose. Open a pull request that changes one line in
models.yaml. That PR is the deploy, and it is reviewable by someone who was not in the training run. - Warm, then shift traffic. The next two sections.
Getting warm before traffic
There are three places weights can be when a request arrives, and they differ by orders of magnitude in how long they take to become useful: in device memory, in the node’s page cache or local disk, or in object storage. The whole game is moving the boundary earlier.
| Technique | Description |
|---|---|
| Node-local cache volume | A host path or local NVMe directory shared by every pod on the node, keyed by digest. The second pod on a node loads from disk instead of the network. Simple, and the single biggest win. |
| Pre-pull DaemonSet | A low-priority pod on every GPU node that downloads the digests named in models.yaml into the cache volume. It runs when the config merges, not when traffic arrives, so the download happens on someone else's schedule rather than a user's. |
| Init container | Downloads into the cache if absent, verifies the digest, and exits. Guarantees correctness before the server container starts, and is a no-op when the pre-puller already did the work. |
| Parallel ranged download | Object storage saturates a link far better with several concurrent ranged reads than with one sequential stream. Most CLIs do this by default; check the concurrency setting, because the default is often tuned for small objects. |
| Memory-mapped load | Formats designed for zero-copy loading let the OS map the file rather than read it into a buffer first. Combined with a warm page cache this makes the second load on a node dramatically faster than the first. |
| Synthetic warm-up requests | After load, send a few requests through the real path before passing readiness. They compile kernels, allocate the KV cache and touch the code paths a real request will. Without this the first user pays for all of it. |
Cold-start arithmetic
Put numbers on it, so warming is a budget rather than a hope.
Weights size, derived rather than looked up:
bytes = parameters × bytes_per_parameter
8.03e9 params × 2 bytes (16-bit) = 16.1 GB
70e9 params × 2 bytes = 140 GB
70e9 params × 0.5 bytes (4-bit) = 35 GB
Transfer time = bytes ÷ effective throughput. Label the throughput; it is the
number that differs most between environments.
16.1 GB from object storage at 250 MB/s ....... 64 s
16.1 GB from object storage at 1.2 GB/s ....... 13 s (parallel ranged reads,
fast networking)
16.1 GB from node-local NVMe at 3 GB/s ........ 5 s
16.1 GB already in page cache ................. ~0 s
Then add, for a realistic total:
+ host-to-device copy, bounded by the PCIe or interconnect link
+ runtime init and kernel compilation (tens of seconds, once per process)
+ warm-up requests (a few seconds)That table is the argument for the node-local cache in one line: it is the difference between a minute and five seconds, repeated on every scale-up event for the life of the service. And it feeds directly into the headroom calculation in GPU autoscaling, where time-to-ready sets how many replicas you must keep running idle.
Rollback, and the pointer that makes it fast
Because the config names a digest, rollback is reverting one line. But it is only fast if the previous digest is still on the nodes. So set a cache retention rule: keep the current and previous digests on every node, evict older ones by least-recently-used. That single rule turns a rollback from a 60-second-per-node download into a process restart.
# Cache pruning, run by the pre-puller after each config change.
# Keeps exactly the digests named in models.yaml plus the previous release.
KEEP=$(yq '.serving[].digest' deploy/models.yaml; git show HEAD~1:deploy/models.yaml | yq '.serving[].digest')
for d in /var/lib/model-cache/*/; do
name=$(basename "$d")
echo "$KEEP" | grep -qx "$name" || { echo "pruning $name"; rm -rf "$d"; }
doneThe other half of a fast rollback is not having shifted all your traffic in the first place, which is what canary and blue-green deploys is about.
The same thing without Kubernetes
On a plain VM the structure is identical and systemd expresses it cleanly: one unit that fetches and verifies, one that serves, with an ordering dependency between them.
# /etc/systemd/system/model-fetch.service [Unit] Description=Fetch and verify model weights After=network-online.target Wants=network-online.target [Service] Type=oneshot RemainAfterExit=yes EnvironmentFile=/etc/model.env # MODEL_URI, MODEL_DIGEST, CACHE_DIR ExecStart=/usr/local/bin/fetch-model.sh TimeoutStartSec=1800 # /etc/systemd/system/model-server.service [Unit] Description=Inference server Requires=model-fetch.service After=model-fetch.service [Service] EnvironmentFile=/etc/model.env ExecStart=/opt/venv/bin/python -m server Restart=on-failure RestartSec=10 KillSignal=SIGTERM TimeoutStopSec=180 # let generations finish User=app [Install] WantedBy=multi-user.target
Requires plus After is the pair that matters: Requires means the server does not start if the fetch failed, After means it does not start until the fetch finished. Using only After gives you a server that starts happily on a failed download and then crash-loops with a file-not-found. And TimeoutStopSec is the systemd spelling of the grace period — the same three minutes, for the same reason.