Baking Model Weights Into an Image or Mounting Them at Runtime

The weights have to reach the GPU somehow. Either they are inside the image, in which case you pay for them when you build and when you pull, or they are outside it, in which case you pay when the container starts. Nothing makes the copy free; the question is only where you would rather it happened.

The two shapes

Baked means a COPY or a download inside the Dockerfile puts the checkpoint into a layer. The image is self-contained: given the tag, a node can produce a working container with no other dependency. It is also enormous, and its digest changes whenever the weights change.

Mounted means the image contains only code and the weights arrive at runtime — from object storage fetched by an init container, from a shared filesystem mounted as a volume, or from a persistent volume populated once and reused. The image stays small and the model becomes configuration.

There is a third shape people reach for and should not: downloading weights in the application’s own startup path from a public model hub. It makes every container start depend on a third party’s availability and rate limits, it usually needs a credential in the serving container, and it fails in the least observable place. If you fetch at runtime, fetch from your own bucket.

What baking costs

• Every weight change is a rebuild and a full redeploy. Swapping a checkpoint means a new image, a new tag and a rollout, which is a heavier operation than it sounds when the artefact is tens of gigabytes. If you are evaluating checkpoints daily, this is the wrong shape.
• The push is a real engineering problem. A single enormous layer stresses registry limits and any proxy in the path; that is the subject of fixing a push timeout on a large image, and it is the most common reason teams abandon baking.
• Registry storage multiplies. Each tag with different weights is a distinct set of blobs. Retain ten versions of a large model image and you are storing ten copies of the weights, so a lifecycle policy stops being optional.
• Layer order matters more than usual. Put the weights in their own layer, added before the code layer if the weights change less often. A code change should not invalidate the weights layer, and it will if they share an instruction.

What you buy is atomicity and reproducibility. The image digest pins code and weights together, so a rollback is one tag change and there is no combination of “old code, new weights” to reason about during an incident. For a model that changes monthly and serves production traffic, that is worth a great deal.

What mounting costs

• Start-up now has a network step that can fail. Every cold container copies gigabytes before it serves anything, and every failure mode of your object store — throttling, credential expiry, a partial read — is now a start-up failure. It needs retries, a timeout and an honest readiness probe so the pod is not sent traffic mid-download.
• Something must guarantee code and weights match. The image no longer pins them together. A tokeniser change shipped with code against a checkpoint that expects the old one produces garbage output rather than a crash. Version the weights path explicitly and fail loudly on mismatch; do not use a latest prefix.
• Shared volumes have their own limits. A read-write-many filesystem serving weights to many pods is a throughput bottleneck at exactly the moment you are scaling out, which is the moment every replica reads at once.

What you buy is a small image, fast rollouts of code, and the ability to change the served model without a build. If several services share one checkpoint, mounting also means storing it once rather than once per image.

Node cache behaviour decides most of it

This is the part that decides the argument in practice, and it is about how often a node sees a cold pull rather than about the sizes involved.

A container runtime caches image layers on the node. If your replicas land on long-lived nodes and roll out rarely, a baked image is pulled once per node per version and thereafter start-up reads from local disk — the fastest possible path, with no network in it at all. Baking wins clearly.

If nodes are ephemeral, the calculation inverts. On spot capacity, on a cluster that scales GPU nodes to zero between batches, or on any serverless container runtime, a new node has an empty cache and pulls the whole image before the container starts. Now the weights cross the network on every scale-out event, and they do it through the image pull path, which you cannot instrument or retry as precisely as your own fetch. Mounting from a store in the same region — or a volume pre-populated and attached read-only — is usually faster and always more debuggable.

So the question to answer first is not “how big is the checkpoint” but “how often does a node start cold”. On a spot GPU node pool the answer is “constantly”, and that alone settles it.

Choosing, and the hybrid

Bake when the model changes on a release cadence rather than a development one, when nodes are long-lived, when the deployment target may be air-gapped or otherwise cannot reach an object store, and when you want one digest to identify the whole serving artefact.

Mount when the model changes often, when nodes are ephemeral, when several services share a checkpoint, or when the same code must serve different models per tenant or per environment.

The hybrid is worth knowing about because it gets most of both. Bake a default checkpoint into the image so the container is self-sufficient and starts with a warm cache, and allow an environment variable to point at a mounted path that overrides it. Production runs the baked path; evaluation runs override it without a build. The cost is one branch in the loading code and a clear rule about which path is authoritative.