Cold Starts on Serverless Inference

A cold start on a serverless function is a few hundred milliseconds of runtime initialisation. A cold start on serverless inference is dominated by something else entirely: moving tens of gigabytes of weights into GPU memory. That is a bandwidth problem, and bandwidth problems can be estimated rather than guessed at.

What a cold start is made of

When a request arrives and no worker is warm, the sequence is roughly:

• Schedule a GPU. Find capacity, which on contended accelerators is itself variable and is the one term with no upper bound.
• Pull the image. CUDA, the framework and the runtime. Frequently several gigabytes; cached on the node if you are lucky.
• Fetch the weights. From object storage or a distributed cache to local disk, unless already present.
• Load weights to VRAM. Disk or page cache to host memory to device memory, over PCIe.
• Initialise and warm. Allocate the KV pool, compile or select kernels, run a warm-up pass. Compilation-based stacks pay a substantial one-off here.
• Then prefill your prompt — the only part that would have happened anyway.

Note how many of those are byte-movement steps and how few are computation. That is the structural difference from an ordinary serverless cold start, where the dominant term is a language runtime initialising. Here the runtime is a rounding error against tens of gigabytes of weights, which means the usual advice — trim your dependencies, lazy-import, keep the bundle small — has almost no purchase. The lever is distance to the weights, not the size of the code.

The arithmetic, with the assumptions labelled

Nobody has timed this for you and no figure below is a measurement. What follows is a lower bound implied by byte counts and stated bandwidths — useful precisely because it shows that the floor is already high.

ASSUMPTIONS (change them for your own estimate)
  model            8B params, BF16          -> 16 GB of weights
  object storage   sustained 500 MB/s       -> assumption
  node-local NVMe  sustained 2 GB/s         -> assumption
  PCIe Gen4 x16    ~25 GB/s practical       -> ~64 GB/s theoretical, derated

COLD, weights not on the node
  fetch to disk    16 GB / 0.5 GB/s   =  32 s
  disk -> VRAM     16 GB / 2 GB/s     =   8 s        (disk-bound, not PCIe)
  init + warm-up                      =   2-20 s     (stack-dependent)
                                        ---------
                                        ~42-60 s

WARM NODE, weights in page cache
  RAM -> VRAM      16 GB / 25 GB/s    = 0.6 s
  init + warm-up                      = 2-20 s
                                        ---------
                                        ~3-21 s

Same arithmetic, 70B in BF16 (140 GB): multiply the transfer terms by 8.75.

Two conclusions survive any reasonable change to those assumptions. First, the dominant term is wherever the weights have to come from, so the entire engineering effort in this area is about making that distance shorter — node-local caches, peer-to-peer distribution between workers, streaming the first layers so compute can start before the last ones land. Second, on a cold node the load time is comfortably longer than any sane request timeout, which is why cold starts present to callers as errors rather than as slowness.

Why serverless GPUs scale to zero at all

It is worth being clear that this is a deliberate trade and not a deficiency. An accelerator that is idle still costs its owner the full hourly rate. For bursty or long-tail workloads — a model called a few hundred times a day, or one of thirty fine-tunes each serving one customer — keeping a warm replica per model is the dominant cost and scale-to-zero is what makes the offering economically possible at all. You are buying a lower floor price with a worse cold percentile. That is a reasonable deal for a nightly job and a bad one for a chat interface, and the mistake is only ever in the matching.

Four ways to hide it

A fifth, which is not really hiding: route the request somewhere warm. If more than one endpoint can serve the model, a cold start becomes a routing decision rather than a user-visible wait.

There is also a way to sidestep the problem entirely when the thing you are serving is a fine-tune. Low-rank adapters are small — megabytes against the base model’s tens of gigabytes — so a server that keeps one base model resident and swaps adapters per request turns “load a model” into “load a few megabytes”. Run the same arithmetic on a 40 MB adapter and the transfer terms collapse to milliseconds. This is why hosted fine-tuning offerings can serve hundreds of customer-specific variants without a warm replica for each, and why a full-weight fine-tune and an adapter have completely different cold-start economics despite behaving identically at the API.

What the caller can do

• Separate your timeouts. A connection that has been accepted but has produced no tokens for forty seconds is very probably loading a model, not broken. A single overall deadline cannot distinguish those; a connect timeout plus a generous first-token timeout plus a tight inter-token stall timeout can. See choosing a timeout.
• Retry a cold-start failure differently. It is not a capacity error and not a bug; the second attempt usually lands on the now-warm worker. One immediate retry is often more effective than exponential backoff here — but only if the first request is genuinely dead, or you have paid twice.
• Warm on a schedule if the pattern is predictable. A cheap synthetic request before the working day starts converts a user-visible cold start into a machine-visible one.
• Segregate your percentiles. Cold starts create a second mode in the latency distribution, and a p99 computed across both modes describes neither. Tag requests by whether they hit a warm path.

The number to actually track is the cold rate — what fraction of requests hit an unwarmed worker — rather than the cold penalty, which you cannot change. The cold rate is a function of your traffic shape against the provider’s idle timeout, and it moves for reasons you can see: a change in traffic distribution across the day, a new region, a deploy that reset every warm worker at once. If the provider does not label cold responses, a first-token latency above a few multiples of your warm p99 is a serviceable proxy, and a rising cold rate is a much earlier signal of a capacity or configuration change than a rising p99 would be.