Serverless GPU: What It Is Good For

Serverless GPU trades a higher price per second for not paying when idle. Whether that is a good trade is a single comparison: the cost of the idle hours you would have paid for against the cost of the cold starts you will now incur. Both sides are computable from your own traffic, and the answer is usually clear once they are on the same line.

What serverless GPU actually is

The name covers several products with a common shape: you supply a container or a model, the platform holds no instance for you while idle, and it starts one when a request arrives. Billing is by execution time, typically per second or finer, at a rate above the equivalent dedicated instance.

The cost you are buying away is idle time. The cost you are accepting is the cold start, which for a GPU workload is much larger than for a function-as-a-service CPU workload because it includes moving weights into device memory. The stages are the same ones derived in GPU autoscaling: scheduling, image, weight load, warm-up. Platforms differ enormously in how much of that they have optimised away — snapshotting memory state, keeping images resident, pre-loading popular weights — so the only figure worth using is the one you measure on the platform you are evaluating.

The utilisation break-even, derived

Compare a dedicated instance running continuously against serverless paying only for busy time.

Labelled assumptions

  P_d   dedicated price per GPU-hour, paid whether busy or not
  P_s   serverless price per GPU-hour of execution
  u     utilisation: the fraction of wall-clock time the GPU is busy
        serving requests

Cost over one hour:
  dedicated   = P_d
  serverless  = P_s × u

Serverless is cheaper when  P_s × u < P_d,  i.e.

  u* = P_d / P_s          ← the break-even utilisation

Worked, with a serverless rate 2.5× the dedicated rate:
  u* = 1 / 2.5 = 0.40

  Below 40% utilisation, serverless is cheaper.
  Above 40% utilisation, the dedicated instance is cheaper.

With a serverless premium of 4×:   u* = 25%
With a serverless premium of 1.5×: u* = 67%

The premium is the whole story: the higher it is, the lower the utilisation
at which you should switch back to dedicated capacity.

Compute your own u honestly, and note what it means: it is GPU-busy seconds divided by wall-clock seconds, over a full week including nights and weekends. Teams consistently overestimate it, because they think of the busy hours. A service handling 20,000 requests a day at 3 GPU-seconds each is using 60,000 GPU-seconds out of 86,400 — 69% — but the same volume concentrated in an eight-hour working day is 60,000 out of 28,800, which does not fit at all and means more than one instance during the day and zero utilisation for sixteen hours.

u = (requests_per_day × mean_gpu_seconds_per_request) / (86,400 × instances)

  20,000 req/day × 3 GPU-s = 60,000 GPU-s of work per day
  one instance             = 86,400 GPU-s available per day
  u = 0.69 if traffic were uniform across the day

  Concentrated into 8 hours (28,800 s), you need at least
  60,000 / 28,800 = 2.1 instances during those hours, and the daily
  average over 3 provisioned instances is 60,000 / 259,200 = 0.23.

That second figure — 23% — is the one to compare against u*, because it is
what you actually pay for with dedicated capacity that cannot scale to
zero overnight.

How often you pay the cold start

Now the other side. A serverless platform keeps an instance warm for some idle timeout after each request; a request arriving after the timeout pays the cold start. So cold-start frequency is a property of your inter-arrival distribution, not of your request volume.

Assume arrivals are Poisson at rate λ per second — a reasonable first
model for independent users, though bursty traffic will be worse.

Inter-arrival times are exponential, so the probability that a gap exceeds
the idle timeout T_idle is:

  P(gap > T_idle) = e^(−λ · T_idle)

Expected cold starts per day ≈ (arrivals per day) × e^(−λ · T_idle)

Worked, T_idle = 300 s (5 minutes):

  λ = 1/60  (one request per minute, 1,440/day)
    e^(−300/60) = e^−5 = 0.0067
    → 1,440 × 0.0067 ≈ 10 cold starts/day

  λ = 1/600 (one per 10 minutes, 144/day)
    e^(−300/600) = e^−0.5 = 0.607
    → 144 × 0.607 ≈ 87 cold starts/day — 60% of requests

  λ = 1/3600 (one per hour, 24/day)
    e^(−300/3600) = e^−0.083 = 0.92
    → 24 × 0.92 ≈ 22 cold starts/day — almost every request

Cost of those cold starts, with a 45 s cold start:

  10 cold starts × 45 s = 450 s/day of extra billed time (if billed) and,
  more importantly, 10 users per day waiting 45 s longer than the rest.

  87 cold starts × 45 s = 65 min/day, experienced by 60% of your users.

The pattern in those three lines is the useful result: cold starts are worst not at low volume nor at high volume, but at the volume where the mean gap is comparable to the idle timeout. Very frequent traffic keeps the instance warm; very sparse traffic means few users total. The uncomfortable middle is where a majority of requests are cold and there are enough of them for that to matter.

The two levers are therefore obvious. Increase T_idle if the platform lets you, which trades idle cost for hit rate — and note that setting it long enough is exactly re-provisioning the instance, returning you to the break-even in the previous section. Or reduce the cold start itself, which is where inference cold starts and weight warming apply.

Four workload shapes

A fifth shape is worth naming because it is common and is neither: an interactive product whose traffic collapses overnight. The best answer there is usually neither pure option — dedicated capacity sized to the daytime floor, with the peak and the overnight tail served by something that scales to zero. That hybrid is more work, and it is what most mature deployments converge on.

Making the cold start smaller

Both sides of the trade improve if the cold start shrinks, so it is worth attacking directly before concluding the arithmetic is against you. Decompose it first, because the four stages have completely different fixes and teams routinely optimise the smallest one.

1. Container image pull. Reduce it by shrinking the image — the layer arithmetic in containerising an AI service applies unchanged, and removing CUDA wheels you do not need is the usual large win. Some platforms cache images per account; if yours does, the first invocation after a deploy is the slow one and the rest are not, which changes what you should be measuring.
2. Weight download. The largest stage for most models, and the one platforms differ on most. Ask whether weights can be baked into a snapshot, mounted from a fast shared volume, or cached on the execution host. If the only option is downloading from object storage on every cold start, the arithmetic is much worse than the platform’s marketing implies and you should measure it yourself.
3. Model load and device transfer. Use a format that supports memory-mapped loading, and load in the smallest precision you can accept. Halving the bytes halves this stage and the one before it — inference quantization covers what the quality cost actually is.
4. Warm-up. Kernel compilation and first-allocation costs, paid once per process. Send a synthetic request during initialisation rather than letting the first real user pay for it, and check whether your runtime supports caching compiled kernels between processes.

Two structural options are worth knowing about. Some platforms can snapshot a process after initialisation and restore it, which collapses stages three and four to a restore of memory pages — where it is available, it changes the decision entirely. And a small always-warm instance in front of a scale-to-zero pool lets you answer immediately with a cheaper model while the large one starts, which converts a cold start from a wait into a degradation.

Measure the four stages separately by logging a timestamp at each boundary in your own container. A single “cold start took 52 seconds” figure tells you nothing about which of the four to attack, and the split is usually surprising.

What the arithmetic leaves out

Four things the two derivations above do not capture, each of which can change the decision.

• Concurrency behaviour per instance. A dedicated server batches concurrent requests, which is where most of a GPU server’s throughput comes from. If the serverless platform gives each request its own instance, you lose batching entirely, and per-request GPU seconds rise — so the utilisation figure you compared is not measuring the same work. Check whether the platform supports concurrent requests per instance before comparing prices at all.
• Available hardware. Serverless platforms offer a subset of accelerator types, and a model that needs a large-memory part may simply not be offerable. Check the list before the spreadsheet.
• Storage and transfer. Weights have to reach the instance. Some platforms charge for the cached artefact, some for the transfer, some for both. This is a fixed monthly cost that does not scale to zero even when compute does.
• Capacity guarantees under load. Scale-to-zero also means no reserved capacity. If the platform cannot give you fifty instances at your peak, you have exchanged a cost problem for an availability problem — and that risk is highest at exactly the moments your traffic spikes.

The way to settle it is to measure two numbers on the actual platform before committing: cold start time for your model, and per-request GPU seconds at your real concurrency. Put those into the formulas above with your own traffic and the answer falls out. Everything else in the decision is preference.