Deciding Whether a Cloud Run Model Service Needs Minimum Instances

“Set minimum instances to avoid cold starts” is advice without a number in it. The decision is arithmetic: what a held instance costs per month against how many of your requests a cold start actually delays, and for a model-calling service the second number is usually smaller than people assume.

The question, stated properly

Minimum instances buy one thing: the elimination of scale-from-zero latency for some fraction of requests. They do not make warm requests faster, they do not increase throughput, and they do not help at all with the latency of the model call itself — which for a service whose job is to call a model is most of the response time.

That last point reframes the whole decision and it is the thing this essay is really about. If a request spends 4 seconds waiting on a model and 3 seconds on a cold start, the cold start is a 43% increase on an already-slow request. If a request spends 40 milliseconds and 3 seconds on a cold start, it is a 75-fold increase. Cold starts hurt fast services far more than slow ones, and a model service is a slow service. The intuition people import from a low-latency API service does not transfer.

An upper bound on what an idle instance costs

Google documents that instances kept running by the minimum-instances feature do incur billing costs, and that under request-based billing idle instances are billed at a lower rate than active ones. The active rates are published; the reduced idle rate is what you should read off the pricing page for your region before finalising a number. So the honest derivation is an upper bound, computed at the active rate, which the true figure sits below.

Assumptions, all stated: a tier-1 region, one held instance with 1 vCPU and 1 GiB, a 30-day month, request-based billing, and the free tier already consumed by other traffic. Google’s Cloud Run pricing page lists $0.000024 per vCPU-second and $0.0000025 per GiB-second at tier-1 rates, with a free tier of 180,000 vCPU-seconds and 360,000 GiB-seconds per month.

seconds in a 30-day month = 30 x 24 x 3600 = 2,592,000

CPU     2,592,000 s x 1 vCPU  x $0.000024  = $62.21
Memory  2,592,000 s x 1 GiB   x $0.0000025 = $ 6.48
                                             -------
upper bound per held instance-month          $68.69

Charged at the reduced idle rate the real figure is lower.
This is a ceiling, not an estimate.

So: under seventy dollars a month per held instance, as a ceiling. That is a small number for a production service and a large one for seventeen internal tools that each hold an instance. Both of those are real situations and the arithmetic is the same; what differs is how many instances you are about to multiply it by.

How many requests a cold start actually touches

This is the side people skip, and it is the side that usually decides the answer. A cold start affects a request only if that request arrives when no instance is running. Once one is up it stays up for a while, and with concurrency above 1 it absorbs several simultaneous requests without starting another.

Work it through for your own traffic rather than in the abstract. Two patterns bracket most real services:

• Steady traffic. A service receiving a request every few seconds through the working day essentially never scales to zero during it. The cold starts are the first request of the morning and whatever happens at the edges of the day — a handful out of thousands. Minimum instances are buying you almost nothing here, and crucially the traffic itself is doing the job for free.
• Sparse, bursty traffic. A service used twenty times a day, in unpredictable bursts, scales to zero between every burst. A meaningful fraction of those twenty requests eat a cold start. Here minimum instances change the experience materially — and here the cost of holding an instance is being paid to serve twenty requests, which is roughly $3.40 per request at the ceiling above.

The uncomfortable observation is that the two cases point the same way for opposite reasons. Where traffic is steady, minimum instances are unnecessary. Where traffic is sparse, they are extremely expensive per affected request. The band where they are both necessary and proportionate is narrower than the advice suggests: enough traffic to make the per-request cost reasonable, spread unevenly enough that scale-to-zero still happens, and latency-sensitive enough that the seconds matter.

Before reaching for minimum instances, measure your own p99 startup time, because the whole calculation depends on it. A service that starts in 400 milliseconds does not have a cold start problem worth $68 a month. A service that loads weights for 90 seconds has a problem that minimum instances do not really solve either, because the instances still restart on deploy and on rebalancing — that service needs its startup fixed, along the lines in the container-failed-to-start page.

What you are buying is best-effort

Google’s minimum instances documentation is explicit that the setting is a best-effort target: the count can temporarily drop below your configured floor because of infrastructure rebalancing, application crashes, quota limits or regional capacity issues. Google recommends configuring at least three minimum instances where high availability is the goal.

Two things follow. A minimum of one is not a guarantee of one, so a design that assumes an instance is always warm is assuming something the platform does not promise. And “set it to 3 for availability” triples the number derived above — a ceiling of roughly $206 a month per service on the same assumptions — which is a different conversation from the one that started with “it’s only sixty-something dollars”.

There is a cheaper thing to try first, and it is genuinely underrated: raise concurrency. For a service that spends its time waiting on a model, going from concurrency 1 to concurrency 8 means one instance covers eight simultaneous requests, which both reduces how often you scale from zero and cuts the instance count you pay for under load. It costs one flag and no ongoing money. The full argument is concurrency for inference services on Cloud Run.

When the answer is yes

Set minimum instances when three things are all true. The service is on a human-interactive path where seconds are visible — not a batch job, not a webhook, not anything a queue sits in front of. Traffic genuinely goes to zero for stretches, which you can confirm from the instance-count metric rather than assume. And the startup is slow for a reason you cannot remove, such as weights that must load.

Set them temporarily, too, which is a use people forget: ahead of a launch, a demo, or a known traffic spike, minimum instances are a reasonable few days of insurance that you then turn off. It is a runtime setting, not an architectural commitment.

Otherwise, do not. The default of zero is right for most model services most of the time, and the money is better spent on the thing that actually dominates the response — the model call, its timeout, and what happens when it is slow. The version of this argument that goes deeper into what a cold start consists of is in minimum instances and cold starts.