Setting Minimum Instances on Cloud Run to Avoid Cold Starts

Minimum instances is usually described as the fix for cold starts. It is the fix for one specific cold start — the one from zero — and on an inference service that is often the smaller half of the problem.

What a cold start is made of

Five things happen between a request arriving at an idle service and that request being answered, and they have very different durations:

• Scheduling. The platform finds capacity and starts an instance. Fast, and largely out of your hands.
• Image pull. Proportional to image size. A slim Python image is seconds; a CUDA base image with weights baked in can be many gigabytes.
• Process start. Interpreter startup, imports, framework initialisation. Importing a deep-learning stack is not free — it is frequently seconds on its own.
• Model load. Reading weights into memory or VRAM and initialising the accelerator. This is the dominant term for local inference and it is measured in tens of seconds to minutes.
• First-inference warmup. Kernel compilation, memory allocation, cache population. The first real request is slower than every subsequent one even after the service reports ready.

A service proxying a hosted API pays the first three and stops. A service holding its own weights pays all five, and the last two dominate so completely that optimising image size is not worth the effort until they are handled.

Service-level and revision-level floors

Google’s documentation distinguishes two settings that look identical from a distance. --min sets a service-level minimum, with the YAML annotation run.googleapis.com/minScale. --min-instances sets a revision-level minimum, with the annotation autoscaling.knative.dev/minScale.

# Revision-level: this revision keeps two instances warm.
gcloud run deploy inference --image=IMAGE --min-instances=2 --region=us-central1

# Service-level: the service keeps two warm, distributed across revisions.
gcloud run services update inference --min=2 --region=us-central1

The distinction bites during a rollout. A revision-level floor applies to each revision that has it, so a deployment that briefly runs two revisions with a floor of two is holding four warm instances and paying for four. A service-level floor is a property of the service and does not double. If you are splitting traffic between revisions regularly, the service-level setting is almost always the one you want.

What the floor does and does not remove

A minimum instance floor guarantees that some instances exist. It does not guarantee that the instance handling a given request is one of them.

Concretely: with a floor of two and a concurrency of four, the service absorbs eight simultaneous requests warm. The ninth causes a new instance to start, and that request waits through the entire five-stage cold start described above. Minimum instances removes scale-from-zero latency and nothing else; every scale-up event above the floor still pays full price. For a service whose traffic is spiky rather than merely intermittent, this is why setting a floor of one improves the median and leaves the p99 exactly where it was.

There is a second-order effect worth knowing. Warm instances are kept warm, but the platform does not promise which instances survive a deployment or an infrastructure event, so an instance can be replaced and the replacement pays the cold start. A floor makes cold starts rare on a healthy service; it does not make them impossible, and code that assumes the model is already loaded because the service has been up for a week will eventually be wrong.

Warm is also not the same as initialised. An idle instance under request-based billing has its CPU throttled between requests, so background work — a token refresh loop, a periodic cache warm, a metrics flush — does not reliably run while no request is in flight. Anything your service does on a timer either needs the CPU-always-allocated setting or needs to move out of the service entirely and into a job or a scheduler. Discovering this from a credential that expired on an instance that had been warm all night is a memorable afternoon.

Working the cost

The economics turn on billing mode, and Google’s documentation is explicit that the two behave differently. Under request-based billing, instances kept warm but idle are charged at a reduced idle rate. Under instance-based billing, the full rate applies for the whole lifetime of the instance regardless of whether it is doing anything.

That matters enormously here because GPU services on Cloud Run are instance-based only. A minimum of one instance on a GPU service is a decision to rent a GPU continuously — there is no idle discount to soften it. The arithmetic is unforgiving and it is also simple:

monthly_cost = min_instances
             * hourly_rate_for_the_configured_cpu_memory_gpu
             * 730                      # hours in an average month

requests_per_month_to_break_even =
      monthly_cost / value_of_one_avoided_cold_start

Fill in the hourly rate from Google’s current Cloud Run pricing for your exact CPU, memory and GPU configuration and region; it is not quoted here because it varies by all four and changes. The point of writing it as a formula is the second line: a warm floor is worth its cost only if enough requests would otherwise have hit a cold start to justify it. For an internal tool used twice an hour, that number is tiny and the honest answer is to accept the cold start. For a user-facing endpoint at sustained traffic, the floor costs less than the abandonment.

1. Measure how often scale-from-zero actually happens, from the instance count metric. Many services that feel cold are not scaling to zero at all.
2. Set the floor to the number of instances your quiet-period traffic needs at your configured concurrency, not to 1 by reflex.
3. Confirm your billing mode, because on GPU services the idle discount you may be assuming does not exist.
4. Re-check after any concurrency change; the two settings multiply, and changing one silently changes how much the other buys.

The levers that are not min-instances

• Startup CPU boost allocates extra CPU during instance startup, which shortens the import and load stages measurably on CPU-bound initialisation and costs nothing at steady state.
• Shrink the import graph. Loading a full training framework to run inference is common and expensive. An inference-only runtime cuts both image size and process start.
• Fetch weights concurrently rather than as one stream, which is Google’s documented recommendation for models above roughly 10 GB.
• Split the service in two. A single service that both proxies a hosted model and runs a local one has the worst cold start of the two applied to all of its traffic, and pays a warm floor sized for the expensive half. Two services let the cheap path scale to zero freely while only the expensive path carries a floor, and the routing between them is a few lines.
• Raise concurrency instead. If each instance can hold more work, traffic growth creates fewer instances and therefore fewer cold starts. This is often a bigger win than a floor and it costs nothing — within the limits of the workload.