Keeping a Vertex AI Endpoint Warm to Avoid Scale-to-Zero Latency

A Vertex AI endpoint with no replicas costs nothing and answers nothing. The setting that decides which of those you have is minReplicaCount, and it is worth understanding what a replica has to do before it can serve before deciding what to set it to.

What a replica start actually does

Deploying a model to an endpoint creates a deployed model with a DedicatedResources block naming a machine type, an optional accelerator, and replica bounds. Starting a replica means all of the following, in order, before it can accept a prediction: a machine of that type is provisioned; the serving container image is pulled; the model artefacts are fetched from Cloud Storage into the container; the framework loads them and builds its execution graph; and the container passes its health check.

For a small scikit-learn model that is fast enough to ignore. For anything with meaningful weights it is not, and the dominant term is usually the artefact fetch plus the framework load rather than the machine provisioning. A GPU replica adds driver and CUDA context setup and, for many serving stacks, a compilation or warm-up pass on first inference. None of that is charged to a request in a way you can see from the outside; it appears as a single very slow first prediction, or as a timeout.

That is the shape of the problem minReplicaCount addresses: not “make the load faster” but “do the load before anybody is waiting, and keep the result”.

Setting a minimum replica count

The flag is on the deploy, and the gcloud reference for ai endpoints deploy-model documents the meaning of each value: for normal deployments --min-replica-count must be at least 1, and a value of 0 enrolls the deployment in the scale-to-zero feature instead.

1. Upload the model, if it is not already in the registry, with gcloud ai models upload, naming the serving container image and the artifact URI.
2. Create the endpoint: gcloud ai endpoints create --region=europe-west4 --display-name=summariser.
3. Deploy with an explicit floor and ceiling:

   gcloud ai endpoints deploy-model ENDPOINT_ID \
     --region=europe-west4 \
     --model=MODEL_ID \
     --display-name=summariser-v3 \
     --machine-type=n1-standard-8 \
     --accelerator=type=nvidia-tesla-t4,count=1 \
     --min-replica-count=1 \
     --max-replica-count=4 \
     --traffic-split=0=100

4. Send one prediction and time it, then wait past your idle window and time another. With a minimum of 1 the two should be comparable; a large gap means something other than replica count is doing the loading, such as a lazily initialized model inside your own serving container.

The last point in that list is the one that catches people. A minimum replica keeps the container running. If the container defers loading the model until the first request reaches it, a warm replica is still cold in the only sense that matters. Load in the container’s startup path, and make the health check fail until the load has finished, so Vertex does not route to a replica that is not ready.

Autoscaling metrics above the minimum

Between the minimum and the maximum, Vertex scales on a target metric. The gcloud flag is --autoscaling-metric-specs and it accepts a key with a target, including cpu-usage, gpu-duty-cycle and request-counts-per-minute; more than one may be given at once, and the deployment scales to satisfy whichever is furthest from its target.

gcloud ai endpoints deploy-model ENDPOINT_ID \
  --region=europe-west4 \
  --model=MODEL_ID \
  --machine-type=n1-standard-8 \
  --min-replica-count=2 \
  --max-replica-count=8 \
  --autoscaling-metric-specs=gpu-duty-cycle=60,request-counts-per-minute=600

For a GPU deployment, CPU utilisation is close to meaningless as a scaling signal — the CPU is mostly waiting on the accelerator — so gpu-duty-cycle is the one to reason about. It has a known weakness with batched LLM serving: a server that batches requests can hold duty cycle high and steady while the queue grows, so the metric reports healthy utilisation during exactly the period when latency is degrading. Where that is the failure mode, scale on request rate as well and let the higher of the two win.

Set the minimum from the traffic floor you actually have, not from zero. If the endpoint receives a request every few minutes throughout the working day, a minimum of 1 costs one replica and removes the entire class of problem. If traffic is genuinely bursty around known times, raising the minimum on a schedule is cheaper than keeping the peak warm all day.

If you want scale to zero instead

The opposite end of the same setting exists, and it is the right choice for a development endpoint or a rarely used internal model where paying for an idle GPU is the larger problem. Google’s autoscaling documentation describes enabling it by specifying min_replica_count of 0 in the DedicatedResources of the deploy-model request, targeting the v1beta1 Vertex Prediction API.

Three documented constraints shape how it behaves. The Vertex AI autoscaling documentation states that when minReplicaCount is 0, initialReplicaCount must be greater than zero and no greater than maxReplicaCount; that scale to zero cannot be enabled on shared public endpoints, though other endpoint types are compatible; and that a ScaleToZeroSpec block carries a min_scaleup_period, the duration a model server must have been running before it is considered for scale-down at all, so a deployment does not attempt to drop to zero immediately after starting even with no traffic.

The honest framing is that this is the same trade as everywhere else in this cluster, with the numbers moved: an idle accelerator is expensive enough that scale to zero is attractive, and a cold model load on a GPU is slow enough that the first request after idle will be visibly bad. Pick the end you can defend to the person who pays either bill.

What a warm replica does not cover

A minimum replica count fixes exactly one thing: the first request after an idle period on an endpoint that would otherwise have no replicas. It does not help with three neighbouring problems that look similar from the client side.

• Scale-out latency. Traffic above what the current replicas serve still waits for new replicas, which pay the full start described at the top of this page. The minimum sets the floor, not the rate of climb; the ceiling and the metric target set that.
• Deployment transitions. Deploying a new model version to the same endpoint starts new replicas from cold. Split traffic gradually with --traffic-split so the new deployment is warm before it carries meaningful load.
• Quota and capacity. A minimum replica count is not a reservation of accelerator capacity. Region-level GPU availability is a separate constraint with its own quota names, and it is the thing that turns a scale-out into a failure rather than a delay. Check the accelerator quota for your region before assuming a --max-replica-count is achievable.