Autoscaling a Vertex AI Endpoint by Traffic

Autoscaling on a Vertex AI endpoint is four fields on the deployed model, not a separate policy object. Getting them right is mostly about knowing which metric applies to your machine shape and what the floor costs you when nothing is happening.

The fields the API actually takes

A model deployed to an endpoint carries a DedicatedResources block, and Google’s REST reference for it lists the relevant fields:

• machineSpec — machine type and any accelerator type and count. Fixed for the deployment; changing it means a new deployment.
• minReplicaCount — the floor. This is the number you pay for continuously.
• maxReplicaCount — the ceiling. This is your cost cap and your throughput cap simultaneously.
• autoscalingMetricSpecs — a list of metricName plus target pairs. The target is a percentage expressed as an integer.
• spot — whether to run on Spot capacity, which is cheaper and can be reclaimed.

There is no cooldown, no step size and no schedule. Unlike a managed instance group, this is a small surface, and everything you can influence is in the list above. That is a limitation and also a mercy: there is not much to get subtly wrong.

Deploying with bounds

1. Create the endpoint if it does not exist. It is a routing object and costs nothing on its own:

   gcloud ai endpoints create \
     --region=us-central1 \
     --display-name=classifier-endpoint

2. Deploy the model onto it with explicit bounds. Do not accept defaults here — both numbers are cost decisions:

   gcloud ai endpoints deploy-model ENDPOINT_ID \
     --region=us-central1 \
     --model=MODEL_ID \
     --display-name=classifier-v3 \
     --machine-type=n1-standard-4 \
     --min-replica-count=1 \
     --max-replica-count=6 \
     --traffic-split=0=100

3. Override the metric target where the default is wrong for your workload. The Python SDK exposes this as a named argument:

   from google.cloud import aiplatform
   
   aiplatform.init(project="PROJECT_ID", location="us-central1")
   
   endpoint = aiplatform.Endpoint("projects/PROJECT_ID/locations/us-central1/endpoints/ENDPOINT_ID")
   model = aiplatform.Model("projects/PROJECT_ID/locations/us-central1/models/MODEL_ID")
   
   endpoint.deploy(
       model=model,
       deployed_model_display_name="classifier-v3",
       machine_type="n1-standard-4",
       min_replica_count=1,
       max_replica_count=6,
       autoscaling_target_cpu_utilization=50,
       traffic_percentage=100,
   )

4. Confirm what landed, because the deployment is what the autoscaler reads and not what you meant:

   gcloud ai endpoints describe ENDPOINT_ID --region=us-central1 \
     --format='yaml(deployedModels[].dedicatedResources)'

Which metric the autoscaler uses

Google documents the CPU metric as aiplatform.googleapis.com/prediction/online/cpu/utilization, with a default target of 60 when machineSpec.acceleratorCount is 0 and no target is set explicitly. The autoscaler compares observed utilization against that target and adds or removes replicas to close the gap.

The rule people miss is what happens when an accelerator is attached. On a GPU-backed deployment, CPU utilization is a poor proxy for load — the CPU may sit near idle while the GPU is saturated, so a CPU-only policy will refuse to scale a deployment that is visibly overloaded. GPU duty cycle is the metric that reflects the actual bottleneck, and on a GPU deployment it is the one to set a target on, alongside CPU rather than instead of it. Where both are specified, the autoscaler scales on whichever is furthest above its target, which is the behaviour you want.

Choosing the target itself: a target of 60 means the autoscaler aims to keep replicas at 60% utilized, so 40% of capacity is headroom for the time it takes a new replica to become ready. That readiness time is the real input. A replica that takes four minutes to load a model needs a lower target — more headroom — because four minutes of traffic growth has to fit inside it. A replica that is ready in thirty seconds can run hotter. Setting a high target on a slow-starting model gives you an autoscaler that is always four minutes behind the traffic, which looks exactly like an autoscaler that does not work.

Watching it react

Two things are worth watching and they are different signals. Replica count tells you what the autoscaler did; utilization tells you why.

gcloud monitoring time-series list \
  --project=PROJECT_ID \
  --filter='metric.type="aiplatform.googleapis.com/prediction/online/replicas"
            AND resource.labels.endpoint_id="ENDPOINT_ID"' \
  --format='table(points[].value.int64Value, points[].interval.endTime)'

Drive load against it with a tool that reports latency percentiles rather than a loop of curl, and ramp rather than stepping straight to peak — a step tells you nothing about how the autoscaler behaves in between. What you are looking for is the shape: utilization climbs, replica count follows some minutes later, latency spikes in the gap and recovers. That gap is your readiness time and it is the number that decides your target. The general method is in load testing an inference endpoint.

Scale-down is deliberately slower than scale-up, and this is correct behaviour rather than lag. Removing a replica that is about to be needed costs another full startup, so the autoscaler is conservative about it. Do not tune against a brief post-burst period of apparent over-provisioning.

Why the floor is the expensive decision

A deployed model with minReplicaCount of 1 is a machine running continuously, billed by node-hour, whether or not a single prediction is served. Overnight, at weekends, and during the six months after the project that needed it was quietly abandoned. This is the single largest cost difference between a dedicated endpoint and calling a shared Gemini model, where an idle hour costs nothing.

So the floor deserves a real decision. Where the endpoint serves an interactive path with continuous traffic, 1 is a floor and 2 is the honest one, because a single replica is a single point of failure and a restart is downtime. Where the endpoint serves a batch or internal workload that runs for two hours a day, the right answer is often not an online endpoint at all — batch prediction exists precisely so you do not hold a machine open for twenty-two idle hours.

The ceiling deserves the same care from the other direction. Setting maxReplicaCount high “just in case” means a traffic anomaly or a retry storm can scale you to that number and bill for it, and it also means you will hit accelerator quota before you hit your own ceiling — what RESOURCE_EXHAUSTED looks like covers that collision. Set the ceiling to the largest number you are willing to pay for in an hour, not the largest number the quota allows.