Deploying a Model to a Vertex AI Endpoint

Deploying to Vertex AI is not one operation. It is three resources created in order, and almost every confusing error comes from treating them as one thing.

Three resources, not one

A Model is a registry entry: a container image, an optional pointer to artifacts in Cloud Storage, and the two HTTP routes the container answers on. It costs nothing and serves nothing. An Endpoint is a stable URL and an IAM boundary. It also costs nothing on its own and, freshly created, returns an error for every request because nothing is behind it. A DeployedModel is the join between the two: a model placed on machines attached to an endpoint, with a replica range and a share of the endpoint’s traffic. That third object is the only one that consumes accelerator quota and the only one that bills.

Holding the three apart explains the error messages. A permission failure on upload is about the registry; a quota failure at deploy time is about machines; a 404 from your prediction call after a successful deploy usually means you called the endpoint before any deployed model had traffic assigned to it.

Upload the model

The upload step is where the container contract is declared. Google documents --container-image-uri as required and --artifact-uri as the path to “the directory containing the Model artifact and any of its supporting files”, which is handed to the running container as the AIP_STORAGE_URI environment variable.

gcloud ai models upload \
  --region=us-central1 \
  --display-name=sentiment-v3 \
  --container-image-uri=us-central1-docker.pkg.dev/PROJECT/serving/sentiment:3.1 \
  --artifact-uri=gs://PROJECT-models/sentiment/v3/ \
  --container-health-route=/health \
  --container-predict-route=/predict \
  --container-ports=8080

If you omit the route flags, the container is expected to answer on the defaults Vertex AI injects; setting them explicitly is worth the two lines because the values also become AIP_HEALTH_ROUTE and AIP_PREDICT_ROUTE inside the container, and a mismatch between what you registered and what your server routes is the single most common cause of a deploy that never becomes healthy. The full contract is in the custom container page.

Create the endpoint and deploy

1. Create the endpoint: gcloud ai endpoints create --region=us-central1 --display-name=sentiment. Note the numeric endpoint ID it returns; the display name is not an identifier.
2. Deploy the model onto it, naming the machine shape and the replica range explicitly.
3. Wait. The command returns when the operation is accepted; the deployment is not serving until the health route has returned 200. For a GPU-backed container this can be many minutes.

gcloud ai endpoints deploy-model ENDPOINT_ID \
  --region=us-central1 \
  --model=MODEL_ID \
  --display-name=sentiment-v3 \
  --machine-type=n1-standard-4 \
  --min-replica-count=1 \
  --max-replica-count=3 \
  [email protected] \
  --traffic-split=0=100

Two flags earn attention. Google’s CLI reference documents --min-replica-count with “a value of 0 enables scale-to-zero”, which changes the cost profile of a rarely-used endpoint completely and buys you a cold start in exchange. --service-account is the identity the container runs as, which is a different thing from the identity of whoever calls the endpoint; leave it unset and the container inherits a default service account that is usually broader than you want. The gcloud reference for deploy-model lists both, along with --accelerator and --autoscaling-metric-specs.

Call it with a signed request

Vertex AI online prediction takes an OAuth 2.0 bearer token, not an API key. The regional host matters: a request to the wrong REGION-aiplatform.googleapis.com for the endpoint’s region fails as a missing resource rather than as a routing error.

curl -X POST \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
  https://us-central1-aiplatform.googleapis.com/v1/projects/PROJECT/locations/us-central1/endpoints/ENDPOINT_ID:predict \
  -d '{"instances": [{"text": "the refund arrived a week late"}]}'

The :predict method wraps and unwraps for you: it expects an instances array and hands your container a body of that shape. If your server speaks some other schema — an OpenAI-compatible chat body, say, or a raw tensor — use :rawPredict instead, which passes the body through untouched. The CLI mirrors this with gcloud ai endpoints predict, raw-predict and stream-raw-predict, the last of which is how you get a streamed response out of a custom container.

The response envelope is worth reading once rather than assuming. A :predict response carries a predictions array alongside a deployedModelId and the model resource name, which is how you find out which revision answered when several share the endpoint under a traffic split. Logging that field costs nothing and is the difference between “quality dropped this afternoon” and “quality dropped on the ten percent of traffic going to v4”. Access logging is off by default; --enable-access-logging on the deploy command sends per-request timestamps and latency to Cloud Logging, and Google’s CLI reference is explicit that this is opt-in.

One request-shape limit deserves planning around. Online prediction is built for small payloads and there is a size ceiling on the request body, so a workload that wants to score a large document per call should pass a Cloud Storage URI and let the container fetch it, rather than inlining the content. If the payload is large and the latency does not matter, that workload is not an online prediction at all — it is a batch prediction job, which has no such constraint and costs substantially less.

The second deploy is the interesting one

An endpoint can carry several deployed models at once, and --traffic-split is what makes it a release mechanism rather than a hosting detail. Deploying version four with --traffic-split=0=90,DEPLOYED_MODEL_ID_V3=10 — where 0 is the placeholder for the model being deployed in this command — puts the new revision behind the same URL at a tenth of the traffic, with no DNS change and no client change. Roll back by updating the split; the old replicas are still running.

The corollary is a billing trap. Setting a deployed model’s traffic share to zero does not undeploy it. Its replicas stay up and keep billing until you run gcloud ai endpoints undeploy-model with its deployed model ID. Endpoints accumulate idle revisions this way, and because the endpoint itself looks fine, nobody notices.

Where the first attempt fails

• The container never passes its health check. Vertex AI starts probing as soon as the container starts. If your process loads weights after binding the port, it answers the probe before it can serve, or fails to answer at all while loading. Load first, bind second.
• Accelerator quota is zero in that region. GPU machine types draw on a named custom-model-serving quota per region, and a new project frequently starts at zero. That surfaces at deploy time, not at upload time — see requesting an increase.
• The caller lacks one permission. Calling :predict needs aiplatform.endpoints.predict and nothing else. A caller granted a viewer role reads the endpoint fine and gets a permission denied on the prediction.
• You deleted the endpoint before undeploying. An endpoint with a deployed model attached refuses deletion. Undeploy each deployed model ID first, then delete.