Deploying a Model From Vertex AI Model Garden

A Model Garden deploy looks like one click and is really two decisions: which machine shape the model is verified on, and whether your project holds accelerator quota for it in that region. Get those wrong and the job fails after you have waited for it.

Find the model and its supported shapes

The gcloud surface for Model Garden is small and stable. Google’s CLI reference documents gcloud ai model-garden models list with a --model-filter flag that filters on model names or display names, and a --can-deploy-hugging-face-models flag that narrows the list to Hugging Face models Model Garden can deploy.

gcloud ai model-garden models list --model-filter=gemma

gcloud ai model-garden models list --can-deploy-hugging-face-models

The identifier format is the thing to memorise, because every other command takes it: publisher/model@version for a Model Garden listing, as in google/gemma2@gemma-2-9b, or the ordinary Hugging Face convention such as meta-llama/Meta-Llama-3-8B for a Hugging Face model. A Cloud Storage URI is accepted for your own weights.

Before deploying, ask which machine shapes the model is actually verified on. Google documents gcloud ai model-garden models list-deployment-config as returning the supported machine specifications for a given model, which saves you discovering by failure that an 8B model does not fit on the accelerator you assumed.

gcloud ai model-garden models list-deployment-config \
  --model=google/gemma2@gemma-2-9b

The deploy command

One command creates the endpoint, uploads a model backed by Google’s serving container, and deploys it. Google’s reference for gcloud ai model-garden models deploy documents --model and --region as required, with --endpoint-display-name, --machine-type, --accelerator-type, --accelerator-count and --accept-eula alongside them.

gcloud ai model-garden models deploy \
  --model=google/gemma2@gemma-2-9b \
  --region=us-central1 \
  --endpoint-display-name=gemma-9b \
  --machine-type=g2-standard-12 \
  --accelerator-type=NVIDIA_L4 \
  --accelerator-count=1 \
  --accept-eula

1. Accept the licence. --accept-eula is not decoration: gated models will not deploy without it, and for a Hugging Face gated model you also need --hugging-face-access-token carrying a token that has been granted access to that repository.
2. Wait for the operation. A GPU-backed serving container has to pull an image and load weights before its health route answers; this is minutes, not seconds.
3. Call the resulting endpoint exactly as you would any other, with :predict or :rawPredict depending on the schema Google’s container exposes for that model family. The endpoint deploy page covers the request shape.

The quota it consumes

This is where a first deploy usually dies, and the error is a quota message rather than anything about the model. Serving accelerators are governed by per-region custom model serving quotas named after the accelerator — custom_model_serving_nvidia_l4_gpus is the L4 one, with parallel metrics for T4, A100 and H100 families — and a new project commonly holds zero in the region you picked.

One detail is worth knowing because it changes how much headroom you need to ask for. Google’s Vertex AI release notes record that custom model serving quota is now calculated from a deployed model’s real-time usage of compute resources; previously the resources were deducted against the deployment’s maxReplicaCount. Under the older accounting, an endpoint configured to burst to twenty replicas reserved twenty replicas’ worth of quota whether or not it ever scaled. Under the current accounting it does not, which means a generous maxReplicaCount is no longer something you pay for in quota alone.

Calling what you deployed

A deploy from Model Garden does not give you a Gemini-shaped API. It gives you an endpoint fronting a serving container Google selected for that model family — commonly a vLLM or text-generation server — and the request body that endpoint accepts is whatever that container accepts, not something Vertex AI standardises across listings. Two open models deployed the same way can expect two different bodies.

So the first thing to do after a successful deploy is find out what you have, rather than guessing. Describe the deployed model and read the container spec back: gcloud ai models describe MODEL_ID --region=us-central1 shows the container image, the ports and the health and predict routes it was registered with. If the predict route is the Vertex envelope, you send instances to :predict. If the container exposes a chat completions path of its own, you send its native body to :rawPredict, which passes it through untouched, and receive the container’s native response back the same way. Streaming, where the container supports it, goes through :streamRawPredict.

Two practical consequences follow, and both are the sort of thing that shows up a week later. Client code written against a managed Gemini endpoint does not port to a Model Garden endpoint without a translation layer — the auth is the same, the URL shape is the same, and everything inside the body is different. And sampling parameters are the container’s, not Vertex AI’s: a field name that works against one open model may be ignored rather than rejected by another, which fails silently by producing default behaviour instead of an error. Assert on a parameter’s effect once rather than assuming it was honoured.

Spot capacity and reservations

Two flags on the deploy command change the economics substantially. --spot deploys on Spot VMs, which are cheaper and can be reclaimed — acceptable for an internal evaluation endpoint, not for anything with an availability commitment, because a reclaim is an outage you did not schedule. --reservation-affinity does the opposite: it draws from a shared reservation you have already paid to hold, which is how you make a deploy deterministic in a region where on-demand accelerators are contended.

The failure mode of on-demand GPU capacity is not a quota error, and people conflate the two. Quota is your project’s administrative allowance; capacity is whether the machines physically exist free in that zone right now. You can hold quota for sixteen L4s and still fail to deploy because the region is full. A reservation is the only instrument that fixes the second problem.

What you have afterwards

You have an endpoint billing by node-hour for as long as it exists, whether or not anybody calls it. This is the sharpest difference between deploying an open model from Model Garden and calling a managed Gemini model, which is billed per token and costs nothing at idle. A self-hosted 9B model on a single L4 that serves fifty requests a day is almost always more expensive than the equivalent managed calls, and the reasons to do it anyway — data residency, a licence you need, a tuned checkpoint, predictable latency under load — are real but should be stated out loud before the endpoint is created.

Undeploy is the other half of the lifecycle and the one that gets forgotten. Removing an experiment means undeploying the deployed model and then deleting the endpoint; deleting the Model Garden listing from your registry does nothing to the running replicas.