Provisioning a Vertex AI Endpoint With Pulumi

The obvious Pulumi program — create a gcp.vertex.AiEndpoint, point it at a model, call pulumi up — does not exist, because that resource does not accept a model. Knowing which of the two real paths you are on before you write anything saves an afternoon.

The endpoint resource does not deploy a model

gcp.vertex.AiEndpoint creates the serving surface and nothing else. Pulumi’s registry documentation is explicit that models are deployed into an endpoint afterwards, through the API’s EndpointService.DeployModel and EndpointService.UndeployModel operations. The resource exposes trafficSplit, which is a map from deployed-model ID to percentage — and those IDs are generated by a deploy call the resource itself never makes.

So a program containing only AiEndpoint converges on an empty endpoint that accepts no predictions. It is not broken; it is finished, and it is half of what you wanted. The properties it does take are worth knowing because several of them are immutable once set:

• location and region — the Vertex region, e.g. us-central1. Not changeable in place.
• network — a fully-qualified VPC path of the form projects/PROJECT_NUMBER/global/networks/NAME. It takes the project number, not the project ID, and this is a common silent failure.
• dedicatedEndpointEnabled — gives the endpoint its own DNS name rather than the shared regional one.
• encryptionSpec.kmsKeyName — CMEK, also fixed at creation.
• trafficSplit — a JSON string, not an object, in the TypeScript SDK.

The one-resource path: Model Garden

If the model is a Model Garden publisher model or a Hugging Face model, there is a resource that does the whole job: gcp.vertex.AiEndpointWithModelGardenDeployment. It creates the endpoint and deploys the model into it as one unit, which is what you wanted in the first place.

import * as gcp from "@pulumi/gcp";

const served = new gcp.vertex.AiEndpointWithModelGardenDeployment("served", {
  location: "us-central1",
  publisherModelName: "publishers/google/models/paligemma@paligemma-224-float32",
  modelConfig: {
    acceptEula: true,
    modelDisplayName: "paligemma-224",
  },
  deployConfig: {
    dedicatedResources: {
      machineSpec: {
        machineType: "g2-standard-16",
        acceleratorType: "NVIDIA_L4",
        acceleratorCount: 1,
      },
      minReplicaCount: 1,
    },
  },
});

acceptEula is not decoration. Model Garden entries carry licence terms and the deployment fails without it, so the flag is effectively a required field for most models. For a Hugging Face model, swap publisherModelName for huggingFaceModelId and supply modelConfig.huggingFaceAccessToken for gated repositories.

The custom-model path

For a model you trained or containerised yourself, there is no single-resource equivalent, and this is where people write a broken program. The honest shape is: create the endpoint declaratively, upload the model, then perform the deploy as an explicit step that Pulumi triggers rather than models.

1. Create the endpoint with gcp.vertex.AiEndpoint, exporting its ID.
2. Upload the model artefact — the container image URI plus the GCS path to the weights — so it exists in the Vertex model registry.
3. Run the deploy through a command resource, keyed on the model version so that a new version triggers a new deploy and an unchanged one does not:

import * as command from "@pulumiverse/command"; // or @pulumi/command
import * as pulumi from "@pulumi/pulumi";

const endpoint = new gcp.vertex.AiEndpoint("inference", {
  name: "inference",
  displayName: "inference",
  location: "us-central1",
  region: "us-central1",
});

const deploy = new command.local.Command("deploy-model", {
  create: pulumi.interpolate`gcloud ai endpoints deploy-model ${endpoint.name} \
    --region=us-central1 \
    --model=${modelId} \
    --display-name=inference-v${modelVersion} \
    --machine-type=g2-standard-8 \
    --accelerator=type=nvidia-l4,count=1 \
    --min-replica-count=1 --max-replica-count=3 \
    --traffic-split=0=100`,
  triggers: [modelVersion],
});

This is a compromise and it should be labelled as one in the code. The command resource has no read step, so Pulumi cannot detect drift in the deployment — if somebody undeploys the model by hand, the next pulumi up will not notice. Accept that, or write a dynamic provider with a real read against the Vertex API. The same trade-off appears in the Cloudflare Vectorize case, where the provider is also missing a resource.

Machine spec and replica counts

machineSpec.machineType and machineSpec.acceleratorType are not independently free. Google constrains which accelerators attach to which machine families — the G2 family exists to carry L4 GPUs, and an A2 or A3 machine type is how you get A100 or H100 class hardware. Picking a mismatched pair produces an error at deploy time rather than at plan time, because Pulumi is not validating the combination; the API is.

minReplicaCount is the number worth arguing about. Vertex bills dedicated endpoints for provisioned replicas rather than for requests, so a minReplicaCount of 1 on a development endpoint is a machine with a GPU attached running continuously whether or not anybody calls it. That is the same failure mode as an idle GPU node pool, arriving through a different door.

Replacement, drift and protecting the endpoint

Several of the endpoint properties listed earlier cannot be changed in place, so editing them is a delete and a create. On a serving endpoint that is downtime, and the preview is where you find out: Pulumi prints replace against the resource and marks the diff, rather than update. Read that word before confirming. The region, the VPC network and the encryption specification are the usual culprits, and all three are properties somebody edits believing them to be cosmetic.

Guard the ones you cannot afford to lose. Pulumi’s resource options are the mechanism, and three of them earn their place on a production endpoint:

const endpoint = new gcp.vertex.AiEndpoint("inference", {
  name: "inference",
  displayName: "inference",
  location: "us-central1",
  region: "us-central1",
}, {
  protect: true,                    // destroy is refused until this is removed
  ignoreChanges: ["trafficSplit"],  // owned by the deploy step, not by this program
  retainOnDelete: false,
});

protect makes pulumi destroy fail on that resource until somebody deliberately unsets it, which is exactly the friction you want between a mistyped stack name and a deleted production endpoint. retainOnDelete is the different, sharper tool: it drops the resource from state while leaving it alive in the cloud, which is useful when handing ownership to another stack and dangerous otherwise, because the resource keeps billing with nothing tracking it.

ignoreChanges on trafficSplit is the specific fix for the specific problem this resource creates. The traffic split is a map keyed by deployed-model IDs, and those IDs are generated by a deploy call your Pulumi program never made. If you declare the field, every out-of-band deploy puts state and reality out of step; if you ignore it, the deploy step owns it and the program stops arguing. Declaring it and then fighting the diff is the one option with no upside.

pulumi refresh is the command for reconciling state with what is actually there, and it is worth running before a risky update. Be clear about its limit, though: refresh asks each provider to read its resource, so it can only see what a provider can read. The command resource driving the custom-model deploy has no read implementation at all, which means a refresh returns cleanly and has verified nothing about whether the model is still deployed. A green refresh on this stack is a weaker statement than it looks.

Finally, the concurrency question. Pulumi Cloud grants a lease on a stack so that at most one update runs at a time, and a second one fails with a 409 and the message that another update is currently in progress. pulumi cancel revokes that lease — and Pulumi’s own troubleshooting documentation warns that cancelling somebody else’s update makes their update fail immediately. Treat it the way you would a Terraform force-unlock: confirm nothing is running first.

Deletion is where this bites

An endpoint with a model still deployed to it cannot be deleted. On the Model Garden path, AiEndpointWithModelGardenDeployment exposes a deletionPolicy that governs whether the underlying endpoint and model go with it — set it deliberately, because the default may leave orphans that keep billing.

On the custom path, the command resource needs a matching delete that undeploys the model before Pulumi destroys the endpoint, otherwise pulumi destroy fails halfway and leaves a stack in a partially-destroyed state that has to be untangled by hand. This is the single strongest argument for putting dev and prod endpoints in separate stacks: a failed destroy in dev should never be able to touch prod state.