Pulumi Stacks for Separate Dev and Prod Model Endpoints

The reason to use stacks rather than an if (env === "prod") in one deployment is not tidiness. It is that two stacks have two state files, and a mistake in one cannot reach into the other. Every other benefit is downstream of that.

Where the isolation comes from

A Pulumi stack is an independently configurable instance of a program, with its own state and its own resources. Two stacks of the same project running the same code produce two disjoint sets of resources that know nothing about each other.

Compare that with the alternative people reach for first: one stack, one state file, a boolean, and both environments’ resources inside it. That configuration is one bad pulumi destroy away from taking production with it, and — more insidiously — one shared resource away from a dev change causing a prod incident. On model serving that shared resource is usually the endpoint, the deployed model, or the quota they draw from.

pulumi stack init dev
pulumi stack init prod
pulumi stack ls
pulumi stack select dev

Note that a stack is not a branch. Both stacks run the same code at whatever commit you deploy from; the difference between them is configuration, not source. If dev and prod need genuinely different code, that is a signal the difference should be a configuration value you have not extracted yet.

Configuration per stack

Per-stack configuration lives in Pulumi.<stack-name>.yaml — so Pulumi.dev.yaml and Pulumi.prod.yaml beside the project’s own Pulumi.yaml. Pulumi’s documentation notes these are not created by pulumi stack init; they are created and managed by pulumi config.

pulumi config set --stack dev  gcp:region        us-central1
pulumi config set --stack dev  minReplicaCount   0
pulumi config set --stack dev  machineType       n1-standard-4

pulumi config set --stack prod gcp:region        us-central1
pulumi config set --stack prod minReplicaCount   2
pulumi config set --stack prod machineType       g2-standard-8

pulumi config set --stack prod --secret upstreamApiKey "sk-..."

--secret encrypts the value in the stack’s YAML file, so the file is safe to commit. Use it for anything that would be a credential, and note that dev and prod encrypt under different keys — which is another form of the isolation this page is about.

The dev/prod split above is deliberately about capacity rather than topology. A dev endpoint that scales to zero and a prod endpoint that holds two warm replicas are the same resources with different numbers; that is what you want. When the shapes start to differ structurally, your dev environment has stopped testing anything.

One program, two shapes

import * as pulumi from "@pulumi/pulumi";
import * as gcp from "@pulumi/gcp";

const cfg = new pulumi.Config();
const stack = pulumi.getStack();          // "dev" or "prod"

const endpoint = new gcp.vertex.AiEndpoint(`inference-${stack}`, {
  name:        `inference-${stack}`,
  displayName: `inference (${stack})`,
  location:    cfg.require("gcp:region"),
  region:      cfg.require("gcp:region"),
  labels: {
    environment: stack,
    managed_by:  "pulumi",
  },
});

export const endpointId = endpoint.id;
export const environment = stack;

Two things here are doing real work. Including pulumi.getStack() in the resource name means the cloud resource names differ per environment, so even if the two stacks somehow targeted the same project you would get two resources rather than a collision or a silent adoption. And the environment label is what makes cost attribution possible later, which matters a great deal when the resource is an accelerator — see per-environment budgets for model calls.

Better still is to give each stack its own cloud project or account entirely, set through the provider’s own configuration key. Then the isolation is enforced by IAM rather than by naming discipline, and a credential that can deploy dev cannot see prod at all.

The two ways this leaks

Stack references. pulumi.StackReference reads another stack’s outputs, with a fully-qualified name of the form <organization>/<project>/<stack>. It is the correct tool for a genuine layering — a shared networking stack that both environments consume. It is a hole when the reference crosses environments.

// Correct: same environment, different layer.
const net = new pulumi.StackReference(`acme/network/${stack}`);

// Wrong: prod endpoint now depends on the dev stack existing,
// and a dev destroy breaks prod's next update.
const dev = new pulumi.StackReference("acme/inference/dev");

Make the stack name in a reference a function of the current stack, never a literal. A hard-coded environment name in a stack reference is the single most reliable way to reintroduce the coupling you split the stacks to avoid.

Resources that are not in the program. The quota is the classic one. Two stacks in the same cloud project draw from the same per-project accelerator quota and, for hosted model APIs, the same tokens-per-minute allocation. Pulumi cannot isolate what it does not create. If a dev load test can exhaust prod’s capacity, your stacks are separate and your environments are not — and the fix is separate projects or separate quota allocations, not more Pulumi.

When two pipelines update one stack

Separate stacks solve the cross-environment problem. They say nothing about two updates to the same stack, which is what you get the first time two merges land close together.

Pulumi Cloud handles it with leases: it allows at most one update to a particular stack at a time, and a second one is rejected with a 409 and a message saying another update is currently in progress. The important difference from Terraform’s state lock is that there is no wait — Pulumi’s update fails immediately rather than queueing behind the lease. There is no equivalent of -lock-timeout to lean on.

So the serialisation has to come from your CI system. Use a concurrency group in GitHub Actions or a resource_group in GitLab CI, keyed on the stack name, so the second pipeline waits instead of failing:

concurrency:
  group: pulumi-${{ github.ref_name }}-prod
  cancel-in-progress: false

cancel-in-progress: false is the part that matters. Cancelling an in-flight infrastructure update to start a newer one is almost always wrong: you interrupt a partially-applied change and start another on top of it.

pulumi cancel revokes a lease when one is genuinely stuck, and Pulumi’s troubleshooting documentation is blunt that cancelling somebody else’s update makes their update fail immediately. After a cancelled or crashed update, run pulumi refresh before anything else — resources may have been created without being recorded, and the next up would otherwise try to create them again.

Two flags make drift visible rather than something you discover during an incident. pulumi up --refresh reconciles state with reality before computing the update, which is the right default for a long-lived prod stack. And pulumi preview --expect-no-changes fails if the stack does not match its program — run it on a schedule against prod and you learn about a console edit the day it happens, not the week you next deploy.

One command deserves its own warning. pulumi stack rm refuses to remove a stack that still has resources unless you pass --force, and --force deletes the stack’s state while leaving every cloud resource running. On a GPU-backed endpoint that is an orphan nobody is tracking and nobody will destroy. If you are tearing an environment down, run pulumi destroy first and stack rm second — in that order, every time.

Promoting a change

1. Make the change and run pulumi preview --stack dev. Read it; do not skip to up.
2. pulumi up --stack dev. Verify against the dev endpoint.
3. pulumi preview --stack prod from the same commit. This is the step that catches the class of bug where a change is fine in dev because dev is configured differently — a replacement that is harmless on a zero-replica endpoint and an outage on a serving one.
4. Look specifically for replace in the prod preview. Pulumi marks replacements clearly, and on a model endpoint a replacement means a period with no serving capacity. Where it matters, use deleteBeforeReplace: false semantics or stage the change so the new resource exists before the old one goes.
5. pulumi up --stack prod.

In CI, pass --stack explicitly on every command. Relying on the selected stack is fine at a terminal where you can see it in the prompt; in a pipeline it is a state carried between steps, and the day it is wrong is the day you find out that pulumi destroy also respects the selected stack. This is the same argument as locking Terraform state from the other direction: the tooling should make the dangerous operation require an explicit target.