Infrastructure as Code for AI Stacks

Terraform for an AI stack differs from Terraform for a web stack in two specific ways, and both cause the same symptom — a plan that wants to change something you did not change. The first is that GPU capacity is mutated by controllers outside your state. The second is that the most valuable things in the stack are secrets, which must never be written to a state file. Everything else is ordinary.

Repository layout that survives a year

Split by blast radius, not by service. The rule that pays for itself: things that change hourly must not share a state file with things that would take a day to rebuild.

infra/
  modules/
    gpu-pool/          # node group, taints, labels, device plugin
    inference-queue/   # queue + dead-letter + alarms
    model-bucket/      # object storage for weights, versioning, lifecycle
    secrets/           # secret containers only, never values
    observability/     # metric exporters, dashboards, alert rules
  envs/
    dev/
      main.tf
      backend.tf       # its own remote state
      terraform.tfvars
    staging/
    prod/
      main.tf          # composes the modules; nothing clever lives here
      backend.tf
      terraform.tfvars

Three rules that follow from this shape. Environments are directories, not workspaces — a directory diff is reviewable and a workspace is invisible in a pull request. Modules take inputs and return outputs and contain no environment names. And the root module composes; if there is logic in envs/prod/main.tf that is not in envs/dev/main.tf, production is no longer being tested by staging.

A GPU pool module

The module’s job is to encode the decisions you do not want repeated: which instance types are acceptable, what taint keeps ordinary workloads off, what labels the scheduler needs, and what the capacity bounds are.

# modules/gpu-pool/variables.tf
variable "name"            { type = string }
variable "cluster_name"    { type = string }
variable "instance_types"  { type = list(string) }   # more than one: capacity
variable "capacity_type"   { type = string  default = "ON_DEMAND" }
variable "min_size"        { type = number }
variable "max_size"        { type = number }
variable "desired_size"    { type = number }
variable "gpu_label"       { type = string }         # e.g. "nvidia-a100"
variable "subnet_ids"      { type = list(string) }
variable "tags"            { type = map(string) default = {} }

# modules/gpu-pool/main.tf  (shape; adapt resource names to your provider)
resource "<provider>_node_group" "this" {
  cluster_name   = var.cluster_name
  node_group_name = var.name
  subnet_ids     = var.subnet_ids
  instance_types = var.instance_types
  capacity_type  = var.capacity_type

  scaling_config {
    min_size     = var.min_size
    max_size     = var.max_size
    desired_size = var.desired_size
  }

  labels = {
    accelerator      = var.gpu_label
    "workload-class" = "inference"
  }

  taint {
    key    = "nvidia.com/gpu"
    value  = "present"
    effect = "NO_SCHEDULE"
  }

  tags = merge(var.tags, {
    "cost-center" = var.tags["cost-center"]
    "managed-by"  = "terraform"
  })

  lifecycle {
    ignore_changes  = [scaling_config[0].desired_size]
    create_before_destroy = true
  }
}

Two arguments there are the whole point of the module. instance_types is a list because a pool restricted to one type in one zone is a pool that cannot scale during a shortage; naming several compatible types lets the provider satisfy you from whatever it has. Spot and preemptible GPUs explains why that diversity matters even more when capacity_type is spot.

And tags is not decoration: those tags are what makes the cloud bill attributable later. Enforce a required set — cost centre, environment, owner, service — with a variable validation block so an untagged pool cannot be created at all. Cost allocation across teams is entirely downstream of this decision.

Drift, and the autoscaler that causes it

Drift is any difference between what state records and what exists. Most drift is somebody clicking in a console. In an AI stack, most drift is a robot doing its job.

The cluster autoscaler changes desired_size constantly — that is its entire function. If Terraform also manages that field, every plan proposes to set it back to whatever was in the tfvars, and every apply causes a scale event. The result is either an apply that terminates GPU nodes serving traffic, or a team that stops running apply because it is scary. Both are bad, and the fix is one line:

lifecycle {
  ignore_changes = [scaling_config[0].desired_size]
}

Terraform still owns min_size and max_size — the bounds, which are policy — and the autoscaler owns the position within them, which is operations. That division is the correct general rule wherever a controller shares a resource with your state file. The same applies to replica counts on Deployments managed by an HPA, and to any tag a cost or security tool writes back.

Detect the rest of the drift on a schedule rather than discovering it during an incident:

# Runs nightly. Exit code 2 means "changes present" — that is the signal,
# not a failure of the command.
terraform plan -detailed-exitcode -lock=false -out=/dev/null
code=$?
case $code in
  0) echo "no drift" ;;
  2) echo "DRIFT DETECTED"; terraform show -json /dev/null > /dev/null
     notify "terraform drift in $ENVIRONMENT"; exit 0 ;;
  *) echo "plan failed"; exit $code ;;
esac

Send the result to a channel a human reads, not to a pager. Drift is almost never urgent and almost always informative: it tells you which resources people are editing by hand, which is a list of things your modules should probably expose properly.

Secrets never go in state

This is the rule that catches teams out, and it is absolute: anything Terraform manages is written to state in plaintext, including values marked sensitive. The sensitive = true flag redacts a value from console output. It does not encrypt it in the state file. A provider API key placed in a Terraform variable is a provider API key in an object-storage bucket.

So Terraform creates the container and the access policy, and something else puts the value in.

# Terraform: create the secret, not its value.
resource "<provider>_secret" "openai_key" {
  name        = "prod/model-provider/primary"
  description = "Primary provider key. Value set out of band; rotated by the rotation job."

  lifecycle {
    ignore_changes = [value]     # never manage the material
  }
}

resource "<provider>_iam_policy" "read_provider_key" {
  # grant only the serving role, only this secret, only read
}

# Outside Terraform, once, from an operator machine:
#   provider-cli secret set prod/model-provider/primary --value-from-stdin
# and thereafter from the rotation job.

Then treat the state file itself as sensitive regardless: remote backend, encryption at rest, state locking, access restricted to the apply role, and no state files in the repository. Terraform state contains database endpoints, ARNs, generated passwords for anything you did let it create, and the full topology of your system. Secrets management for AI systems covers the runtime half, and rotating a provider key without downtime covers the job that writes the new value.

Quotas and capacity are not resources

The most common way an AI Terraform run fails is not a syntax error. It is an apply that succeeds in the plan and then cannot get the machines, because the account has a limit on accelerator instances in that region, or because there is no capacity to give.

Neither of those is something Terraform can create. Handle them as preconditions instead:

• Record the quota you rely on as data, not as faith. Where the provider exposes a quota lookup, read it in a data source and assert on it, so the plan fails with “quota is 8, this needs 12” rather than the apply failing halfway.
• Raising a quota is a ticket with a lead time. It is a step in the project plan with a date, not a step in the pipeline. Ask early.
• Capacity is separate from quota. Having permission to launch twelve accelerators does not mean twelve exist for you today. Reservations and committed-capacity products exist for exactly this, and their names and terms differ by provider — check the current documentation rather than a blog post.
• Make the failure legible. A pool that cannot reach min_size should raise an alert that says so, because the symptom otherwise arrives as pods stuck in Pending, several layers away from the cause.

Plan in CI, apply from one place

The workflow that avoids most incidents is boring and worth stating.

1. Pull request runs fmt, validate and a plan against the real state, with read-only credentials, and posts the plan as a comment. The plan is what gets reviewed — the diff of the HCL is not sufficient, because a one-line change can replace a node group.
2. A policy check runs against the plan. Required tags present, no security group open to the world, no unencrypted bucket, no GPU pool with a max size above the agreed ceiling. Machine-checked beats reviewer-remembered.
3. Apply happens from CI on merge, never from a laptop. One set of credentials, one audit trail, one place where state locks.
4. Read the plan for the words that mean downtime. must be replaced and forces replacement on a node group, a bucket or a database are the lines to stop at. On a GPU pool serving traffic, a replacement is an outage unless create_before_destroy is set and the quota exists for both to be alive at once.
5. Nightly drift detection as above, reporting to a channel.

The one manual escape hatch worth keeping is a documented procedure for breaking a stuck state lock, because it will happen during an incident and the sequence is not something to improvise.