Terraform State Locking for a Shared GPU Cluster Config

Two engineers running terraform apply on the same GPU node pool at the same time is not a merge conflict — it is two writers to one state file, and the loser’s resources become invisible to Terraform while continuing to bill. Locking is the fix, and the mechanism changed recently enough that most published examples are now the deprecated one.

Why GPU config is the worst thing to race on

State corruption is bad everywhere. It is worse on accelerator capacity for two reasons that are specific to this workload.

The first is that GPU node pools are the expensive resource in the account, so a node pool that falls out of state is a node pool nobody is tracking and nobody destroys. It shows up as a line on next month’s bill rather than as an error. The second is that GPU capacity is often scarce: if a concurrent apply destroys and recreates a node pool, the recreate can fail on stock-out in the zone, and now you have neither the old pool nor the new one, in a region where you cannot immediately get the hardware back. A lock turns both of those into a message telling the second engineer to wait.

The backend block

Terraform’s S3 backend now performs locking natively, using a lock object stored beside the state, and HashiCorp’s documentation describes use_lockfile as controlling whether a lockfile is used for locking the state file, defaulting to false. With it enabled Terraform writes a .tflock object next to the state key for the duration of the operation.

terraform {
  required_version = ">= 1.10"

  backend "s3" {
    bucket       = "acme-tfstate-prod"
    key          = "gpu-cluster/terraform.tfstate"
    region       = "us-east-1"
    encrypt      = true
    kms_key_id   = "arn:aws:kms:us-east-1:123456789012:key/abcd-1234"
    use_lockfile = true
  }
}

Three things about this block are load-bearing. bucket, key and region are the only required arguments, so everything else here is a deliberate choice. encrypt covers the lock file as well as the state. And key is one path per configuration — if the GPU cluster and the application share a key, they share a lock, and every application deploy blocks on every cluster change.

Bucket versioning is not optional in practice even though Terraform does not require it. Locking prevents concurrent writes; versioning is what lets you recover from a bad write that was perfectly serialised.

The permission everybody forgets

This is the failure that sends people back to DynamoDB believing the feature does not work. The lock object is a different S3 key from the state object, and a policy written for state access does not cover it. HashiCorp’s documentation lists s3:GetObject, s3:PutObject and s3:DeleteObject as required on the lock file — for example on arn:aws:s3:::mybucket/path/to/my/key.tflock.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "StateObject",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::acme-tfstate-prod/gpu-cluster/terraform.tfstate"
    },
    {
      "Sid": "LockObject",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::acme-tfstate-prod/gpu-cluster/terraform.tfstate.tflock"
    },
    {
      "Sid": "ListBucket",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::acme-tfstate-prod"
    }
  ]
}

Note that s3:DeleteObject on the lock is what releases it. A role granted get and put but not delete will acquire a lock it can never release, and every subsequent apply — including its own next one — will block. That failure looks exactly like a colleague leaving a lock open, which is why it can go undiagnosed for a while.

If you are using workspaces, the state path becomes <workspace_key_prefix>/<workspace_name>/<key> with a default prefix of env:, so the resource ARNs above need a wildcard rather than a literal path. Terraform workspaces are a reasonable way to separate a dev and a prod GPU pool; if you are on Pulumi instead, the equivalent separation is one stack per environment.

Migrating off the DynamoDB table

You cannot flip everybody at once, because an engineer on an older Terraform binary will not honour a lock they cannot see — which is worse than no locking, since everyone believes they are protected. HashiCorp’s documented migration path is to run both mechanisms simultaneously for a window.

1. Add use_lockfile = true while keeping dynamodb_table set. Terraform takes both locks; old binaries take only the DynamoDB one and are still excluded.
2. Add the three .tflock permissions to every role and to your CI role. Run terraform init -reconfigure everywhere.
3. Pin a minimum version in required_version so a stale binary fails loudly instead of applying without the lockfile.
4. After every consumer is on the new version and has applied at least once, remove dynamodb_table, re-init, and delete the table. Delete it last: a table removed from config but still holding a stale lock item is confusing to diagnose.

What the lock does not protect against

The lock is a guarantee about one state file. It is not a guarantee about your cloud account, and conflating the two is how people end up surprised by a corruption they thought they had designed out.

It is scoped to the state key, not to the resource. If the GPU node pool is described in two configurations — say a platform repository and an application repository that both grew a copy — they hold two different locks and can apply simultaneously all day. Each will observe the other’s changes as drift and revert them, in a loop that looks like flapping infrastructure rather than like a configuration problem. The lock cannot help; only deleting one of the two definitions can.

It is not a transaction. If an apply dies halfway — the runner is preempted, the network drops, somebody presses Ctrl-C twice — the lock is released and the state records what Terraform managed to write. Resources created after the last state write exist in the cloud and not in state. On a GPU pool that is an expensive orphan, and the recovery is to import it back, not to rerun and hope. Rerunning creates a second one.

Read operations take the lock too. terraform plan locks state by default, which surprises people who expect a read to be free. That is deliberate — a plan refreshes state — but it means a long apply blocks everyone’s plans as well as their applies. The two flags that matter:

# In CI: wait for the lock rather than failing the pipeline.
terraform apply -lock-timeout=10m

# Speculative, read-only plan on a branch. Never on apply.
terraform plan -lock=false -refresh=false

-lock-timeout defaults to zero, meaning Terraform gives up immediately if the lock is held. In an automated pipeline that turns a normal five-minute overlap into a red build, so set it on every command in CI. -lock=false is the opposite: it is defensible only for a speculative plan you will not apply, and using it on apply or destroy removes the entire protection this page is about.

Locking is the backstop, not the design. If two pipelines can reach the same state, the correct fix is upstream: one pipeline per state file, with the CI system serialising runs — a concurrency group in GitHub Actions, a resource_group in GitLab CI. Then the lock exists to catch the human running a command from a laptop, which is what it is good at. A team relying on lock contention as its scheduling mechanism will spend a lot of time reading the next section.

When the lock is stuck

A cancelled CI job or a killed laptop leaves the lock held. Terraform fails with Error acquiring the state lock and prints the lock ID, who created it, the operation and when. Read that block before doing anything — most of the time the answer is that a colleague is mid-apply and you should wait, and force-unlocking a live apply is how you get the corruption the lock existed to prevent.

# Only after confirming no apply is running.
terraform force-unlock 4a1b7f2e-9c33-4f10-8a5e-7b2c9d0e1f34

If you genuinely cannot tell whether an apply is running, the safer move on S3-native locking is to inspect the lock object directly — aws s3api head-object on the .tflock key gives you its last-modified time, and a lock older than your longest plausible apply is stale. On a GPU cluster where a node pool replacement can take a long time, “longest plausible apply” is longer than your intuition suggests.