Managing Secrets Manager Entries for Model API Keys With Terraform

The standard pattern — an aws_secretsmanager_secret plus an aws_secretsmanager_secret_version holding the key — writes your model provider’s API key into the Terraform state file in plaintext. The usual mitigation is to encrypt the state bucket. That is worth doing and it is not an answer.

Why the value ends up in state

Terraform stores the full attribute set of every managed resource so it can compute the next diff. secret_string is a managed attribute, so its value is in state — and the provider documentation says as much, recommending sensitive = true on the variable to keep it out of logs and console output. That flag governs display, not storage.

Encrypting the state bucket moves the problem rather than solving it, because the state file is not only at rest in S3. It is on the laptop of whoever ran the last plan. It is in the CI job’s working directory and possibly its artefact store. It is readable by every principal with access to the bucket, which is a much larger set than the principals you would grant secretsmanager:GetSecretValue to. And terraform show -json prints it in full to anyone who can run it.

For a model provider key this matters more than average, because the blast radius is spend. A leaked key is not just data access; it is somebody else’s inference bill on your account until you notice.

Write-only arguments

Terraform 1.11 introduced write-only arguments, which are sent to the provider and never persisted to state. The AWS provider exposes secret_string_wo on aws_secretsmanager_secret_version, paired with secret_string_wo_version — an integer you increment to tell Terraform that the unseen value has changed.

variable "provider_api_key" {
  type      = string
  sensitive = true
  ephemeral = true
}

resource "aws_secretsmanager_secret" "model_key" {
  name                    = "prod/model-provider/api-key"
  description             = "API key for the upstream model provider."
  kms_key_id              = aws_kms_key.secrets.arn
  recovery_window_in_days = 30
}

resource "aws_secretsmanager_secret_version" "model_key" {
  secret_id                = aws_secretsmanager_secret.model_key.id
  secret_string_wo         = var.provider_api_key
  secret_string_wo_version = 1
}

The version integer is the part people misunderstand. Terraform cannot diff a value it does not store, so it has no way to know the key rotated. Bumping secret_string_wo_version from 1 to 2 is the signal that produces a new secret version. Change the key without bumping it and nothing happens; bump it without changing the key and you create an identical new version, which is harmless.

Mark the input variable ephemeral as well as sensitive. Ephemeral values are not persisted to state or plan files at all, which closes the second leak — a plan file containing the value you carefully kept out of state.

The alternative: Terraform never sees it

There is a stronger position available, and on a team with humans holding provider keys it is often the right one: Terraform creates the secret container and the access policy, and the value is written by something else entirely.

resource "aws_secretsmanager_secret" "model_key" {
  name       = "prod/model-provider/api-key"
  kms_key_id = aws_kms_key.secrets.arn
}

# No aws_secretsmanager_secret_version at all.
# The value is placed once, by a human or a rotation lambda:
#
#   aws secretsmanager put-secret-value \
#     --secret-id prod/model-provider/api-key \
#     --secret-string file://key.txt

The trade is real and worth stating plainly. You gain a secret whose value has never been near a Terraform run, a plan file or a CI log. You lose the property that a fresh terraform apply produces a working environment — someone must place the value, and if they do not, the failure appears at runtime in the consuming application rather than at apply time. Document that step next to the resource, because the person hitting it will be reading this file.

If you go this route, do not attach a lifecycle block with ignore_changes to a version resource as a substitute. That keeps the value in state and merely stops Terraform reverting it, which is the worst of both.

The read policy

The secret is only as protected as the policy on it. Two actions matter and they are usually granted together when only one is needed: secretsmanager:GetSecretValue reads the current value, and secretsmanager:DescribeSecret reads the metadata without the value. An application needs the first; a health check or an inventory job needs only the second.

data "aws_iam_policy_document" "read_model_key" {
  statement {
    sid       = "ReadModelKey"
    actions   = ["secretsmanager:GetSecretValue"]
    resources = [aws_secretsmanager_secret.model_key.arn]
  }

  statement {
    sid       = "DecryptWithSecretsKey"
    actions   = ["kms:Decrypt"]
    resources = [aws_kms_key.secrets.arn]
    condition {
      test     = "StringEquals"
      variable = "kms:ViaService"
      values   = ["secretsmanager.${data.aws_region.current.name}.amazonaws.com"]
    }
  }
}

The KMS statement is the one that is forgotten until the first runtime failure. A secret encrypted with a customer-managed key requires the caller to hold kms:Decrypt on that key as well as the Secrets Manager permission — the read fails with an access denied that names KMS, not Secrets Manager. The kms:ViaService condition narrows that grant so the role cannot use the key to decrypt anything else.

Scope the resource ARN to the specific secret. A policy granting secretsmanager:GetSecretValue on * gives a service that needs one model key the ability to read every database password in the account — see scoping a Secrets Manager read policy for the narrower forms.

Reading it at runtime

Do not resolve the secret in Terraform and inject the value as an environment variable on the compute resource. A data.aws_secretsmanager_secret_version data source puts the value back in state, undoing everything above, and an environment variable is visible to anything that can describe the task or function.

Pass the ARN and let the runtime fetch it. On ECS, the task definition takes a secrets array mapping an environment variable name to a secret ARN, and the agent resolves it at container start using the execution role. On Lambda, call GetSecretValue at cold start and cache the result for the life of the execution environment — fetching per invocation adds latency to every request and pushes you toward the Secrets Manager request rate quotas for no benefit.

Whatever you do, keep it out of the image. A key baked into a container layer is present in every registry that ever pulled it and survives a rotation you thought was complete — why a secret baked into an image never goes away covers what that costs. And once rotation is on the table, triggering rotation is the resource that pairs with this one.