Storing a Model Provider API Key in AWS Secrets Manager

A provider key in a Lambda environment variable is readable by anyone with lambda:GetFunctionConfiguration, appears in CloudFormation drift output, and cannot be changed without redeploying the function. Moving it to Secrets Manager fixes all three, and the interesting part is what happens on rotation day.

Why not an environment variable

Lambda environment variables are encrypted at rest, which is the fact usually cited in their defence, and it is not the relevant one. The problems are structural:

• They are part of the function configuration. Any principal that can describe the function can read them, and that permission is routinely granted to read-only roles because it sounds harmless.
• Changing one is a deployment. UpdateFunctionConfiguration replaces the execution environments, so a key rotation becomes a cold start for every concurrent execution and has to move through whatever pipeline owns the function.
• They are duplicated per function. Five functions that call the same provider hold five copies, and rotating means finding all five.

The environment variable does not disappear, it changes content: it holds the secret’s name or ARN, which is not sensitive.

Storing the key

1. Create the secret with a structured value rather than a bare string. Provider credentials acquire companions — an organisation id, a project id, a base URL for a proxy — and a JSON object leaves room for them without a second secret.
2. Give it a name with a path, because Secrets Manager ARNs end in a random six-character suffix and IAM policies are much easier to write against a prefix.

aws secretsmanager create-secret \
  --name prod/inference/provider-key \
  --description "Model provider API key, prod" \
  --secret-string '{"apiKey":"sk-live-REPLACE","baseUrl":"https://api.example.com/v1"}'

The default encryption key is the AWS managed key aws/secretsmanager. Passing --kms-key-id with a customer managed key buys you two things worth having: a key policy as a second authorisation gate, and CloudTrail Decrypt events attributable to a principal. It also adds kms:Decrypt to the list of permissions the reader needs, which is the most common cause of a policy that looks complete and fails.

Granting the execution role

One action, one resource. Amazon’s own example grants exactly secretsmanager:GetSecretValue on the secret’s ARN:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "arn:aws:secretsmanager:us-east-1:111122223333:secret:prod/inference/provider-key-*"
    },
    {
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": "arn:aws:kms:us-east-1:111122223333:key/KEY-ID",
      "Condition": {
        "StringEquals": {
          "kms:ViaService": "secretsmanager.us-east-1.amazonaws.com"
        }
      }
    }
  ]
}

The trailing -* on the secret ARN absorbs the random suffix. The kms:ViaService condition means the role can use the key only through Secrets Manager, so a compromised role cannot decrypt arbitrary ciphertext with it. Drop the second statement entirely if you are using the AWS managed key.

Reading it without the SDK

Calling GetSecretValue on every invocation costs a network round trip and a request charge for a value that changes monthly. The usual fix is a module-level cache, which is correct but never expires. AWS publishes a layer for this — the AWS Parameters and Secrets Lambda Extension — that runs a local HTTP server beside your handler and caches for you.

The contract is small and worth memorising: it listens on localhost:2773, the path is /secretsmanager/get?secretId=NAME_OR_ARN, and every request must carry the header X-Aws-Parameters-Secrets-Token set to the function’s AWS_SESSION_TOKEN environment variable. That header is the authorisation: without it the extension refuses, which stops any other process in the environment from reading your secrets over the loopback.

import json, os, urllib.request

SECRET_ID = "prod/inference/provider-key"
ENDPOINT = "http://localhost:2773/secretsmanager/get?secretId=" + SECRET_ID

def get_key() -> str:
    req = urllib.request.Request(ENDPOINT)
    req.add_header("X-Aws-Parameters-Secrets-Token", os.environ["AWS_SESSION_TOKEN"])
    with urllib.request.urlopen(req, timeout=2) as resp:
        payload = json.loads(resp.read())
    return json.loads(payload["SecretString"])["apiKey"]

def lambda_handler(event, context):
    key = get_key()          # served from the extension cache, not the API
    ...

Note the standard-library HTTP call. The whole point of the extension is that it is runtime-agnostic and needs no dependency — using requests here, as several samples do, reintroduces a package to the deployment bundle to avoid one.

In a VPC, the extension needs a Secrets Manager interface endpoint. It is making the same API call the SDK would have made, from the same network, and a function with no route to the service will simply time out against localhost — a confusing symptom for a call that never left the machine. The same applies to Bedrock’s own endpoint.

Rotating without a redeploy

Secrets Manager versions a secret with staging labels: AWSCURRENT is what you get when you ask for no version, AWSPENDING is the candidate mid-rotation, and AWSPREVIOUS is the last one. A rotation moves the labels; it does not delete anything, which is what makes rollback a label move rather than an incident.

Most model providers have no rotation Lambda you can wire up, because their key-issuing APIs are not uniform. The honest workflow for a provider key is therefore four steps, and the reason it works without a deployment is the four-phase model rotation uses anyway:

1. Create a second key in the provider’s dashboard. Both keys are now valid — this overlap is the entire safety margin.
2. aws secretsmanager put-secret-value --secret-id prod/inference/provider-key --secret-string '...'. This creates a new version and moves AWSCURRENT to it.
3. Wait out the cache. With the default 300-second TTL, functions can serve the old key for up to five minutes after the change. Nothing is broken during that window; both keys work.
4. Revoke the old key at the provider. Not before step three.

That TTL is the number to reason about. Lowering SECRETS_MANAGER_TTL to 60 shortens the window at the cost of five times the API calls; setting it to 0 disables caching and turns every invocation into a request. And if you need the change to take effect immediately for a compromised key, append &versionStage=AWSCURRENT to the request path, which Amazon documents as the way to pin the version you want rather than accept whatever the cache holds. The broader question of how often to rotate provider keys at all is a policy decision this mechanism just makes cheap.