Storing a Provider API Key in Google Secret Manager

Cloud Run can mount a Secret Manager secret two ways, and the difference is not cosmetic: one is resolved once when the instance starts, the other is read from Secret Manager on use. If you want rotation without a redeploy, only one of them does that.

Creating the secret

A secret is a container with versions. The value lives in a version; the secret itself is just a name and a replication policy. Create it with automatic replication unless you have a data-residency reason to pin locations, in which case pin them — you cannot change a secret’s replication policy afterwards.

gcloud secrets create openai-api-key \
  --replication-policy=automatic \
  --labels=owner=platform,rotation=quarterly

# add the value from a file, not from your shell history
printf '%s' "$KEY" > /tmp/key && \
  gcloud secrets versions add openai-api-key --data-file=/tmp/key && \
  shred -u /tmp/key

Do not pass the value with --data-file=- after echoing it, and do not put it in a command line argument. Both end up in shell history, and on a shared build machine both end up in the process list where any other process can read them. This is a small point that has caused real incidents.

It is worth being precise about what this buys you over the obvious alternative, because “use Secret Manager” is usually asserted rather than argued. A key passed with --set-env-vars is stored in the revision spec as plain text. Anyone who can read the service — roles/run.viewer, which is routinely granted across a team — can print it:

gcloud run services describe inference-svc --region=us-central1 \
  --format='value(spec.template.spec.containers[0].env)'

It is also in your deploy command, and therefore in the CI job’s log, and in the Cloud Audit Log entry for the deploy, indefinitely. A secret reference stores only the secret’s name and version in the revision, so all three of those places hold a pointer rather than a credential. That is the actual difference, and it is a large one.

Granting the runtime service account

Grant on the secret, not on the project. A project-level roles/secretmanager.secretAccessor gives the service every secret in the project, which defeats most of the point of having separate secrets:

gcloud secrets add-iam-policy-binding openai-api-key \
  --member="serviceAccount:svc-inference@PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/secretmanager.secretAccessor"

Note which identity that is. It is the service’s runtime service account — the one you pass to --service-account — not your own account and not the Cloud Build account. If the grant is missing, the revision fails to start and reports the generic container-failed-to-start error rather than anything about permissions, which is a genuinely misleading symptom and worth recognising on sight.

Two mount types, and only one rotates

Google’s Cloud Run secrets documentation states the difference plainly. Secrets exposed as environment variables are resolved at instance startup time, and Google recommends pinning those to a specific version rather than using latest. Secrets mounted as a volume are fetched from Secret Manager, so the latest version is used — and the documentation says this works well with secret rotation.

The mechanism follows from where the read happens. An environment variable is a fixed string in the process environment; there is no way for it to change while the process runs, so the value has to be resolved when the instance boots and it stays that way for the life of the instance. A volume is a filesystem path, and a read from a path can return something different than it did last time.

# environment variable — pin the version, because it will not change
gcloud run deploy inference-svc \
  --image=us-docker.pkg.dev/PROJECT_ID/repo/inference:1.4.0 \
  --region=us-central1 \
  --service-account=svc-inference@PROJECT_ID.iam.gserviceaccount.com \
  --update-secrets=OPENAI_API_KEY=openai-api-key:4

# volume mount — 'latest' is the point
gcloud run deploy inference-svc \
  --image=us-docker.pkg.dev/PROJECT_ID/repo/inference:1.4.0 \
  --region=us-central1 \
  --service-account=svc-inference@PROJECT_ID.iam.gserviceaccount.com \
  --update-secrets=/secrets/openai/key=openai-api-key:latest

Note the asymmetry in what latest means. On the env-var form it means “whatever was latest when this instance booted”, which is genuinely worse than a pin because it is non-deterministic across a fleet: instances started before a rotation hold the old value and instances started after hold the new one, indefinitely, with no event marking the difference. That is the failure Google’s recommendation is aimed at.

Two mechanical details about the volume form save an afternoon. The mount path is a directory that Cloud Run creates, and mounting over a path your image already populates hides whatever was there — so mount at a dedicated path such as /secrets/openai rather than into /app/config next to files the application also needs. And the mount is read-only: an application that expects to rewrite its own config file in place will fail on the write, which surfaces as a permission error on a path you own and reads as nonsense until you know why.

Reading it in the service

The volume form means the application reads a file. Read it per use rather than caching it at import, or you have rebuilt the env-var behaviour in your own code:

import functools
import time
from pathlib import Path

KEY_PATH = Path("/secrets/openai/key")


@functools.lru_cache(maxsize=1)
def _cached(bucket: int) -> str:
    return KEY_PATH.read_text().strip()


def api_key() -> str:
    # Re-read at most once every 60 seconds. A file read from a
    # tmpfs-backed mount is cheap, but not free on a hot path.
    return _cached(int(time.time()) // 60)

Sixty seconds is a choice, not a rule. The trade is how long a compromised key stays usable after you disable it against how often you touch the filesystem. For a key rotated on a schedule, minutes is fine; for a key you may need to revoke in an incident, seconds is the number you want and the read cost is still small.

Rotating without a redeploy

1. Add the new value as a new version. The old version stays enabled, so both keys work during the overlap:

   gcloud secrets versions add openai-api-key --data-file=new-key.txt

2. Wait past your cache interval and confirm the service is using the new key — from the provider’s side if it exposes per-key usage, or by a request tagged in your own logs.
3. Disable the old version rather than destroying it. Disabling is reversible for as long as you need it to be:

   gcloud secrets versions disable 4 --secret=openai-api-key

4. Destroy it once you are certain, which is irreversible:

   gcloud secrets versions destroy 4 --secret=openai-api-key

No gcloud run deploy appears in that sequence, and no new revision is created. That is the property worth having: rotation becomes an operation on the secret rather than a deployment, which means it can be done during an incident by somebody who does not have deploy rights. The wider procedure, including the provider-side half, is rotating a provider API key without downtime.

Where this still goes wrong

A key baked into an image is permanent. If a key was ever in a Dockerfile, a build argument or a committed .env, it is in a layer in Artifact Registry and in every cached copy of that layer. Deleting the file in a later layer does not remove it. That key must be rotated at the provider; there is no cleanup that makes it safe.

Logs and error reporting. A key read into a variable gets into a stack trace, a request dump, or a debug log eventually. Do not include it in anything you serialise, and treat a header-dumping middleware as a secret-handling component.

The secret is still readable by anyone who can deploy. A principal with roles/run.developer can deploy a revision that mounts the secret and prints it. Secret Manager protects against a leaked image and a leaked repository; it does not protect against someone with deploy access to the service, and no mount type changes that.