Injecting Secrets Into a Kubernetes Pod From a Cloud Secret Manager

The External Secrets Operator reads a secret from a cloud provider and writes it into the cluster as an ordinary Kubernetes Secret. That is the whole feature, and understanding that the output is an ordinary Secret explains both why it is easy to adopt and why the rotation story surprises people.

What the operator does and does not do

ESO runs a controller that reconciles ExternalSecret resources. Each one names a store, names a remote key, and names a Kubernetes Secret to produce. The controller fetches, templates, and writes. The resulting Secret is indistinguishable from one you created by hand: same base64-encoded data, same namespace scoping, same RBAC.

The consequences follow directly. Anything that can read Secrets in the namespace can read your provider key, because ESO did not change the trust model — it changed where the value is authored. Encryption at rest is still whatever etcd encryption you configured. And the git repository now holds a resource that names a secret without containing one, which is the actual win: your manifests are safe to commit, reviewable, and diff-able, while the value lives in a store with audit logging and IAM.

Against the alternative of committing an encrypted value — see sealed secrets for a GitOps-managed key — the trade is custody. ESO keeps the value in one system of record and requires the cluster to be able to reach it; sealed secrets keep the value in git in encrypted form and require the cluster to hold a private key you must never lose.

Installing and authenticating

1. Install the operator via its Helm chart into its own namespace, with CRDs installed by the chart. Pin the chart version in your values file rather than tracking latest; the CRD group has already moved from v1beta1 to v1 once.
2. Give the controller an identity in the cloud account. On EKS that is IRSA or Pod Identity: an IAM role trusted by the cluster’s OIDC provider and annotated onto the controller’s service account. Do not use a static access key here — you would be storing a credential in the cluster in order to avoid storing a credential in the cluster.

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets --create-namespace \
  --set installCRDs=true \
  --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::123456789012:role/eso-reader

kubectl -n external-secrets rollout status deploy/external-secrets

The IAM role wants exactly two actions on exactly the secrets it should see: secretsmanager:GetSecretValue and secretsmanager:DescribeSecret, plus kms:Decrypt if the secret uses a customer-managed key. Scope the resource properly — the six-character ARN suffix trap on scoping a read policy to one secret applies to this role more than to any other, because this one is cluster-wide by construction.

The store and the ExternalSecret

A SecretStore is namespaced; a ClusterSecretStore is global and referenced from any namespace. Prefer the namespaced form when teams share a cluster, because a ClusterSecretStore plus a broad IAM policy means any namespace that can create an ExternalSecret can read any secret the controller can read.

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: aws-secretsmanager
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-west-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: model-provider-keys
  namespace: inference
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secretsmanager
    kind: ClusterSecretStore
  target:
    name: model-provider-keys
    creationPolicy: Owner
  data:
    - secretKey: OPENAI_API_KEY
      remoteRef:
        key: prod/providers/openai
        property: api_key

property selects one field out of a JSON secret, which is what you want when the stored value is an object rather than a bare string. creationPolicy: Owner makes the operator the owner of the target Secret, so deleting the ExternalSecret deletes the Secret — correct for GitOps, and a surprise if you expected the value to survive. Use dataFrom instead of data to pull every field of a JSON secret into keys of the same names.

Check that it reconciled before deploying anything that depends on it:

kubectl -n inference get externalsecret model-provider-keys \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].message}{"\n"}'
kubectl -n inference get secret model-provider-keys -o jsonpath='{.data.OPENAI_API_KEY}' | base64 -d | head -c 7

Why the pod does not see the new value

This is the section the vendor tutorials skip and the one that costs an afternoon. refreshInterval: 1h governs how often ESO re-reads the remote store and updates the Kubernetes Secret. It says nothing about the pod. Kubernetes has two ways to expose a Secret to a container and they behave completely differently:

• As environment variables (envFrom or valueFrom.secretKeyRef) the values are read once, when the container starts. A process’s environment cannot be changed from outside afterwards. Updating the Secret changes nothing in a running pod, ever, regardless of refresh interval.
• As a projected volume the kubelet updates the files in place when the Secret changes, on its own sync period. The application still has to re-read the file — a process that loaded it at boot is in exactly the same position as the env-var case.

So there are three workable designs. Mount as a volume and re-read the file on every use or on a timer; that is the only one where rotation is genuinely transparent. Or accept a restart and make it automatic, by hashing the secret into a pod annotation so a change rolls the deployment. Or, if the application must use environment variables, detect the 401 and exit, letting the restart pick up the new value — crude, but honest about what env vars can do.

Whichever you pick, set the client-side behaviour to match the rotation cadence on the storage side; the two halves are described together on automating key rotation with a Secrets Manager trigger.

The failures worth recognising

• SecretSyncedError with an access-denied message. The controller’s role cannot read that ARN. Check the ARN in the error against the policy resource, including the suffix.
• The Secret exists but is empty. Usually property naming a JSON field that does not exist. The remote fetch succeeded, so the condition may read Ready; check the actual data.
• A deleted ExternalSecret took the Secret with it and the workload started failing. That is creationPolicy: Owner working as designed. If you need the Secret to outlive its declaration, creationPolicy: Orphan is the field, and you should know you have now created something git no longer manages.
• Rate limiting from the cloud store. Hundreds of ExternalSecret resources with a short refresh interval make a lot of API calls. Lengthen the interval and consolidate related keys into one JSON secret pulled with dataFrom.