Sealed Secrets for a GitOps-Managed Model API Key

Sealed Secrets solves one problem precisely: a Kubernetes Secret cannot be committed to git, and a GitOps repository wants every resource committed. It solves it with asymmetric encryption, which means the interesting questions are all about the private key that never leaves the cluster.

What sealing actually guarantees

A controller runs in the cluster and holds a keypair. The kubeseal CLI fetches the public half and encrypts your Secret into a SealedSecret custom resource. Applying that resource causes the controller to decrypt it and create the corresponding Secret. Encryption is one-way from the outside: nobody with the repository and the public key can recover the plaintext.

Three things follow that determine whether this is right for you. First, the guarantee is about git, not about the cluster: once the controller unseals it, an ordinary Secret exists and anyone with RBAC to read Secrets in that namespace can read the key. Sealed Secrets is not a substitute for tightening that RBAC. Second, the cluster is the system of record for the private key, so cluster loss is key loss unless you planned otherwise. Third, there is no audit trail of reads, because after unsealing there is nothing left to audit — that is the property Vault and a cloud secret manager have and this does not.

Sealing a provider key

1. Install the controller into a namespace you will remember — conventionally kube-system — and note both the namespace and the controller name, because kubeseal needs them to find the public key.
2. Build the Secret without ever writing it to disk. Generate it with --dry-run=client and pipe it straight into kubeseal. A plaintext manifest that exists for thirty seconds in your working directory is a plaintext manifest that ends up in shell history, an editor swap file, or a stray git add -A.
3. Commit the sealed output and apply it through your GitOps controller like any other resource.

kubectl create secret generic model-provider-keys \
  --namespace inference \
  --from-literal=OPENAI_API_KEY="$OPENAI_API_KEY" \
  --dry-run=client -o yaml \
  | kubeseal \
      --controller-namespace kube-system \
      --controller-name sealed-secrets \
      --format yaml \
  > clusters/prod/inference/model-provider-keys.sealed.yaml

The result is a resource safe to read in a pull request:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: model-provider-keys
  namespace: inference
spec:
  encryptedData:
    OPENAI_API_KEY: AgBv1r8f...            # ciphertext, not base64 plaintext
  template:
    metadata:
      name: model-provider-keys
      namespace: inference
    type: Opaque

The template block is worth knowing about: it is copied verbatim onto the Secret the controller creates, so labels, annotations and the Secret type are set there. If you need a kubernetes.io/dockerconfigjson Secret rather than an Opaque one, that is where the type goes.

Scope is the security boundary

By default a sealed value is bound to both the name and the namespace in its metadata. Move it to another namespace, or rename it, and decryption fails — deliberately. Without that binding, anyone who could create resources in any namespace could take your ciphertext, apply it under a name their own pod mounts, and read the plaintext. The project documents three scopes, and the recommendation is to prefer the strictest that works (bitnami-labs/sealed-secrets on GitHub):

• strict — the default. Bound to name and namespace. Renaming requires re-sealing.
• namespace-wide — bound to the namespace only, so the Secret can be renamed within it. Useful when a chart generates the name.
• cluster-wide — bound to nothing. Any namespace, any name. Convenient for a value shared across namespaces, and the one to justify in review, because it removes the protection above entirely.

Scope is chosen at seal time with --scope and recorded as an annotation on the resource, so it is visible in the diff. A pull request that changes a sealed secret from strict to cluster-wide is a security change disguised as a one-line YAML edit; it is worth a lint rule.

Key custody and renewal

The controller renews its sealing key periodically — the project documents a default renewal interval of 30 days — and, crucially, does not delete the old keys. Every previous private key is retained so that existing SealedSecret resources continue to decrypt. New seals use the newest key. This is why a repository full of sealed secrets from two years ago still applies cleanly, and it is also why “the controller rotated its key” is not a rotation of anything meaningful from a security standpoint: nothing was re-encrypted and no old ciphertext became undecryptable.

Which makes the backup the operational centre of this system. The keys are Secrets in the controller’s namespace carrying a well-known label:

kubectl get secret -n kube-system \
  -l sealedsecrets.bitnami.com/sealed-secrets-key \
  -o yaml > sealed-secrets-keys.backup.yaml

That file is every secret in your repository in decryptable form. It is not a git artefact and not a wiki attachment; it belongs in the same class of storage as a root key, and the backup job that produces it should be as monitored as the one that backs up your database. Losing it is recoverable only by re-sealing every secret you still have the plaintext for — and the plaintext of a provider key you rotated six months ago exists nowhere.

Restoring into a new cluster

Rebuilding a cluster is the moment this design is tested. Applying the repository to a fresh cluster with a fresh controller produces SealedSecret resources that cannot be decrypted, and the symptom is workloads stuck without their Secrets while every manifest looks correct. The order that works is: install the controller, restore the key backup, restart the controller so it loads the restored keys, and only then let the GitOps controller sync.

kubectl apply -f sealed-secrets-keys.backup.yaml
kubectl -n kube-system rollout restart deploy/sealed-secrets
kubectl -n kube-system rollout status deploy/sealed-secrets

Rehearse it. A disaster recovery plan that assumes this works is untested until somebody has restored into an empty cluster and watched the Secrets appear — the same argument made at cluster scale on disaster recovery for an AI system. And if the answer to “who can restore the sealing key” is one person, that is the actual single point of failure, not the cluster.