Using HashiCorp Vault for Model Provider Credentials

Vault’s headline feature is dynamic secrets: credentials created on demand with a lease and revoked when the lease expires. That does not apply to an OpenAI or Anthropic key, and starting from a clear view of why saves you from designing around a capability that is not there.

A provider key is a static secret

A dynamic secrets engine works by holding a privileged credential for some system and calling that system’s API to create a new short-lived credential per request — a database user, an IAM user, a signed certificate. The engine can do this because the target exposes a create-credential operation and a delete-credential operation.

Most model providers expose neither, or expose key creation only through a console. There is no upstream engine for them, and one could not be written without that API. So a provider key goes into the key-value engine as a static secret, and static KV secrets carry no lease: Vault stores and returns the value, and there is no expiry attached to it.

What Vault still buys you is worth listing precisely, because it is substantial and it is not the thing the marketing leads with. One place the value lives, with versioned history and a documented rollback. An audit device that records every read, with which identity and when — which is the property sealed secrets structurally cannot give you. Policy expressed against paths rather than against a cloud provider’s ARN grammar. And a short-lived token: the workload’s ability to read the secret expires, even though the secret does not. That is a real reduction in blast radius, and it is a different claim from “the key rotates itself”.

Storing it in KV v2

Mount the engine at a path that names what it holds, and keep provider keys separate from application config so a read policy can be narrow.

vault secrets enable -path=providers kv-v2

vault kv put providers/openai \
  api_key="$OPENAI_API_KEY" \
  owner="platform" \
  rotated_at="2026-08-11"

vault kv get -mount=providers openai
vault kv metadata get -mount=providers openai

The v2 engine’s path layout is the thing that trips people up in policy files. The CLI takes providers/openai, but the underlying API path is providers/data/openai for the value and providers/metadata/openai for its history. A policy written against the CLI-shaped path silently grants nothing:

# model-provider-read.hcl
path "providers/data/openai" {
  capabilities = ["read"]
}

path "providers/metadata/openai" {
  capabilities = ["read", "list"]
}

vault policy write model-provider-read model-provider-read.hcl

Set max_versions on the mount deliberately. KV v2 keeps prior versions, which is what makes rollback possible after a bad rotation, and also means an old key remains readable to anyone with the read capability long after you stopped using it. If you revoke keys at the provider on rotation — and you should — the retained versions are harmless history; if you do not, they are a set of live credentials with a longer reach than you intended.

Authenticating the workload

The workload needs to prove who it is without holding a credential, which is the bootstrapping problem every secret store has. In Kubernetes the answer is the pod’s service account token: Vault validates it against the cluster’s API and maps it to a role.

1. Enable and configure the Kubernetes auth method, pointing Vault at the cluster’s API server. Running Vault inside the cluster lets it use the in-cluster address.
2. Bind a role to a specific service account and namespace, attach the read policy, and set a token TTL. The TTL is the short-lived part of this design.

vault auth enable kubernetes

vault write auth/kubernetes/config \
  kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"

vault write auth/kubernetes/role/inference \
  bound_service_account_names=inference \
  bound_service_account_namespaces=inference \
  policies=model-provider-read \
  ttl=1h

Bind both the name and the namespace. A role bound only by service account name is satisfied by a service account of that name in any namespace, which in a shared cluster means anyone who can create a namespace can read your provider key. This is the same class of mistake as a cluster-wide sealed secret scope, and it is equally invisible in review.

Getting it into the process

The Vault Agent Injector adds a sidecar that authenticates with the pod’s service account, renders secrets to a shared in-memory volume, and keeps them refreshed. It is driven entirely by pod annotations (HashiCorp, Vault Agent Injector annotations):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference-worker
  namespace: inference
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "inference"
        vault.hashicorp.com/agent-inject-secret-openai: "providers/data/openai"
        vault.hashicorp.com/agent-inject-template-openai: |
          {{- with secret "providers/data/openai" -}}
          export OPENAI_API_KEY="{{ .Data.data.api_key }}"
          {{- end }}
    spec:
      serviceAccountName: inference
      containers:
        - name: worker
          image: registry.example.com/inference-worker:1.4.2
          command: ["/bin/sh", "-c"]
          args: [". /vault/secrets/openai && exec python -m worker"]

Two details in that template are the ones people get wrong. The path is providers/data/openai with the data segment, matching the policy. And the value is at .Data.data.api_key with data twice — once for the API envelope and once for the KV v2 wrapper. Without a custom template the injector writes both the data and the metadata of a KV secret into the file, which is why almost every real deployment supplies one.

The refresh story matches the injector’s behaviour rather than your expectations: for non-leased secrets such as KV v2, the agent re-renders on a static-secret render interval, and your process still has to re-read the file. Sourcing it into environment variables at start-up, as above, means a changed key needs a restart — the same constraint described on injecting secrets from a cloud secret manager, and for the same reason: a process’s environment is immutable from outside.

Where a real dynamic lease applies

The dynamic story does exist for one important case: a model hosted by a cloud provider you already have an engine for. Bedrock is authenticated with AWS credentials, and Vault’s AWS secrets engine mints those on demand with a lease, revoking them on expiry.

vault secrets enable aws

vault write aws/roles/bedrock-invoke \
  credential_type=assumed_role \
  role_arns=arn:aws:iam::123456789012:role/bedrock-invoke

# Returns access key, secret key, session token — with a lease.
vault read aws/creds/bedrock-invoke

This is the arrangement the row’s promise actually describes, and it is worth knowing which of your providers can be moved onto it. Anything reached through a cloud provider’s IAM — Bedrock, Vertex AI through workload identity federation, Azure OpenAI through managed identity — can use ephemeral credentials. Anything reached with a bearer token issued by a console cannot, and pretending otherwise is how a design ends up with a rotation mechanism that never runs.