Secrets Management When Running Two Providers at Once

A migration window is the only time an application holds working credentials for two providers at once, and it is exactly when the credential for one can be handed to a client pointed at the other. The structure that prevents it is small, and it has to be in place before the second provider is added.

Route and credential are one decision

The failure this section exists to prevent looks like this: a router picks a provider, and separately, further down, an HTTP client reads a credential from configuration. Two independent reads of what is logically one decision. When they disagree — because a flag flipped between them, or because the client was constructed at startup and the route is per request — you get an authentication error at best, and at worst a request to the wrong endpoint with a key that happens to work.

Make the resolver return everything the call needs, together, as one value:

@dataclass(frozen=True)
class Binding:
    provider: str        # "acme"
    base_url: str
    model: str           # provider-specific model identifier
    credential_ref: str  # a name, not a secret
    auth_style: str      # "bearer" | "header" | ...
    auth_header: str     # "Authorization" | "x-api-key" | ...

def resolve(ctx: RequestContext) -> Binding: ...

Everything downstream takes the binding and nothing reads configuration on its own. The secret itself is fetched from the store by credential_ref at the point of the call, so the secret value has the shortest life and the narrowest scope of anything in the path, and never enters a log, a trace attribute or an exception context that carries the binding.

The auth_style and auth_header fields are there because the credential-attaching step is genuinely provider-specific: a bearer token in an Authorization header, a vendor-named header, or a query parameter are all in use across providers, and some require an additional version header alongside. Confirm the exact header names against each provider’s current authentication documentation rather than carrying a convention across.

A naming scheme that survives a second provider

Most codebases start with one flat variable per secret, named after the vendor. That is fine for one provider and becomes ambiguous at two, because there is now a second axis — which environment, which tenant, which key generation — and flat names encode it by convention.

Give the reference a fixed structure with a fixed number of segments, so that a missing segment is a parse error rather than a name that resolves to something else:

llm/<environment>/<provider>/<purpose>/<generation>

llm/prod/acme/chat/2
llm/prod/acme/embeddings/2
llm/prod/borealis/chat/1
llm/staging/borealis/chat/1

• Environment first, so store-level access policy can be written as a prefix rule — production workloads can read llm/prod/* and nothing else, and a staging service literally cannot read a production credential.
• Purpose separately from provider, so a chat key and an embeddings key can be rotated and scoped independently. They often have different spend profiles and different blast radii.
• A generation counter, so a rotation is a new path rather than an overwrite. The rotation described in handling API key rotation during a provider migration depends on the old value still existing.

If you must expose these as environment variables, derive the names from the same structure mechanically rather than hand-writing them, so that the mapping cannot drift between two services. And keep the number of things a human types to one — the environment — because every other segment is derivable.

Selecting by environment, tenant and share

During a migration the route is a function of more than configuration. The usual inputs are environment, an explicit per-tenant override, and a traffic share for everyone else.

def resolve(ctx) -> Binding:
    if ctx.env != "prod":
        return BINDINGS[ctx.env][DEFAULT_PROVIDER]
    if ctx.tenant in PINNED:                 # explicit overrides win
        return BINDINGS["prod"][PINNED[ctx.tenant]]
    bucket = digest64(f"{ctx.tenant}") % 100  # stable, not random
    prov = "borealis" if bucket < ROLLOUT_PCT else "acme"
    return BINDINGS["prod"][prov]

The important detail is the hash. Choosing per request at random means a single user’s consecutive requests land on different providers, which makes any quality complaint unreproducible, splits conversation state across two systems, and halves the hit rate of any provider-side prompt cache. Hashing a stable identifier gives you a fixed cohort: the same tenants are on the new provider all day, they can be named, and increasing the percentage only ever adds tenants rather than reshuffling them. Use a hash function that is stable across processes and releases — not the language’s built-in string hash, which may be randomised per process.

Keep an explicit pin list above the percentage. You will need to move one complaining customer back immediately without changing the rollout, and you will need to move your own internal tenant forward first. Both are pins, and a rollout without them turns every individual decision into a global one.

Per-tenant keys

If tenants supply their own provider credentials, the same resolver answers the question, with one extra lookup and several extra obligations.

• Encrypt per tenant, not per table. Envelope encryption with a per-tenant data key means a single leaked ciphertext or a mis-scoped query cannot expose everybody.
• Never return the value. Store a display fragment — the last four characters — for the UI, and make the full value write-only through the API. A settings screen that shows a saved key back to the user is a key that will be copied into a support ticket.
• Validate on save with the cheap probe, and record when it last worked. A tenant key that was revoked on the provider’s side fails at request time otherwise, and it will look like your outage.
• Decide what happens when it fails. Falling back to your own credential when a tenant’s key is rejected means billing their usage to you, silently. That may be the right product decision, but it has to be a decision, and it should be visible in usage data rather than only in the invoice.

When the secret store is unavailable

A resolver that fetches from a secret manager on every request has made that manager a hard dependency of every inference call. That is usually the wrong trade, and the fix is a short-lived in-process cache: hold the resolved secret for a few minutes, refresh in the background, and on a fetch failure keep serving the cached value rather than failing the request.

Two consequences to accept deliberately. Rotation propagates on the order of the cache TTL rather than instantly, which is the argument for the overlap window in the rotation page rather than an argument against caching. And a compromised key stays usable for the TTL after you delete it from the store — which is why revocation at the provider, not deletion from the store, is the action that actually stops a leaked credential. Deleting the secret makes your services stop using it; revoking it makes everyone stop using it.

Redaction and blast radius

Two habits cover most of the remaining exposure.

First, redact at the boundary rather than at the call site. Exception handlers that print a request object, HTTP client debug logging, and crash reporters that serialise local variables will all happily emit a header dictionary. Keep the secret out of any object that gets logged in the first place — attach it inside the client’s send path — and add a scrubber over your log pipeline as the second line of defence, not the first. Test the scrubber by asserting that a synthetic key-shaped string does not survive a deliberately triggered error.

Second, make each credential as narrow as the provider allows. Separate keys per environment, per purpose and per service means that the answer to “what do we rotate” after an incident is a short list rather than everything, and that a compromised staging key cannot spend production budget. Where a provider supports restricted or read-scoped keys, use them for anything that only needs to read usage or list models. The cost of many narrow keys is a naming scheme, which you have from the section above.