Rotating a Provider Key Without Downtime

Key rotation causes downtime for exactly one reason: somebody revokes the old key while something is still using it. Every safe procedure is a way of guaranteeing that cannot happen, and the guarantee needs a measurement, not a wait. This page is the procedure, the measurement, and the case where the order has to be reversed.

The invariant

One sentence, and everything else follows from it:

A credential must be accepted by the provider before it is distributed to clients, and every client must have stopped using the old credential before it is revoked.

The consequence is that there must be a period during which both keys work. That period is the dual-key window, and its length is not a guess — it is bounded by the longest time any consumer might still hold the old value, which is the maximum of your config refresh interval, your secret cache TTL, your deploy rollout time and your longest-lived process. If you cannot state that number for your system, that is the actual finding of this page, and the section on caching is why.

The prerequisite most teams are missing is that the provider must let you have two live keys at once. Most do; some limit the count per account or per project. Check before planning a rotation, because a provider that permits only one key forces a different design — usually routing all traffic through a single component that can swap the key atomically, which is one of the arguments in building an internal model gateway.

The dual-key window, step by step

1. Make consumers key-agnostic first. Before any rotation, the application must read the key from the secret store or environment by role — PROVIDER_KEY_PRIMARY — and never reference a specific key. If any code path has a key literal, or a key named after its creation date, fix that and ship it before continuing. This step is the one that makes every future rotation uneventful.
2. Create the new key at the provider. Same scopes, same rate limits, same project. Label it with the date and the rotation ticket. Do not disable anything.
3. Write it to the secret store as the secondary. Two named entries: .../primary and .../secondary. Nothing reads the secondary yet.
4. Smoke test the new key out of band. One cheap request from an operator machine, against the same provider endpoint the service uses. A key that was created with the wrong project or scope fails here, at zero cost, rather than in production. This step takes ten seconds and catches the most common mistake in the procedure.
5. Promote: secondary becomes primary. Swap the two values in the secret store, or point the primary alias at the new version. Both keys are still valid at the provider — this only changes which one clients pick up.
6. Roll the fleet. Restart or signal every consumer so it re-reads. Wait out the longest cache TTL and the longest-running process. Note the wall-clock time you finished.
7. Verify the old key is unused. The next section. Wait for a clean observation window — 24 hours is a good default because it covers daily batch jobs, which are the things people forget.
8. Disable, do not delete. If the provider supports disabling a key, disable it and wait another day. A disabled key can be re-enabled in seconds; a deleted key cannot. This converts a missed consumer from an incident into a five-minute blip.
9. Delete, and record it. Note in the rotation ticket when the key was created, promoted and destroyed. That record is what an auditor asks for and what tells you next time how long the window really needs to be.

Proving the old key is unused

Step seven is the one people skip, and it is the only step that converts hope into evidence. There are three ways to get the evidence, in descending order of reliability.

# A key fingerprint safe to log. Distinguishes keys; reveals nothing.
import hashlib

def key_fingerprint(key: str) -> str:
    return hashlib.sha256(key.encode()).hexdigest()[:12]

# On every outbound provider call:
log.info("provider_call", extra={
    "provider": provider,
    "key_fp": key_fingerprint(active_key),
    "status": status,
})

# Then, before revoking:
#   count requests grouped by key_fp over the last 24h
#   the old fingerprint must be exactly zero

The usual survivors of an incomplete rollout are: a nightly batch job, a staging environment sharing the production key, a developer’s local .env, a webhook receiver in another repository, and a partner integration. The last one is the expensive one, because you cannot restart it — plan a longer window when a key is shared outside your own systems.

The cache that ruins it

Reading a secret on every request is slow and expensive, so almost every implementation caches it. That cache is what makes rotation take hours instead of seconds, and it is usually invisible: a module-level variable read once at import, which means the effective TTL is the process lifetime.

# The failure: read once at import, cached until the process dies.
API_KEY = secrets_client.get("prod/model-provider/primary")   # never re-read

# Better: bounded TTL, refreshed lazily, with a forced refresh on 401.
import time, threading

_lock = threading.Lock()
_cached = {"value": None, "expires": 0.0}
TTL_SECONDS = 300

def api_key(force: bool = False) -> str:
    now = time.time()
    with _lock:
        if force or _cached["value"] is None or now >= _cached["expires"]:
            _cached["value"] = secrets_client.get("prod/model-provider/primary")
            _cached["expires"] = now + TTL_SECONDS
        return _cached["value"]

def call_provider(payload):
    resp = post(url, key=api_key(), json=payload)
    if resp.status_code == 401:
        resp = post(url, key=api_key(force=True), json=payload)   # once, not a loop
    return resp

The forced refresh on a 401 is what turns a five-minute TTL into a self-healing system: a client holding a revoked key recovers on its next request instead of at the next deploy. Guard it so it retries once and does not become an authentication storm against your secret store — one retry, then fail and let the error surface.

Write the number down. “Our secret cache TTL is 300 seconds and our longest-running process is a 30-minute batch job, so the dual-key window is at least 30 minutes” is the sentence that makes the procedure above safe rather than superstitious.

Compromise reverses the order

Everything above optimises for zero downtime, which is correct for scheduled rotation. If the key has leaked, the priority inverts: the cost of the key remaining valid exceeds the cost of an outage, and the gradual procedure is exactly wrong.

1. Revoke first. Immediately, before anything else. Accept the errors.
2. Create and distribute the replacement. The procedure above, compressed, with the smoke test still in it.
3. Then investigate. Pull the provider’s usage log for the compromised key and look for calls you cannot account for: unfamiliar source addresses, models you do not use, volume spikes, unusual hours. Spend from a stolen key is the whole motivation for theft — denial of wallet covers what the abuse looks like.
4. Find the exposure path. A commit, a log line, a client-side bundle, a screenshot in a ticket, a third party’s breach. Rotating without finding it means rotating again next month.
5. Check the blast radius. If the key had broader scopes than the service needed, the incident is larger than the spend. Narrowing scopes is the fix that stops the next one being worse.

The tell that you have this backwards is a team that hesitates to revoke because they are unsure what will break. That hesitation is a symptom of the inventory problem in the verification section, not a judgement call.

Making it routine

Rotation that happens quarterly by hand is rotation that stops happening. Two mechanisms make it routine.

Schedule it, and make it boring. A recurring job that runs the create-promote-verify-revoke sequence on a fixed cadence, with the verification step gating the revoke and a human approval between promote and revoke. If it runs monthly, the procedure is exercised often enough that the emergency version is familiar.

Make the window observable. Alert when two keys have both been in use for longer than the planned window — that is a stuck rotation, and stuck rotations are how organisations end up with keys nobody dares touch.