Why Rotating an API Key Broke Every Test in CI

Every job in the matrix is red, all of them on the first request, and the only change was a scheduled key rotation. The body is short and unhelpful, so the diagnosis has to come from where the request went rather than from what it said.

The error you are looking at

On the Claude API a bad credential is a 401 with a typed envelope. Anthropic’s errors reference documents the shape: a top-level error object carrying type and message, alongside a request_id.

{
  "type": "error",
  "error": {
    "type": "authentication_error",
    "message": "invalid x-api-key"
  },
  "request_id": "req_011CSHoEeqs5C35K2UUqR7Fy"
}

Anthropic lists authentication_error at 401 for a key that is malformed, revoked or expired, and permission_error at 403 for a key that is valid but not allowed to touch the resource. That distinction does the first half of the diagnosis for you: a 403 after a rotation means the new key reached the API and is scoped wrongly, and nothing below about stale caches applies. See Anthropic’s errors documentation for the full status-to-type table. OpenAI returns a 401 with an error object whose code is invalid_api_key, and its message quotes a masked prefix of the key it received — which is the single most useful string in this whole investigation, because it tells you which key arrived.

Three checks that narrow it to one cause

1. Prove which key arrived, without printing it. Add a step that reports the length of the variable and its last four characters, and nothing else. A length of zero means the secret did not resolve at all — a scoping problem. A length that is one greater than you expect means a trailing newline. Last-four that matches the old key means a cache. Never echo the value: CI log masking only redacts strings the runner knows are secrets, and a substring or a base64-encoded form of the key is not masked.
2. Read the request id. Anthropic returns a request-id header on every response, exposed by the Python and TypeScript SDKs as _request_id on the response object and present in the error body. If you have a request id, the request left your network and reached the provider, so DNS, proxies and egress allowlists are innocent. If you do not, they are not.
3. Run the same call from a fresh shell with the new key. If it succeeds locally and fails in CI with the same key material, the key is fine and the delivery of it is not. If it fails in both, stop looking at CI — the key itself is wrong, revoked or scoped to a different workspace than the one you are calling.

Where a stale key survives a rotation

• Two secrets with the same name at different scopes. An organisation-level secret and a repository-level secret can share a name, and the narrower one wins. Rotating the one you can see in the organisation settings changes nothing if a repository secret from two years ago is shadowing it. This is the most common cause and it presents as “I definitely updated it”.
• An environment-scoped copy. Deployment environments each carry their own secret set. The test environment’s copy was never in the rotation runbook.
• A key baked into an image. If your test container is built with the key present at build time — an ENV line, a .npmrc, a pip.conf, a config file copied in — the value is in a layer, and the layer is cached. Rotating the secret has no effect until the image is rebuilt with a busted cache.
• A self-hosted runner with a file on disk. Somebody put the key in ~/.bashrc or a .env beside the checkout on the runner host to fix an urgent failure once. It has been shadowing the injected secret ever since, and it does not appear in any settings page.
• The workflow never had the secret. Pull requests from forks do not receive repository secrets, so the variable is an empty string and every request 401s. This looks like a rotation failure if the rotation happened to land the same week the first external contributor opened a PR.
• Whitespace. A key pasted with a trailing newline produces a malformed header value. Length check in step one catches this in seconds.

The one that is not a secret store at all

If your suite replays recorded HTTP interactions, the old key may be inside the recordings. VCR.py matches a request against a cassette on a configurable set of properties, and if the authorization header is part of that match — or if it was recorded unfiltered and a strict record mode is refusing to record anything new — a rotated key stops matching and every replay fails. VCR.py’s none record mode raises on any request that is not already in the cassette, which is exactly the error a non-matching header produces, and it looks nothing like an authentication failure.

The fix is the same as the hygiene rule: filter credential headers at record time so no cassette has ever contained a key, and keep the match on method, URI and body. A cassette that carries a live credential is a secret checked into version control regardless of whether it still works. See recording provider error responses as fixtures for the filtering rules to apply at record time.

Making the next rotation boring

Two changes remove most of this. First, resolve the credential in exactly one place — a single step that reads it and passes it forward — so that a shadowed secret is a visible conflict rather than a silent override. Second, add a preflight job that makes one cheap authenticated call before the suite runs and fails with a message naming the credential, its length and its last four characters. A suite of four hundred tests all failing on a 401 tells you nothing; one preflight failing with “credential resolved, 108 characters, ending 4f2a, rejected with authentication_error” ends the investigation before it starts.