Secrets Management for AI Applications

The generic advice — do not commit keys, use a secret manager, rotate — is correct and insufficient. Inference credentials leak through paths that a normal web application does not have, mostly because everything about a model call is interesting enough that somebody wants to log, trace or record it.

What is different about these keys

Three properties change the risk calculation. A stolen inference key is directly monetisable: an attacker does not need to pivot anywhere, because the key itself buys compute they can resell. Usage is metered, so the damage accrues continuously from the moment of the leak rather than at a single point. And the surrounding code deliberately captures the full text of every request and response for debugging and evaluation, which means there are more copies of the request in more places than in almost any other system you run.

That last one is the crux. A payments integration does not usually log the whole request body to three destinations. An AI application usually does.

Six leaks specific to AI applications

1. The key in the browser

Somebody calls the provider directly from client code, because the SDK works in the browser and the tutorial did it. The key is now in a bundle, and anything in a bundle is public regardless of how it was injected — a build-time environment variable is compiled into the output. The rule is absolute: the provider is called from a server you control. If you need per-user rate limiting on that server, that is a feature you build; it is not a reason to move the key.

2. The full request in the application log

Logging the complete outbound request during debugging is the fastest way to see what is wrong, and it puts the authorisation header into your log aggregator, where it is searchable, retained for months, and visible to everyone with log access — a much larger group than those with secret access.

3. Traces and observability payloads

LLM observability tooling captures prompts, completions, parameters and headers by design. It is genuinely useful and it is a third-party system holding your request stream. Check what it captures by default, confirm whether headers are included, and treat the trace store as having the same sensitivity as the data in the prompts.

4. Cassettes and fixtures in the repository

Recorded interactions carry whatever the request carried. A cassette that was recorded with a real key and committed puts the key in version-control history, where deleting the file does not remove it. Redact on the write path, and scan on commit.

5. The provider’s error, echoed to the user

Error handling that returns the upstream body verbatim can surface organisation ids, project ids, key prefixes and internal endpoints to whoever triggered the error. Normalise errors into your own taxonomy and return only your own codes — which you want to do anyway, for reasons that have nothing to do with security.

6. The prompt itself

Two directions here. Credentials placed into a prompt — a key inside a document being summarised, or a token in a tool definition — travel to the provider and into every log and trace on the way. And in an application where users influence the prompt, anything reachable by the model is potentially reachable by a user who asks the right way, so a system prompt is not a safe place to keep a secret. Give tools their credentials at execution time on your server; never through the model.

Redaction at the logger, not at the call site

Redaction implemented at each call site fails at the one call site somebody added last week. Put it in the serialiser, once, so that nothing can be logged without passing through it.

const SENSITIVE_KEYS = /^(authorization|x-api-key|api[-_]?key|cookie|set-cookie)$/i;
// Provider key formats change; match on shape as a backstop, not as the
// primary control. This catches a key that ended up somewhere unexpected.
const KEY_SHAPE = /\b(sk|rk|pk)-[A-Za-z0-9_-]{16,}\b/g;

export function redact(value: unknown): unknown {
  if (typeof value === "string") return value.replace(KEY_SHAPE, "[redacted]");
  if (Array.isArray(value)) return value.map(redact);
  if (value && typeof value === "object") {
    return Object.fromEntries(
      Object.entries(value).map(([k, v]) =>
        SENSITIVE_KEYS.test(k) ? [k, "[redacted]"] : [k, redact(v)],
      ),
    );
  }
  return value;
}

Then add a test that logs a synthetic key-shaped string through the real logger and asserts it does not appear in the output. It is a five-line test and it is the only thing that keeps the redactor honest as the logging stack changes.

Containing the blast radius

Assume a key will leak eventually and design for the aftermath. Four controls, in rough order of value per hour spent:

• A spend cap on every key. The single most effective control available, because it bounds the loss in money rather than in time-to-detection. A leaked key with a small daily ceiling is an annoyance; the same key with no ceiling is an open account.
• One key per environment and per service. Shared keys make revocation an outage, which means revocation gets delayed, which is the worst possible response to a suspected leak. Separate keys turn revocation into a non-event.
• Scoped or restricted keys where offered. A key limited to specific models or endpoints is worth less to an attacker and its misuse is more visible.
• Alerting on the shape of usage, not just the amount. A leaked key usually shows up as a new pattern — a model you never call, traffic at an hour you never run, a spike in a region you do not deploy to — before it shows up as a large invoice.

Rotation you have actually practised

Rotation policy is worthless if nobody has done it. The thing to build is not a schedule but the capability: two keys valid at once, the active one selected by configuration, so rotation is a deploy rather than a coordinated outage. Without dual-key support, rotating means a window during which some processes have the old key and some the new, and that window is why rotation gets postponed indefinitely.

Rehearse it. Rotate one key in a low-stakes environment on a schedule and time it. The number you want is how long it takes from “this key is compromised” to “this key is revoked”, and the only way to know it is to have done it when nothing was on fire. If the answer is measured in hours because a key is baked into three deployment configurations and a cron job, that is the finding, and it is a better one than any policy document.

Finally, write down what a leak response looks like before you need it: revoke first and investigate second, because the key is continuously spending money while you decide. Everything else in incident response can be careful; this step should be reflexive.