Secrets Management for API Keys in a Prompt Test Suite

A prompt test suite needs a real credential to do anything real, and a pull request from a fork is the one place CI deliberately refuses to provide one. That refusal is correct. Designing around it is the work.

Why the fork has no key

GitHub’s documented behaviour is unambiguous: with the exception of GITHUB_TOKEN, secrets are not passed to the runner when a workflow is triggered from a forked repository, and that GITHUB_TOKEN is read-only. The reason is that a pull request is an offer of arbitrary code to run on your infrastructure. If a fork could see your secrets, the exploit would be one line in a test file, submitted by anyone, requiring no review.

There is a repository setting that turns this off — the option to send secrets to workflows from pull requests. Treat it as unavailable. It converts “anyone can open a pull request” into “anyone can read every secret in the repository”, and no amount of reviewer discipline compensates, because the code runs when the workflow starts rather than when a human approves it.

The same restriction applies to workflows triggered by Dependabot, which is worth knowing because a dependency bump is exactly the change you most want an eval suite to check: a provider SDK minor version can alter a default timeout or retry count.

What the contributor actually sees

The environment variable is not absent in a way anything notices. It is present and empty, or absent and read as an empty string, and the error surfaces at the first request rather than at startup. Depending on the SDK the contributor sees a client-construction error complaining that the API key option must be set and an environment variable was not found, or — worse — a plain 401 from the provider.

A 401 is the confusing one, because it looks like a credential problem the contributor could have caused. They will re-read their diff, then their local setup, then ask in the pull request. This is pure friction generated by your workflow, and the fix is to make the suite detect the condition and say so:

// eval/setup.ts — run before any case.
const key = process.env.PROVIDER_API_KEY;
if (!key) {
  throw new Error(
    "PROVIDER_API_KEY is not set. In CI this is expected on pull requests " +
      "from forks, which cannot receive repository secrets. Run " +
      "`npm run eval:replay` to execute the suite against recorded " +
      "responses, or ask a maintainer to run the live suite.",
  );
}

Better still, detect it in the workflow and skip the live job with an explanation, so the failure never appears as a red check at all:

  live-eval:
    if: github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest

Three ways to give forks a real signal

Skipping is honest but it means external contributions get no feedback, which is precisely when feedback is most valuable. Three patterns give a fork something real, in increasing order of what they cost you.

1. Run the suite against recorded responses. Commit cassettes of provider responses and have the fork’s run replay them. This catches everything about your code — parsing, schema handling, error mapping, prompt assembly — and nothing about the model. It needs no credential and no approval, so it is the default every fork should get. Its blind spot is real and worth stating in the contributing guide: a green replay run says your code is consistent with responses recorded on some past day.
2. Gate the live run behind a maintainer approval. Put the provider key in a deployment environment with required reviewers, and have the live job declare that environment. The job pauses until a maintainer approves, and only then does the runner receive the secret. This is the pattern to reach for when you want forks to get live results at all, because the approval happens before the job with access runs rather than after.
3. Run it from a trusted workflow after the fact. A workflow triggered by workflow_run executes in the base repository’s context with access to secrets, after the untrusted workflow completes. It works, and it is fiddly: it does not check out the pull request head by default, and if you make it do so you have reintroduced the whole problem.

Scoping the key you do use

The key CI holds should be a different key from the one production holds, and it should be the least capable key the provider will issue. Three properties are worth insisting on.

• Separately revocable. The whole point of a distinct CI key is that suspecting a leak leads to rotating one credential in one place, with no deploy. If CI and production share a key, the response to a suspected leak is an outage, so people hesitate — and hesitating is the failure.
• Capped. Where the provider supports a per-key spend limit, set one. A CI key with an account-wide limit is a key that can spend the month’s budget on a loop that a merge introduced at 17:00 on a Friday.
• Scoped to an environment, not the repository. Repository-level secrets are visible to every workflow in the repository, including a new one added in a pull request from a branch. An environment secret is only available to a job that names that environment, and the environment can carry the reviewer requirement.

Leak paths a test suite opens

A test suite exposes credentials in ways a deployed service does not, because its whole purpose is to record and print what happened.

• Masking is exact-match only. The runner redacts occurrences of a secret’s literal value in logs. A key that has been base64-encoded, URL-encoded, JSON-escaped, or split across a line break is no longer that literal string and is printed in full. Any test that dumps a serialised request object is a candidate.
• Verbose HTTP logging prints the Authorization header. Turning on client debug logging to diagnose a CI-only failure is the obvious move and it writes the credential into a build log that, on a public repository, anyone can read.
• Cassettes record headers. Any record-and-replay layer captures the request as sent, including Authorization. Python’s vcrpy takes filter_headers in its configuration for exactly this, and the equivalent exists in the other libraries — but it is opt-in, and the file gets committed. Grep your cassette directory for your key prefix before the first commit, and add a check that does it on every run.
• Failure output includes the request. A test that prints the full request on assertion failure is well-intentioned and prints headers with it. Redact at the point of serialisation, not at the point of printing.
• Artefacts outlive the log. An uploaded artefact containing raw responses is not masked at all. Set a short retention and strip credentials before upload.

The check that catches most of this is cheap: a step that scans logs and artefacts for your provider’s key prefix and fails the build on a hit. It costs a second and it converts the worst class of mistake here from a silent disclosure into a red build.