Keeping Prompt Content Out of CI Logs When a Test Fails

Nobody decides to print a customer’s document into a build log. It happens because a test framework’s job on failure is to show you both sides of the comparison, and one of those sides is a prompt with a real document embedded in it.

How it gets into the log

• The diff printer. A failed equality assertion prints expected and actual in full. If the actual value is an assembled request object, that is the system prompt, every tool schema and every retrieved chunk, rendered as a diff with a few lines of colour around it. Nothing is truncated by default in most runners, and a 12,000-token prompt produces a great deal of output.
• Exception representation. An unhandled error carries a traceback, and several test runners render local variables in the frame. A helper that holds the request in a local variable puts the whole thing in the traceback. Some SDK exception types also include the request body in their string form so that a developer debugging interactively can see it.
• Debug logging left on. Most provider SDKs have an environment variable that logs full request and response bodies. Somebody turns it on to chase a bug, commits the workflow change, and it stays on for a year.
• Snapshot files. A snapshot of a request is a file in the repository containing the prompt, which is a stronger form of the same problem: it is not a log that ages out, it is committed content.
• Uploaded artifacts. A job that uploads its temporary directory on failure uploads whatever fixtures and intermediate files were there, and artifact retention is usually longer than log retention.

Who can read a build log

The threat model is worth stating because it is broader than people assume. On a public repository, build logs are public — anyone, permanently, including the archived copy somebody scraped. On a private one, log access typically comes with read access to the repository, which is a much larger group than the group approved to see production data, and it usually includes contractors and every service integrated with the repository. Logs are retained on a schedule you did not choose and are replicated into whatever observability tool the organisation pipes them to.

Secret masking does not help here. A CI runner redacts strings it knows are secrets; a customer’s address in a retrieved document is not registered as a secret and will never be masked. And if the content is personal data, its presence in a log is a processing location nobody documented, which is a compliance question rather than a tidiness one.

Assert on a projection

The pattern that fixes this is to never compare the raw objects. Compare a projection: a small derived value that keeps everything you need to diagnose a failure and none of the content. A good projection is deterministic, small enough to read in a terminal, and specific enough that a difference in it tells you what changed.

import { createHash } from "node:crypto";

export function summarise(request: AssembledRequest) {
  return {
    templateVersion: request.templateVersion,
    model: request.model,
    toolNames: request.tools.map((t) => t.name).sort(),
    messageRoles: request.messages.map((m) => m.role),
    // Length and digest, never content.
    systemChars: request.system.length,
    systemDigest: createHash("sha256").update(request.system).digest("hex").slice(0, 12),
    chunkCount: request.chunks.length,
    chunkChars: request.chunks.map((c) => c.length),
    chunkIds: request.chunks.map((c) => c.id),
  };
}

// The assertion the runner will print on failure:
expect(summarise(actual)).toEqual(summarise(expected));

A failure now prints something like a differing system digest and an identical everything-else, which localises the bug immediately and reveals nothing. Digests are the workhorse: two truncated hashes that differ tell you the text changed, and a chunk id list that differs tells you the retriever changed its mind — both of which are the actual diagnosis, and neither of which requires the words.

The same idea applies to the response. Assert on the parsed object’s structure, field names, types and whichever few values are genuinely not sensitive — a status enum, a confidence bucket, a token count. If you must assert a sensitive value came back correctly, assert on its digest against a digest stored in the fixture.

When you genuinely need the content

Sometimes the projection is not enough and somebody has to read the actual prompt to understand a failure. Handle that with a deliberate escape hatch rather than by removing the projection.

1. On failure, write the full request and response to a file in a directory reserved for it, and print only the path and the digest to the log.
2. Upload that directory as an artifact only when an explicit flag is set on the run, with the shortest retention your CI product allows. The default run uploads nothing.
3. Gate the flag on the same access control that governs the data. “Anyone who can rerun the job” is usually too broad, and the people who can approve it are usually the people who already have production access.
4. Prefer reproducing locally against a redacted fixture. If the bug only reproduces with real content, that is itself a finding — it means the failure depends on data your fixtures do not represent, which is a gap in the fixture set.

Redacted by construction

The durable version of this is to make it impossible rather than careful. If no fixture in the repository contains real customer content, then a full dump on failure is harmless and none of the discipline above is load-bearing. That means synthetic fixtures with realistic structure — realistic length distributions, realistic unicode, realistic messiness — and a strict rule that recorded traffic is redacted at capture time, the same rule that applies to recorded provider responses.

Add one cheap guard to hold the line: a test that scans the fixture directory for patterns that should never appear — email addresses, card-shaped digit runs, national identifier formats, anything matching your customer id scheme — and fails the build. It will catch the well-meaning debugging session where somebody pasted a real ticket into a fixture to reproduce something, which is how real data gets into a test suite roughly every time it happens. What belongs in a log at all, in tests and in production, is the subject of what to log.