Testing That Log Redaction Runs Before a Failed Test Prints the Prompt

Somebody opens a CI log to see why a test failed and finds a customer’s support ticket, verbatim, in the assertion diff. The application logger redacts. The redaction was never involved: the content was printed by the test framework, out of an object it was handed, on a path nothing in your code controls.

The symptom

It appears in one of a few forms, all of them on failure and none of them on success, which is why it survives review:

• An assertion diff that prints both sides of a comparison, where one side is the rendered prompt.
• A thrown error whose message was built by interpolating the request body, so the stack trace carries it.
• A local-variable dump — pytest prints locals for a failing frame when run with --showlocals (or -l), and CI configurations commonly turn it on to make failures easier to read.
• A console.log in a catch block, written during debugging, which only executes when something goes wrong and therefore never appears in a passing run.
• A recorded cassette or a captured HTTP body attached as a CI artefact on failure.

The common factor: the redaction you wrote lives in the logging path your application code calls. None of the above goes through it.

It is worth being clear about what the exposure actually is, because it is larger than it first looks. A CI log is not a private artefact: it is readable by everyone with access to the repository, it is retained for weeks by default, it is frequently mirrored into a third-party CI provider’s own storage, and links to it get pasted into chat and into issue comments where they outlive the log itself. A prompt printed once into a failed run has therefore been copied to several places you did not choose, and no later fix removes it from all of them. That asymmetry — cheap to leak, expensive to unleak — is why this is worth a dedicated test rather than a code-review convention.

Why the failure path is different

A test framework’s job on failure is to tell you everything it knows. That is the correct default and it is directly opposed to what you need here. Vitest serialises the actual and expected values into a diff; pytest rewrites assertions specifically so it can show you the operands; both attach context deliberately and thoroughly. There is no hook you can add to your logger that intercepts it, because your logger is not on the path.

There are two defences and you want both. Do not put the sensitive value where the framework can find it, and prove on the failure path that the redaction ran. The first is a design rule; the second is the test this page is about, and it exists because the design rule is exactly the kind of thing that holds for eleven months and then does not.

Keep the payload out of the assertion

If the assertion subject is the prompt, the diff is the prompt. So assert on something derived from it that is stable and not sensitive:

// Bad: the whole rendered prompt is the assertion subject.
expect(rendered).toEqual(expectedPrompt);

// Better: assert the properties that the test is actually about.
expect(rendered).toHaveLength(expectedPrompt.length);
expect(sha256(rendered)).toBe(sha256(expectedPrompt));

// Best where the test is about structure: assert on a redacted projection,
// and make the projection the only thing any reporter can reach.
expect(summarise(rendered)).toEqual({
  sections: ["system", "tools", "examples", "user"],
  toolNames: ["lookup_order", "refund"],
  userTokens: 412,
  containsPii: false,
});

A hash comparison fails with two hex strings in the diff, which tells you the values differ and nothing else — that is a real loss of debuggability and it is the trade you are making. Mitigate it by making the projection rich rather than by weakening the rule: a diff of section names, tool names and token counts localises most failures without carrying content. Where you genuinely need the text to debug, write it to a file that is not a CI artefact and print the path.

The same reasoning applies to error construction. A custom error class with a toJSON that omits the payload, and a message built from ids rather than content, means the value cannot leak through a stack trace even when somebody catches and rethrows it three layers up.

The meta-test

Now prove it. The trick is to deliberately fail an assertion inside the test, catch the framework’s own error object, and assert about the string it would have printed. That inverts the usual direction: the subject under test is the failure output itself.

import { describe, it, expect, onTestFailed } from "vitest";
import { renderPrompt, summarise } from "../src/prompt";

const CANARY = "SSN-999-88-7777-CANARY";

function failureText(fn: () => void): string {
  try {
    fn();
  } catch (err: any) {
    // Everything a reporter has to work with.
    return [err.message, err.stack, JSON.stringify(err.actual), JSON.stringify(err.expected)]
      .filter(Boolean)
      .join("\n");
  }
  throw new Error("expected the assertion to fail, but it passed");
}

describe("failure output redaction", () => {
  it("does not carry raw prompt content into a failed assertion", () => {
    const rendered = renderPrompt({ ticket: `customer said ${CANARY}` });

    const text = failureText(() => {
      // The comparison a developer would naturally write, against the
      // projection rather than the prompt. Forced to fail.
      expect(summarise(rendered)).toEqual({ sections: ["definitely-not-this"] });
    });

    expect(text).not.toContain(CANARY);
    expect(text).not.toContain("customer said");
  });

  it("redacts on the reporter hook as well", () => {
    onTestFailed((ctx) => {
      const serialised = JSON.stringify(ctx.errors);
      expect(serialised).not.toContain(CANARY);
    });
    // ... the rest of a test that legitimately handles the canary ...
  });
});

Three notes on making this hold. The canary must be a string that could not appear by accident and that your redactor is not special-cased for — if the redactor only catches things matching a social-security pattern, add a second canary that is an ordinary sentence, because the risk is not only structured identifiers. The helper must fail loudly if the assertion passes, otherwise a refactor turns the whole test into a no-op that reports green. And run this test in the same configuration CI uses, including whatever verbosity and diff settings are set there: a diff truncated at 100 characters locally and unbounded in CI is two different tests.

The second belt: scan the artefact

The meta-test covers the paths you thought of. The paths you did not think of are covered by scanning the output after the fact: a CI step that runs after the test job, greps the captured log and any uploaded artefacts for the canary and for your organisation’s detector patterns, and fails the job if it finds anything. Run it with if: always() semantics so it executes on the failing runs, which are the only runs where it matters, and make it fail the build rather than warn — a warning about a leaked prompt is a leaked prompt.

One caveat worth stating: a scanner that finds nothing has proven nothing about the run where a different customer’s data was in the fixture. Treat it as the backstop and the meta-test as the control. The related problem — keeping a fixture scrubbed without destroying its value — is whether a redacted fixture still reproduces the bug, and the wider question of what belongs in a log at all is what to log.