Testing That a Redacted Fixture Still Reproduces the Original Bug

A customer request triggers a bug. You capture it, scrub the personal data, commit it as a regression fixture — and it passes against the unfixed code. The redaction removed the thing that broke it, and you now have a test that guards nothing and looks like it guards something.

The tension, stated precisely

A fixture has two jobs that pull against each other. It must be safe to commit, which means no personal data, no credentials and nothing that identifies a customer. And it must be faithful, which means it still exercises the code path it was captured for. The reason these conflict is not that the bug is “about” the personal data — it almost never is. It is that redaction changes measurable properties of the input, and the bug is usually about one of those properties.

Nobody notices, because the failure mode is silent in the friendly direction. The test passes. A test that passes for the wrong reason produces no alert, no flake and no investigation, and it will sit in the suite until the same bug recurs in production and somebody says “but we have a regression test for that”.

The properties redaction destroys

Replacing every name with [REDACTED] changes far more than the name:

• Length and token count. The most common casualty, and the one behind most context-window and truncation bugs. A 4,100 token conversation that crossed a limit becomes 3,600 tokens and no longer crosses it. Any bug about a boundary — a truncation point, a chunk split, a summarisation trigger — is a bug about length.
• Script and character class. If the input was Japanese and the placeholder is ASCII, the tokeniser behaves completely differently, and any bug involving encoding, byte length, normalisation or right-to-left handling evaporates.
• Structure inside the value. A redactor that flattens a value to a single token removes newlines, quotation marks, backticks, braces and the stray ``` fence that was breaking your JSON parsing. Delimiter-collision bugs and injection-shaped input are destroyed by exactly this.
• Cardinality and repetition. Replacing forty distinct names with forty copies of one placeholder turns a deduplication or grouping bug into nothing.
• Nullness and shape. A redactor that drops a field rather than replacing it changes the JSON shape, and “this field is present but empty” is a different case from “this field is absent” — frequently the case that broke.
• Ordering and timing. Scrubbing timestamps to a constant collapses a race or an out-of-order-events bug into a straight line.

Property-preserving surrogates

The technique that works is to substitute rather than erase, choosing a replacement that matches the original on whichever properties the bug depends on. This is more work than a regular expression and it is the difference between a fixture and a decoration.

• Same length, same script. Replace a Japanese name with a different Japanese name of the same character count, not with [NAME]. Fake-data generators do this reasonably well per locale; the important part is that you asked for the locale.
• Same token count. Verify it, do not assume it. Run both versions through the tokeniser you actually use and assert equality; a same-length replacement can still tokenise differently, and for a boundary bug the token count is the property that matters.
• Stable pseudonyms. Map each distinct original to a distinct surrogate, consistently, so forty names remain forty names and the same person is the same person in every message. A keyed hash into a name list does this and is reproducible.
• Preserve the awkward characters. If the value contained a newline, a quote or an unbalanced brace, the surrogate contains one too. This feels wrong and is the entire point when the bug was a parsing bug.
• Redact the value, keep the shape. Never delete a key. An empty string, a null and a missing key are three different inputs.

Two tests pulling in opposite directions

The fixture is only trustworthy when both of these exist, and they are written to fail for opposite reasons.

The reproduction test. It must be demonstrated, not assumed, that the redacted fixture still triggers the fault. The honest way is to run it against the unfixed behaviour once and record that it failed — typically by keeping the fix behind a flag during the fixing commit, asserting the fixture fails with the flag off and passes with it on, then removing the flag. If a flag is not practical, the substitute is a property assertion that pins the thing that mattered: expect(countTokens(fixture.input)).toBe(4103) next to a comment naming the 4,096-token boundary. That does not prove reproduction, but it fails loudly if a later scrub or reformat changes the property, and it records why the number is the number.

The safety test. A separate test asserts the fixture contains nothing sensitive: run your detector patterns over every committed fixture, plus a check that none of the known original values survive. Run it over the whole fixture directory rather than per fixture, so a file added by hand next month is covered by construction.

import { describe, it, expect } from "vitest";
import { readFileSync, readdirSync } from "node:fs";
import { countTokens } from "../src/tokens";

const fixture = JSON.parse(readFileSync("fixtures/bug-4102-truncation.json", "utf8"));

describe("bug-4102 fixture", () => {
  // Why this fixture is this size. The bug was a JSON body truncated at the
  // 4,096-token boundary; a scrub that shortens the input disarms it.
  it("still crosses the boundary that caused the bug", () => {
    expect(countTokens(fixture.input)).toBeGreaterThan(4096);
    expect(countTokens(fixture.input)).toBeLessThan(4200);
  });

  it("preserves the unbalanced brace that broke the parser", () => {
    expect(fixture.input).toContain('{"items": [');
    expect(fixture.input.endsWith("}")).toBe(false);
  });

  it("keeps every field the original had, including the empty one", () => {
    expect(Object.keys(fixture.input_json)).toEqual(
      ["customer", "locale", "notes", "attachments"],
    );
    expect(fixture.input_json.notes).toBe("");
  });
});

describe("fixture safety", () => {
  const files = readdirSync("fixtures").filter((f) => f.endsWith(".json"));
  it.each(files)("%s contains no detector hits", (f) => {
    const text = readFileSync(`fixtures/${f}`, "utf8");
    for (const [name, pattern] of Object.entries(DETECTORS)) {
      expect(text, `${name} matched in ${f}`).not.toMatch(pattern);
    }
  });
});

Automated detectors are a floor and not a ceiling: they catch card numbers and email addresses and they do not catch a free-text sentence that identifies somebody by circumstance. Where a fixture came from a real customer, a human should have read it, and the review is worth recording next to the file.

When the fixture cannot be saved

Sometimes the bug really is about the content in a way no surrogate preserves — a specific document, a specific interaction between two customers’ records, something you are not permitted to transform and keep. Say so and choose deliberately rather than shipping a fixture that has quietly stopped working.

The alternatives, in rough order of preference: write the test as a synthetic case that constructs the same property from nothing, which is usually possible once you know which property it was and is the reason identifying it matters so much; keep the real fixture in a restricted store and run it in a separate job with tighter access, at the cost of it not being reproducible by everyone; or, if neither is available, delete the fixture and keep the note. A written explanation of what broke and why it cannot be tested is more useful to the next person than a green test that proves nothing. The choice between a recorded artefact and a constructed one recurs constantly here — see when to record and when to hand-write — and the same scrubbing question applies to your logs on the failure path, in testing that redaction runs before a failure prints.