Migrating a Redaction Pipeline That Sits in Front of an LLM Call

A redaction step that has worked for a year against one provider is three separate contracts in a trench coat, and moving providers tests all three at once: the prompt format has to accept your placeholders, the model has to return them intact, and your rehydrator has to find them again.

The placeholder is an interface

The pipeline itself rarely needs to change. Detection, whether it is regex, a NER model or a hybrid, produces spans; you replace each span with a token and keep a mapping from token to original value for the duration of the request. What breaks on migration is the token, because it is consumed by three parties with different rules.

Choose a token shape with these properties and most of the rest of this page stops applying: uppercase, no whitespace, no characters with meaning in your templating layer or the target prompt format, stable numbering within a conversation, and unlikely to be translated or pluralised. PERSON_1 and ACCOUNT_2 qualify. {{name}}, [redacted] and <PERSON> all fail at least one of them.

Stable numbering is the property people skip. If the same individual becomes PERSON_1 in turn one and PERSON_3 in turn four, the model has no way to know they are the same person, and it will reason about them as two. The mapping therefore has to live for the conversation, not for the request — and that mapping table is itself a store of personal data with its own retention question, which is the part that most often escapes the DPIA.

Syntax collisions with the new prompt format

Curly braces are the classic. A token like {EMAIL} passing through Python’s str.format, an f-string, or a templating library raises a KeyError at best and silently substitutes at worst. Angle brackets are the next one: a token like <PERSON_1> in a prompt built with XML-ish structure gets read as a tag by whatever parses the prompt, and by the model, which has seen a great deal of XML and will happily close your tag for you.

The second collision is with the new provider’s message structure. If you are moving from an API where everything was one string to one with a distinct system parameter and a typed content block list, your placeholders now cross block boundaries — and a detection step that ran over the concatenated prompt will produce spans whose offsets mean nothing once the text is split. Redact per field, after the message structure is decided, not over a flattened string.

The third is the tokenizer. A token that was one or two tokens under the old vocabulary can be five or six under the new one, and a prompt with a hundred redacted entities pays that difference on every call. That is a small cost effect and a larger prompt-drift effect: few-shot examples that were tuned against one placeholder density behave differently at another. This is the general prompt portability problem arriving through a side door.

When the schema forbids the placeholder

This is the failure worth designing against explicitly, because it produces plausible wrong data rather than an error.

Suppose the extraction step now runs under a strict JSON schema, and the schema declares a field as a string with an email format constraint. The input contains EMAIL_1 where the address used to be. The model has to emit something that satisfies the constraint, and EMAIL_1 does not. So it emits an address — a well-formed, entirely fabricated one — and your validator passes it, and it lands in your database looking exactly like a real customer email.

The same happens with any constrained field: an enum that has no “unknown” member, a date field given a redacted date, a numeric field given a redacted account number. The rule is that any field which can receive a redacted value must be able to represent one. Either relax the constraint, or add an explicit sentinel to the enum, or — cleanest — declare those fields as plain strings in the schema and validate them yourself after rehydration, when the real value is back.

Rehydration fails silently

Exact string replacement on the way out assumes the model echoes your token unchanged. Models do not reliably do that. They lowercase it, they pluralise it, they translate it if the conversation is in another language, they insert it into a possessive, and occasionally they rewrite PERSON_1 as Person 1 because that reads better. Every one of those makes your replacement miss, and a miss is silent: the output goes out with a placeholder in it, or with a half-substituted string.

Fail loudly instead. Count the distinct placeholders you injected, count the ones you matched on the way out, and treat any mismatch as an error rather than a warning. Then add a second gate: scan the final output for anything matching your placeholder pattern before it leaves the process. A response containing an unrehydrated token is a bug; a response containing a token you never injected means the model invented one, which tells you the prompt is teaching it the pattern.

def rehydrate(text, mapping):
    out, matched = text, set()
    for token, original in mapping.items():
        pattern = re.compile(re.escape(token), re.IGNORECASE)
        if pattern.search(out):
            matched.add(token)
            out = pattern.sub(original, out)
    missing = set(mapping) - matched
    if missing:
        raise RehydrationError(sorted(missing))     # loud, not a warning
    if PLACEHOLDER_RE.search(out):
        raise RehydrationError("residual placeholder in output")
    return out

The migration, in order

1. Freeze the current behaviour as fixtures. Capture twenty real requests, redacted, together with the exact prompt string the old provider received and the response it returned. These are your before-and-after reference; without them every later difference is an argument.
2. Move redaction after message assembly. Redact each message field individually once the provider-shaped request object exists, so spans and offsets are computed against the text that is actually sent.
3. Normalise the token shape. Convert to uppercase underscore tokens if you are not already using them, and add a single compiled PLACEHOLDER_RE that both the injector and the output scanner share, so they cannot drift apart.
4. Audit the output schema for constrained fields. List every field with a format, enum or numeric constraint that could receive a redacted value, and relax or sentinel each one. This is the step that prevents fabricated values.
5. Add the two gates. Placeholder count in equals placeholder count matched; no residual placeholder pattern in the final output. Both raise.
6. Replay the fixtures against the new provider and diff the prompts byte for byte before you look at the responses. Most of what you will find at this stage is your own templating, not the model.
7. Check the mapping store’s lifetime. It holds plaintext personal data keyed by conversation. Give it a TTL no longer than the conversation, encrypt it at rest, and keep it out of the same log stream as the redacted prompts — a log that contains both is a log that contains neither redaction nor privacy.
8. Run a failure-path test. Force an exception between redaction and the call, and assert that nothing raw reaches the log — the case this test exists for.

What you end up with is a redaction step that is provider-shaped only at its edges, which is the property that makes the next migration cheaper than this one. What the provider was doing for you is worth reading alongside this, because anything it was doing is now your job.