Redaction and Pseudonymisation in an AI Pipeline

Detecting an identifier is half the job. The other half is deciding what to put in its place — and the choice determines whether the model can still do the task, whether the answer can be handed back to the user intact, and whether you have reduced risk or merely moved it.

Engineering guidance, not legal advice. In particular, whether a given transformation is enough to change your obligations is a legal question with a specific answer for your situation, and the last section explains why the intuitive answer is usually wrong.

Four transformations, four uses

Removal

Delete the span. Cheapest, most destructive: the model loses the fact that anything was there, so a sentence about transferring an amount from one account to another arrives as a sentence with two holes in it and no indication that the holes were accounts. Use it only where the identifier is irrelevant to the task.

Masking

Replace with a marker that preserves the type — [CARD], [EMAIL]. The model knows what kind of thing was removed, which is often enough for classification and summarisation. Not enough when two different people appear in one message, because both become [NAME] and the relationship between them collapses.

Pseudonymisation

Replace with a stable, meaningless token — PERSON_1, PERSON_2 — held consistently within a request and, if you choose, across a conversation. The model can now reason about two distinct people, follow references, and produce an answer that talks about the right one. This is the version that preserves task quality, and it is the one worth implementing properly.

Generalisation

Replace with a coarser true statement: an exact date becomes a month, a postcode becomes a region, an age becomes a band. Useful where the value carries meaning the task needs but precision it does not, and the standard tool for quasi-identifiers that no token scheme handles well.

The round trip

Pseudonymisation is worth the effort only if the completion can be restored, or every answer comes back talking about PERSON_1 and your users hate it. The shape:

user text ──▶ detect spans ──▶ allocate tokens ──▶ prompt with tokens
                                     │                      │
                                     ▼                      ▼
                              map: token → value       model provider
                              (stays in your                 │
                               process / your                ▼
                               region, never sent)     completion with tokens
                                     │                      │
                                     └──────▶ rehydrate ◀────┘
                                                  │
                                                  ▼
                                          answer for the user

The single most important property of this diagram is the box that never moves: the map from token to real value stays on your side of the boundary. That is what makes the transformation meaningful. If the map is stored with the same processor that receives the tokens, you have built an elaborate no-op.

Implementing it

// Spans come from the detector: {start, end, type, raw}, sorted.
// Tokens are stable per distinct value within one context.

function pseudonymise(text, spans) {
  const forward = new Map();   // raw value  -> token
  const reverse = new Map();   // token      -> raw value
  const counters = new Map();  // type       -> next ordinal

  let out = "";
  let cursor = 0;

  for (const s of spans) {
    const value = text.slice(s.start, s.end);
    let token = forward.get(value);
    if (!token) {
      const n = (counters.get(s.type) ?? 0) + 1;
      counters.set(s.type, n);
      token = s.type.toUpperCase() + "_" + n;
      forward.set(value, token);
      reverse.set(token, value);
    }
    out += text.slice(cursor, s.start) + token;
    cursor = s.end;
  }
  out += text.slice(cursor);

  return { text: out, reverse };
}

// Rehydrate longest token first, so PERSON_10 is not eaten by PERSON_1.
function rehydrate(completion, reverse) {
  const tokens = [...reverse.keys()].sort((a, b) => b.length - a.length);
  let out = completion;
  for (const t of tokens) {
    out = out.split(t).join(reverse.get(t));
  }
  return out;
}

The sort in rehydrate is not a stylistic detail. Replace in allocation order and PERSON_1 matches the prefix of PERSON_10, producing an answer that attributes one person’s details to another — a privacy incident caused entirely by a replacement order. Longest-first, or use a token format that cannot prefix another, such as a fixed-width ordinal or a delimiter on both ends.

Where the map should live depends on scope. Request-scoped is simplest and safest: build it, use it, drop it, and it never touches a disk. Conversation-scoped tokens keep PERSON_1 meaning the same person across turns, which multi-turn tasks need — but now the map is stored, and a stored map of tokens to real identifiers is exactly as sensitive as the identifiers, needs the same encryption and retention rules, and is one more store that an erasure request has to reach.

Where the round trip breaks

• The model reformats the token. Case changes, hyphenation, or a token folded into a possessive can all defeat an exact-match rehydration. Choose a format models leave alone — uppercase with an underscore survives well — and log the rate at which tokens go missing between request and response, because that number is your correctness metric for the whole feature.
• Tokens are invented. A model given PERSON_1 and PERSON_2 will occasionally produce PERSON_3. Rehydration must leave unknown tokens alone rather than guessing, and something should notice, because an invented token usually indicates a hallucinated participant in the answer.
• Structured output. If the model returns JSON, tokens can land inside keys, or be split across a string boundary during streaming. Rehydrate after parsing, on the assembled values, not on the raw stream.
• Streaming. A token split across two chunks will not match. Buffer at least the maximum token length before emitting, or rehydrate on complete lines.
• Tool calls. A pseudonymised argument passed to a real API is a request for a record that does not exist. Decide explicitly whether tools are inside or outside the boundary; both are defensible, silently mixing them is not.

What pseudonymisation does not buy

Under GDPR, pseudonymised data is still personal data. The regulation treats it as a security and data-minimisation measure — a good one, explicitly encouraged — but not as an exit from scope, because the holder of the map can re-identify. Anonymisation, which would take it out of scope, is a much higher bar and is genuinely hard to achieve on free text where quasi-identifiers survive the transformation.

What it does buy is real and worth stating precisely: the third party in your chain receives less, a breach on their side exposes less, and the population of your staff who can see real identifiers shrinks. That is a meaningful reduction in risk. It is not a reclassification, and describing it as one in a customer questionnaire is the kind of overstatement that is easy to check and expensive to be caught on.