Privacy-Safe Synthetic Data From Real Records

The pitch for synthetic data in regulated settings is that a generated record corresponds to no real person, so the output is not personal data. The pitch is sometimes true. It is not true by construction, and the published work is specific about when it fails.

The claim being made

Fit a generative model to a table of real records — a copula, a Bayesian network, a GAN, a diffusion model over tabular data — then sample from it. The rows that come out are draws from an estimated distribution. No row is a copy of a real row, so the argument goes, no individual is identifiable, and the result can be shared with a partner, a vendor or a research team without the consent and minimisation machinery that real data requires.

Notice what that argument quietly assumes: that the model learned the distribution and not the records. A model with enough capacity relative to its training set can do both, and the distinction between “generalised” and “memorised” is not visible from looking at the output. It has to be tested for.

What the attack literature found

The most directly relevant result is Stadler, Oprisanu and Troncoso, Synthetic Data – Anonymisation Groundhog Day, published at USENIX Security 2022. They evaluate synthetic data generators as a privacy mechanism using linkage and inference attacks, and report that across the generators they tested, synthetic data did not provide a better trade-off between privacy and utility than traditional anonymisation techniques: the settings that preserved useful signal leaked, and the settings that did not leak had destroyed the signal. Their attacks are strongest precisely on the records that are unusual.

Around it sits a broader literature worth knowing by name:

• Membership inference against generative models. Given a candidate record and access to synthetic output, decide whether that record was in the training set. Chen and colleagues formalised a family of these for generative models in GAN-Leaks (2020). Membership alone can be the disclosure — “this person was in the oncology cohort” is the sensitive fact.
• Memorisation in language models. Carlini and colleagues, in The Secret Sharer (2019) and Extracting Training Data from Large Language Models (2021), showed that verbatim training sequences can be recovered from a trained model, with extractability rising sharply with how often a sequence was repeated. If your generator is a fine-tuned language model, this is your threat model, not the tabular one.
• Deduplication as a privacy control. Kandpal, Wallace and Raffel (Deduplicating Training Data Mitigates Privacy Risks in Language Models, 2022) reported that the rate at which a model regenerates training sequences scales with how many times the sequence appeared, which makes deduplication a privacy intervention and not only a quality one.
• Evaluation toolkits. Attack suites for synthetic data exist as software rather than only as papers, which matters because it means “we tested it” is a runnable claim rather than a rhetorical one.

The common thread is that none of these results says synthetic data is useless, and none of them supports the sentence “synthetic data is anonymous”. They say: it depends on the generator, the capacity, the record, and whether anyone measured.

Why outliers are the failure case

The mechanism is worth understanding because it predicts which records are at risk without running anything. A generator fitted to a large dataset learns the common patterns from many examples and the unusual patterns from very few — sometimes from one. When the model reproduces a common pattern it has generalised across thousands of people. When it reproduces a rare combination — a 94-year-old in a particular postcode with a particular rare diagnosis — there was one person that could have come from.

This inverts the usual intuition in an uncomfortable way. The records most in need of protection are the ones a fitted generator protects least, and they are also the ones the dataset is often most valuable for. It is the same tension the distribution-sensitivity axis describes, arriving from the privacy side: keeping the tail is the thing that makes the data useful and the thing that makes it leak.

It also means the naive safety check — “no generated row is an exact copy of a training row” — is close to worthless. Nearest neighbour distance is a far better instrument, and even it is not a proof: a record can be reconstructed from several generated rows without any one of them being close to it.

What differential privacy actually buys

Differential privacy is the only mechanism here that provides a guarantee rather than an argument. A generator trained with a differentially private algorithm — DP-SGD, or a DP fitting procedure for a simpler model — bounds how much the output distribution can depend on any single training record, quantified by a privacy budget usually written as epsilon.

Four things worth being precise about:

• The guarantee is about the mechanism, not the output. It bounds what any attacker can infer about one record’s presence. It does not promise that the synthetic data reveals nothing about the population — that is the entire point of releasing it.
• Epsilon is not a grade. Small values are strong, large values may be nearly vacuous, and published deployments span orders of magnitude. A quoted epsilon means little without the delta, the unit of privacy (one record? one user? one user-day?), and the composition over how many queries.
• The cost lands on the tail. DP works by limiting the influence of individual records, which is definitionally the influence of the rare ones. Expect the unusual patterns to be exactly what the private generator smooths away — which is often acceptable and sometimes destroys the reason you wanted the data.
• Post-processing is free. Anything you compute from differentially private output is still covered by the same guarantee, which is the property that makes DP synthetic data genuinely shareable in a way that a heuristic pipeline is not.

Questions to answer before release

Not a compliance form — the five questions whose answers determine whether the release is defensible.

• What is the threat model? Who receives the data, what auxiliary information do they plausibly have, and what would count as a disclosure? “Someone learns a person was in the dataset” and “someone recovers a field value” are different risks with different tests.
• Was the generator trained with a formal guarantee? If yes, record epsilon, delta, the unit of privacy and the accounting method. If no, then no guarantee exists, and every claim about the output is empirical.
• Were attacks actually run? Nearest-neighbour distance from each synthetic record to the training set, a membership inference attempt, and a targeted attack on the most unusual records in the source. A privacy claim with no attack behind it is an assertion.
• What happens to the outliers? Explicitly: were they removed before fitting, generalised, or left in? This is the single decision that most affects both the risk and the utility, and it should appear in the datasheet.
• What does your regulator’s definition say? Under GDPR the question is whether a natural person is identifiable by means reasonably likely to be used, which is a legal test and not a technical one. Get the legal answer before the technical work, not after — the GDPR page covers the framing.