Differential Privacy in Deployed Systems: What Epsilon Lets an Attacker Learn

Differential privacy is a mathematical guarantee about how much any one person’s presence in a dataset can change what comes out of it. The guarantee is real and it is precise. The parameter that controls it, epsilon, is usually described as a dial from private to useful, which tells you nothing — so this page converts it into the only question that matters: how much can somebody’s belief about you change after seeing the output.

The definition, and why it is shaped like that

A randomised mechanism M satisfies (ε, δ)-differential privacy if, for every pair of datasets D and D' differing in one record, and every set of possible outputs S:

Pr[ M(D) ∈ S ]  ≤  e^ε · Pr[ M(D') ∈ S ]  +  δ

Read it as a bound on distinguishability. Whatever the output is, it was almost exactly as likely to have arisen from the world where your record was included as from the world where it was not. “Almost exactly” is quantified by e^ε, and δ is the probability the bound simply fails.

Two features of the definition do the real work and are usually skipped. It quantifies over every output set, so there is no clever query that escapes it. And it makes no assumption about what the attacker already knows — unlike anonymisation schemes, which fail when the attacker has an auxiliary dataset you did not anticipate. That robustness is the reason the definition is worth its costs. The standard reference is Dwork and Roth’s “The Algorithmic Foundations of Differential Privacy” (2014).

What epsilon means to an attacker

Here is the translation that makes epsilon usable. Because the definition bounds the ratio of likelihoods, it also bounds how far an attacker’s odds can move after seeing the output:

posterior_odds  ≤  e^ε  ×  prior_odds

Worked. An attacker believes, before seeing anything, that there is a
1% chance you are in the dataset.

  prior odds = 0.01 / 0.99 = 0.0101

ε = 0.1  →  ×1.11  →  odds 0.0112  →  posterior probability  1.1%
ε = 1    →  ×2.72  →  odds 0.0275  →  posterior probability  2.7%
ε = 3    →  ×20.1  →  odds 0.203   →  posterior probability 16.9%
ε = 8    →  ×2981  →  odds 30.1    →  posterior probability 96.8%
ε = 10   →  ×22026 →  odds 222     →  posterior probability 99.6%

That table is the whole of what epsilon means, and it reframes the usual conversation. An ε of 1 is a genuinely strong guarantee: a suspicion of 1% can become at most 2.7%. An ε of 3 is meaningful but not comfortable. An ε of 10 permits an attacker to go from a one per cent suspicion to near-certainty, which is not what most people understand by “differentially private”.

Note also that this is a worst-case bound over all possible attackers and all possible outputs. The actual leakage from a particular mechanism is usually far less than the bound permits. That is why large epsilons are sometimes defended in practice, and why the defence is uncomfortable: you are relying on the guarantee not being tight, which is precisely what the guarantee refuses to promise.

Delta is not a second epsilon

δ is the probability that the ε guarantee does not hold at all. It is not a softer version of the same knob; it is a failure probability, and the failure it permits is unbounded — for that δ fraction of outcomes, the mechanism may reveal a record entirely.

The conventional requirement is that δ be much smaller than the inverse of the number of records, often expressed as “cryptographically small”. The reason is easy to see: a mechanism that outputs a random record verbatim with probability δ satisfies (0, δ)-DP and is obviously unacceptable. If δ is 1e-5 and your dataset has a million people, you have licensed, in expectation, ten catastrophic disclosures.

The unit of privacy is the whole argument

The definition says “differing in one record”. Which record is your choice, and it is the single most consequential and least-discussed parameter in any deployment.

When you read that a system is “differentially private with ε = 2”, the first question is which of these it means. If the answer is not stated, the figure cannot be interpreted, and stating your own unit alongside your own epsilon is the single most useful thing you can do for anyone assessing your system.

Composition: the budget is spent, not held

Privacy loss accumulates across every release from the same data. Under basic composition, running two ε=1 mechanisms on the same dataset yields ε=2 overall. Advanced composition and the numerical accounting methods used in modern DP training give tighter bounds than the naive sum, but the direction is unchangeable: every additional query, every additional training epoch, every additional released statistic costs budget, and budget does not regenerate.

This is the operational reality that surprises teams. A privacy budget is not a per-request setting, it is a finite total across the lifetime of the dataset, and somebody has to own the ledger. In practice that means: a decision about total ε, a mechanism that refuses queries once it is exhausted, and a policy about what happens then. Systems without those three are doing privacy-flavoured noise addition rather than differential privacy.

What it costs, honestly

• Accuracy, mostly on the tail. DP works by making rare things indistinguishable from absent. That is the point, and it means the categories you lose are the small ones — minority languages, unusual usage patterns, low-frequency classes. The average metric barely moves and the underrepresented case disappears. This distributional consequence is rarely stated and is the most important one.
• Data volume. The noise required is roughly fixed by the sensitivity and epsilon, while the signal grows with the number of contributors. So DP is affordable at population scale and unaffordable on small datasets. If you have ten thousand users, most interesting DP analyses are simply out of reach.
• Training cost, when applied to model training. DP-SGD clips per-example gradients and adds noise, which is substantially slower per step than ordinary training and converges more slowly. Budget a multiple, not a margin.
• An unresolvable policy question. There is no technical method for choosing epsilon. It is a decision about acceptable risk, it belongs to whoever is accountable for the system, and the honest way to make it is with the posterior-odds table above in front of them rather than a number borrowed from a paper.

What it does not protect against

• Conclusions about groups that happen to apply to you. If a DP analysis establishes that people with a certain characteristic tend to do something, and you have that characteristic, someone can infer something about you. DP explicitly does not prevent this — it protects your participation, not you from statistics.
• Correlated records. The guarantee is about one record changing. If your data appears in several rows, or if your household’s records are correlated with yours, the effective protection is weaker than the stated epsilon.
• Everything outside the mechanism. DP applies to a specific computation. Logs, caches, intermediate files and the raw dataset itself are untouched by it. A DP output from a pipeline that retains raw inputs has moved the risk, not removed it — the framing in privacy-preserving data pipelines is about exactly that boundary.
• Bad implementations. Floating-point rounding in noise sampling, incorrect sensitivity analysis, and side channels in timing have all broken real DP deployments. The definition is sound; the implementations are software.
• Legal obligations, automatically. DP is strong evidence of data minimisation and it is not a self-executing compliance argument. Whether a DP release is personal data remains a question for the specific regime and the specific parameters.