AI Watermarking: Does It Work?

A watermark is a signal put into the output deliberately at generation time. That makes it fundamentally more reliable than detecting synthesis after the fact, and it inherits a different set of problems — one of which is that a watermark can be forged onto text somebody wrote themselves.

What a watermark is, and is not

Three things get confused. Provenance metadata travels alongside the content and can be stripped without touching it. Post-hoc detection infers synthesis from the artefact with no cooperation from anyone. Watermarking sits between: the generator deliberately embeds a statistical signal into the content itself, so it survives copying and format changes in a way metadata does not, but it only exists if the generator chose to put it there.

Two design goals pull against each other throughout: the mark must be detectable with high confidence, and it must not degrade the output. Every choice below is a point on that trade.

Text watermarking and the entropy problem

The mechanism most published schemes build on was described by Kirchenbauer and colleagues, and it is elegant enough to state exactly. At each generation step, hash the preceding token or tokens together with a secret key, use the result to seed a pseudorandom partition of the vocabulary into a “green” list and a “red” list, and add a small bias to the logits of the green tokens before sampling.

Detection needs no model. Recompute the partition at each position — which requires the key — and count how many of the observed tokens fell in their green list. Human text has no reason to prefer green, so the count is binomial around half; watermarked text runs high, and the test is a straightforward statistical one.

Three consequences follow directly from that construction, and they are the whole story of what text watermarking can do:

• It needs entropy. The bias only changes the output where more than one token was plausible. If the next token is essentially determined — a factual completion, a syntactic requirement, a piece of code that must compile, a quoted string — then either the model emits the forced token regardless of colour, which carries no signal, or it emits a green alternative, which is wrong. Low-entropy text therefore cannot be watermarked without damaging it. This is not an implementation weakness; it is arithmetic.
• Confidence grows with length. The test is statistical, so a short passage cannot be classified confidently at any useful false-positive rate. Short outputs are effectively unmarkable, which covers a great many of the uses people most want to catch.
• Detection is centralised. Detection requires the key. Publishing it enables forgery; withholding it means the public cannot verify anything and must ask the provider — who is then the sole authority on whether their own model wrote something. Distributed variants exist and complicate the picture rather than removing it.

Image and audio

Continuous media has far more room to hide a signal, which is why perceptual watermarking predates all of this by decades in the copy-protection world. Modern approaches embed the mark during generation — perturbing the latent, or training an encoder and decoder pair jointly so the mark survives a distribution of transformations — rather than stamping it on afterwards, which makes it harder to remove by cropping or re-encoding. SynthID is the best-known deployed example of the family.

The capacity advantage is genuine: image watermarks can carry more bits, survive more transformation, and be detected on shorter samples than text watermarks can. The adversarial position is the same, and the transformations an attacker may apply are not limited to the ones the decoder was trained against.

The attack literature

• Paraphrase. The standard attack on text watermarks: rewrite the passage with another model. Krishna and colleagues demonstrated that paraphrasing substantially degrades detectors, including watermark detectors, because the token sequence that carried the signal is replaced while the meaning is preserved.
• Editing and mixing. Substituting a fraction of tokens or interleaving human-written sentences dilutes the statistic. Because detection is a test on a proportion, dilution directly reduces confidence.
• Translation round trips and heavy reformatting have similar effects for similar reasons.
• Regeneration for images: pass the image through another generative model at moderate strength and much of the embedded signal is resampled away while the content survives.
• Spoofing — the underrated one. An attacker who can learn something about the green-list partition, for instance by observing many watermarked outputs, can bias their own writing toward green tokens and produce human-written text that a detector calls machine-generated. The threat model here is not evading attribution but manufacturing it, and applied to a student, a journalist or a defendant it is the more serious harm.
• Theoretical limits. Zhang and colleagues argued that under stated assumptions — chiefly, that the attacker has access to a quality oracle and can make quality-preserving perturbations — any watermark can be removed. The assumptions matter and are debated; the result is a reason to treat robustness claims as claims about a specific attacker budget rather than about watermarking in general.

What this means for a mandate

A watermarking requirement is not useless, and the case for it is narrower than it is usually made. Where it works: establishing at scale that a large volume of content came from a cooperating provider, giving platforms a cheap high-precision signal for bulk filtering, and letting providers enforce their own terms. Bulk, statistical, low stakes per item.

Where it does not work: adjudicating an individual case. A watermark finding is a probabilistic result from a proprietary detector against an adversary who can both remove and forge it. Presenting that in a disciplinary hearing or a courtroom as evidence about one person is not supported by the mechanism, whatever the confidence number says.

And the coverage limit is structural: a mandate reaches providers who serve through an API. Weights running on someone’s own hardware can generate unmarked output, and no obligation on the original publisher changes that. Whether a partial regime is still worth having — because most volume does come through commercial providers — is a reasonable policy judgement, and one that should be made with the coverage gap stated rather than assumed away.

Four questions to put to any robustness claim: which attacks were tested, at what output length, at what false-positive rate on human text, and is the detector public or held by the vendor making the claim.