Synthetic Media, Deepfakes and Provenance

There are two ways to respond to convincing synthetic media, and they point in opposite directions: try to recognise the fake, or try to establish the real. The second is structurally more promising and it is not a solution either.

Two opposite strategies

Detection asks a question about the artefact alone: does this image bear traces of synthesis? It needs no cooperation from anyone, works retroactively on existing media, and is an adversarial classification problem.

Provenance asks a question about the artefact’s history: who asserts where this came from and what was done to it? It requires cooperation from capture devices, editing software and distribution platforms, only works going forward, and is a cryptography and governance problem rather than a machine learning one.

Watermarking is a third thing that sits between them — a signal deliberately embedded by a cooperating generator — and has its own page in this cluster.

Why artefact detection erodes

The argument that detection is a losing position is widely held among researchers in the area, and it rests on four specific mechanisms rather than on pessimism:

• The detector is available to the forger. Anything published can be optimised against. A generator can be tuned until the detector no longer fires, and this is a straightforward engineering task rather than a research problem.
• Cues are generation-specific and perishable. Detectors learn artefacts of particular architectures — inconsistent lighting, frequency-domain signatures, physiological implausibilities. Each generation of generators removes some. Cross-generator generalisation is the known weak point in the published evaluations.
• The distribution channel destroys evidence. Real media is re-encoded, resized, cropped and screenshotted on its way through platforms. The subtle statistical traces detectors rely on are exactly what lossy re-encoding removes.
• Base rates punish false positives. Most media is authentic. Applied across a platform, even a small false positive rate produces a large absolute number of authentic items flagged — the same arithmetic that governs text detectors and content moderation, and it is unforgiving.

The counter-argument, which deserves stating: detection does not need to be perfect to be useful in a forensic context where an expert examines one item with time and multiple methods, rather than at platform scale in real time. That is a real distinction, and it is the setting where detection continues to be worth funding.

It is worth separating two things detection is asked to do, because they have different prospects. Establishing that a specific artefact is synthetic, to a standard that would survive challenge, is the hard version and the one the argument above applies to. Establishing that a large volume of accounts is posting content from the same generator is a much easier statistical problem, because it aggregates weak signals over many items, and it is where automated detection genuinely earns its place. Coordinated inauthentic behaviour has always been easier to detect than individual falsity, and synthetic media does not change that.

What a manifest actually proves

The C2PA specification defines a manifest: a set of assertions about an asset — how it was captured or generated, what edits were applied, what ingredients went into it — bound to the asset by a cryptographic hash and signed with a certificate. Editing a signed asset in a conformant tool produces a new manifest that references the previous one, forming a chain.

Be precise about the claim this supports. A valid manifest proves: the holder of this key asserted this history for this exact byte sequence. Everything else is inference. It does not prove the depicted event occurred, that the assertion is true, that the signer is trustworthy, or that the camera was pointed at something real. It is a chain-of-custody claim, of the same species as a signed audit log, and its value is entirely a function of whether the signer is accountable.

That is not a criticism. Chain of custody is exactly what is missing from digital media and its absence is why the liar’s dividend works. But an institution that treats a green tick as “verified true” has misread the specification, and building a public interface around that misreading is a foreseeable failure.

Where provenance breaks

• Stripping. Metadata is removed by re-encoding, screenshots and many upload pipelines. Preservation requires every intermediary to cooperate, and there are many intermediaries. Durable binding through a watermark is one proposed mitigation, which reintroduces the watermark’s own weaknesses.
• The analogue hole. Photograph a screen showing a fabricated image with a provenance-enabled camera and you obtain a genuine, signed capture of a falsehood. The manifest is accurate; the image is a lie. No cryptography addresses this.
• Trust list governance. Verification requires deciding which signers count. That decision — which manufacturers, which software, which news organisations, which states — is a political question wearing a PKI costume, and it determines who is able to produce authenticated media at all.
• Privacy. A provenance chain records device, software and edit history. For a whistleblower, a protester or a source that is an identification risk, which is why the specification supports redaction and why redaction weakens the chain.
• Absence proves nothing. The large majority of authentic media carries no manifest and will not for years. If “unsigned” comes to read as “suspect”, the system harms exactly the people with older phones and fewer institutional resources.

Policy instruments and what each bites on

Mapping instruments to the point where they actually apply pressure:

The pattern across the table is worth stating plainly: instruments bite at distribution and at commercial scale, not at production. Anyone promising that a rule will stop fakes from being made is describing an outcome the instruments cannot deliver. This is a description of the mechanisms, not legal advice, and the law here differs greatly between jurisdictions and is changing quickly.

One closing observation about how to think about the whole area. The harm from synthetic media is not evenly distributed across the things people worry about. Fabricated political video is the case that gets discussed; non-consensual sexual imagery and fraud against individuals are, by volume and by documented complaint, where the damage is concentrated, and both are addressable by existing law aimed at conduct rather than by any technical marker. Policy attention and harm distribution are not well aligned here, which is worth noticing before deciding what to build.