Detecting Image Manipulation and Splicing

A spliced image is two photographs with two different histories sharing one file. Almost every reliable detection method works on that observation: it does not look for the seam in the picture, it looks for the seam in the history.

Three families of evidence

Forensic signals fall into three groups, and they are worth keeping separate because they fail for different reasons and are therefore complementary rather than redundant.

• Compression-level. What the JPEG encoder did to each region, and how many times. Cheap to compute, works on the file alone, destroyed by a single re-encode of the whole image.
• Sensor-level. Traces of the physical camera and its demosaicing pipeline. Strong evidence when you have the camera or a corpus from it, useless for an anonymous image.
• Scene-level. Whether the light, shadows and perspective in the composited region are consistent with the rest of the scene. Survives re-encoding, but is weak, laborious and easy to argue with.

A finding from one family alone is weak. A region flagged by two families that fail for unrelated reasons is much harder to explain away, and that redundancy — not any single detector — is what a serious examination rests on.

Compression history: double quantisation and ghosts

JPEG splits the image into 8×8 blocks, applies a discrete cosine transform to each, and divides the 64 resulting coefficients by a quantisation table indexed by the quality setting, rounding to integers. That rounding is where the evidence comes from.

Open a JPEG, paste a region into it, save again, and the untouched background has been quantised twice: once with the original table, once with the new one. The pasted region has been quantised once with the new table, or with some other camera’s table before that. When a coefficient is divided by q1, rounded, multiplied back, and then divided by a different q2 and rounded again, the histogram of that coefficient across all blocks develops a periodic structure — regularly spaced peaks with empty bins between them. Single-quantised regions do not have it. Plotting the histogram of a mid-frequency DCT coefficient over a sliding window is a usable localisation method with no model at all.

Hany Farid’s “Exposing Digital Forgeries From JPEG Ghosts” (IEEE Transactions on Information Forensics and Security, 2009) turns the same idea into something you can see. Re-save the whole image at every quality factor from, say, 40 to 100, and compute the per-pixel squared difference from the original at each step. A region that was originally saved at quality q shows a local minimum in that difference when the sweep passes through q, because re-quantising with the table it already matches changes it least. If one region’s minimum sits at a different quality from the rest of the image, that region has a different compression history. The “ghost” is that region appearing dark in the difference map at one quality and not at others.

There is a third compression signal that costs almost nothing to check: grid alignment. If the pasted region was not positioned at a multiple of eight pixels, its internal 8×8 block boundaries are offset from the host image’s. Measure blocking-artifact strength as a function of grid offset across the image and the misaligned region stands out. This is called non-aligned double JPEG, and cropping an image has the same effect on the whole frame — which is a useful reminder that these signals detect editing, not deception.

Sensor fingerprints: PRNU and the CFA grid

Photo-response non-uniformity is the strongest camera-identification signal available. Individual photosites on a sensor differ very slightly in gain because of silicon manufacturing variation, and that pattern is fixed for the life of the sensor and effectively unique to it. It behaves as multiplicative noise: the recorded image is roughly the ideal image times one plus a small per-pixel factor. Lukáš, Fridrich and Goljan established the method in “Digital Camera Identification From Sensor Pattern Noise” (IEEE Transactions on Information Forensics and Security, 2006).

The procedure is: estimate the camera’s reference pattern by denoising many images from it and averaging the residuals, then take the residual of the questioned image and correlate it block by block against the reference. Blocks from that camera correlate; a spliced block from anywhere else does not. The correlation is computed on a normalised statistic precisely because the raw correlation varies with block content. It is powerful and it has a hard prerequisite: you need the camera, or a set of images you know came from it. For an image that arrived over the internet from nobody in particular, PRNU has nothing to compare against.

The colour filter array leaves a subtler and more portable trace. Most sensors capture one colour per photosite through a Bayer mosaic and interpolate the other two, so in a camera-original file roughly half the green samples and three quarters of the red and blue samples are computed rather than measured. Interpolated pixels are, by construction, smoother and more correlated with their neighbours than measured ones, on a period-two lattice. A region that came from a different pipeline, or that was rescaled or rotated after demosaicing, has that lattice disturbed or absent. The detector estimates local interpolation-error variance separately on the acquired and interpolated positions and looks for regions where the ratio departs from the rest of the image.

Scene consistency: light, shadows, geometry

These are the signals that survive re-encoding, because they are properties of the depicted scene rather than of the file. They are also the ones that require the most care to state honestly.

Under a Lambertian surface assumption and a distant point light, intensity along an object’s occluding contour varies with the angle between the surface normal and the light direction. The normal along a visible contour is known up to the two image dimensions, so sampling intensities around a contour gives a least-squares estimate of the light direction’s projection into the image plane. Two objects in one photograph whose estimated light directions are incompatible is evidence one was not there. Specular highlights — particularly in eyes, which are near-spherical and reflective — give a second, independent estimate of light position from the highlight’s location on the sphere.

Shadow geometry gives a constraint that needs no photometric model at all. For a single point light, the line joining any point on an object to its corresponding point on that object’s shadow passes through the light’s projected position. Draw those lines for several object-shadow pairs across the image and they must all intersect at one point. If a composited element’s shadow line misses the intersection of the others, either the shadow was painted or the object was added. The same logic applies to vanishing points and horizon consistency for planar structures, and to reflections in flat surfaces.

The weakness is that all of it assumes a single dominant light source and cooperative surfaces. Overcast daylight, multiple lamps, bounce light off a nearby wall, translucent or glossy materials — each widens the error bars until the constraint stops constraining. These methods make a strong case for a badly-made composite and are close to silent on a carefully-made one.

Where every one of these fails

• A screenshot erases everything. Screenshotting a manipulated image and sharing the screenshot gives the whole frame one uniform compression history, one pipeline, and no sensor pattern. Every compression-level and sensor-level test comes back clean on an image that is entirely fabricated.
• Platforms re-encode on upload. Resizing and re-compressing on upload destroys the CFA lattice, resets the block grid, and adds a compression layer on top of the interesting ones. An image analysed after a trip through a social platform has already lost most of what you would want to measure.
• Flat, dark and saturated regions carry no residual. A clear sky has no high-frequency content for PRNU or CFA analysis to work on, and blown highlights are clipped. Absence of a signal in those regions is not absence of manipulation.
• Generative editing is a different problem. Inpainting a region with a diffusion model does not import a second capture history — there is no donor image and no second compression. Splicing detectors are largely blind to it, and the methods that do catch it look for generator artefacts instead, which is a separate and fast-moving field.
• Learned detectors overfit their datasets. Models trained on the standard splicing corpora frequently learn the corpus’s own compression conventions rather than forgery, then collapse on images from anywhere else. Judge a detector by its cross-dataset numbers or not at all — and be careful with its scores, which are usually badly calibrated; see what a confidence score actually means.

The direction the field has moved, given all that, is provenance rather than detection: attaching a signed manifest at capture and at each edit so that the question becomes “what does this file say happened to it, and does the signature hold” rather than “can I find a trace of something that did happen”. The C2PA specification is the main effort there. It does not help with the images you already have, and an absent manifest is not evidence of anything, but it is the only approach that gets stronger rather than weaker as editing tools improve.