Telling a Camera Cut From Fast Motion in Raw Footage

A hard cut and a fast pan produce almost the same amount of change between two consecutive frames. Any detector built on how much the picture changed will confuse them, and the confusion is not a tuning problem — no value of the threshold separates them, because the quantity being thresholded does not distinguish them.

Why the obvious threshold fails

The standard first attempt is mean absolute frame difference: convert both frames to greyscale, take the mean of the absolute per-pixel difference, and call it a cut when it exceeds a threshold. FFmpeg exposes this directly — its scdet filter sets the mean absolute frame difference and a scene score as frame metadata, and select='gt(scene,0.4)' is the widely copied one-liner built on it, both documented in the FFmpeg filters reference.

It works well on edited material with static or slow-moving cameras. It falls apart on raw footage, and the reason is easy to state. If the camera pans quickly enough that the scene shifts by a large fraction of the frame width in one frame interval, then almost no pixel in frame t+1 holds the same content as the pixel at the same coordinates in frame t. The per-pixel difference is therefore large, for exactly the same reason it is large at a cut: the content under each coordinate has been replaced. A colour-histogram distance is more robust than a pixel difference, because a pan mostly preserves the distribution of colours while a cut to a different scene usually does not — but it fails in the other direction, missing a cut between two shots of the same location.

The information that separates the two cases is not in the magnitude of the change. It is in whether the change is explained by motion.

The motion field has structure, not just size

Optical flow estimates, for each pixel or each tracked point, the displacement that maps it from one frame to the next. Two classic methods are in every toolkit: sparse Lucas–Kanade tracking of corner features, and the dense polynomial-expansion method published by Gunnar Farnebäck in 2003, both documented in OpenCV’s optical flow tutorial. What matters here is not which estimator you use but what the resulting field looks like in each case.

• Fast pan. Every point moves in nearly the same direction by nearly the same amount. The magnitude is large and the variance in direction is small. The field is coherent: it is close to a single global translation, with small deviations from parallax and from objects moving independently.
• Zoom or dolly. Vectors radiate from or converge on a focus of expansion. Magnitude grows with distance from that point. Still highly structured, and still explained by a single global model, just an affine one rather than a translation.
• Hard cut. There is no true correspondence to find, because the pixels in the second frame came from a different scene. An estimator asked for one anyway returns a field with no consistent direction, high directional variance, and low match confidence everywhere. Sparse trackers report most of their points as lost.

So the discriminator is coherence, and the cheapest usable measure of it is the circular variance of flow directions weighted by magnitude: low for a pan, high for a cut. That already separates most cases that a magnitude threshold cannot.

Fitting a global motion model

The stronger version turns coherence into an explicit test: try to explain the whole frame with one motion model, and measure how much is left over.

1. Detect corner features in frame t and track them into frame t+1 with sparse Lucas–Kanade. Record how many tracks survive.
2. Fit a global transform — a translation, or an affine or homography if the camera can zoom or roll — to the surviving correspondences using RANSAC, and record the inlier fraction.
3. Warp frame t by that transform and compute the mean absolute difference against frame t+1. Call this the residual.
4. Compare the residual to the unwarped difference. The ratio is the decision statistic.

ILLUSTRATIVE STATISTICS (shapes, not measured values)

                       tracks   RANSAC     raw     residual   residual
                       kept     inliers    MAD     after warp / raw

  static camera        96%      0.94        3         2         0.67
  fast pan             81%      0.88       47         5         0.11
  fast pan + subject   74%      0.71       52         9         0.17
  hard cut             12%      0.15       51        48         0.94

read the last column, not the "raw MAD" column:
  the pan and the cut have nearly identical raw difference (47 vs 51)
  and completely different residual ratios (0.11 vs 0.94)

That table is the whole argument. Raw difference does not separate the two rows that matter; residual ratio separates them by a factor of eight. The decision rule is a conjunction rather than a single threshold: declare a cut when the raw difference is high and the track survival is low and the residual ratio stays near one. Any one of those alone is a detector with a known failure mode; together they are robust.

One practical note on cost. Dense flow over every frame pair is expensive and mostly wasted, because the vast majority of frame pairs are obviously neither. Use the cheap mean-absolute-difference test as a gate and run the flow-based test only on the few percent of frame pairs that exceed it. The expensive test is then run a few thousand times per hour of footage instead of a hundred thousand times, and the pipeline stays affordable at library scale — which is what makes it usable inside shot boundary detection and keyframe extraction.

The whip pan, which defeats both

There is one case where the residual test also fails, and it is worth knowing because it is the case that generates most false positives in handheld and sports footage. In a genuine whip pan the camera moves so fast that the exposure smears the image into horizontal streaks. Corners are destroyed, so Lucas–Kanade has nothing to track; track survival collapses; the RANSAC fit has no correspondences to fit; the residual stays high. Every statistic in the table above reads like a cut, because the evidence a motion model needs has been physically removed by the shutter.

The way out is to stop looking at the frame pair and look at the neighbourhood in time. A whip pan is a transition within one scene: the footage before it and the footage a few frames after it are the same location, the same lighting and largely the same palette. A cut is a transition between scenes and generally is not.

• Histogram recovery. Compare the colour histogram some frames before the event with the histogram some frames after. If the distance is small, the scene came back — a pan. If it stays large, it did not — a cut. Skip the smeared frames themselves when computing this, since their histograms are averages of everything the camera swept past.
• Blur onset and offset. A whip pan has a characteristic profile: sharpness falls over several frames as the camera accelerates and recovers over several as it settles. A cut is a single-frame discontinuity with no sharpness ramp on either side. A Laplacian-variance sharpness measure plotted across a short window makes the two shapes visibly different.
• Directional blur agreement. Motion blur has an orientation, and in a whip pan that orientation agrees with the flow direction measured in the frames just before the smear. That agreement is evidence of continuous motion, and there is nothing to agree with at a cut.

Dissolves need a different test entirely

Everything above detects an abrupt change, and it will miss every gradual transition — a dissolve, a fade to black, a wipe. During a one-second dissolve at 30 fps, each consecutive frame pair differs by roughly a thirtieth of the total change between the two shots, which is smaller than ordinary motion and will never cross any threshold set to avoid firing on movement.

The classical answer is twin comparison: run the abrupt test as usual, and in parallel accumulate the difference between the current frame and a reference frame held from further back. A dissolve produces a small per-pair difference and a steadily accumulating difference against the held reference, which crosses a second, higher threshold after the transition completes. A dissolve also has a distinctive signature of its own — during the cross-fade the frame is a weighted average of two images, so local contrast and edge energy dip and then recover, producing a characteristic valley that a fade detector can key on directly. Whichever you use, it is a separate detector running alongside the cut detector, not a threshold adjustment to it.