Video Frame Deduplication Before Sending to a Model

Most consecutive frames of most video are nearly identical, and a model charges you for every one of them at full price. Removing the duplicates is the highest-leverage preprocessing step available for video input, and the arithmetic for how much it saves is short enough to do before you write any code.

The token count before you do anything

Start with what a naive submission costs. Take a ten-minute clip and sample it at two frames per second — already a heavy reduction from the 30 fps source. That is 1,200 frames. Each frame becomes visual tokens according to the encoder’s patch geometry: a 336×336 input with a patch size of 14 gives a 24×24 grid, so 576 tokens per frame.

ASSUMPTIONS (stated, not measured)
  clip length          10 minutes
  sample rate          2 fps
  tokens per frame     576   (336px input, patch 14 -> 24 x 24 grid)
  input token price    $1.00 per million tokens

BEFORE DEDUPLICATION
  frames    600 s x 2 fps            = 1,200 frames
  tokens    1,200 x 576              = 691,200 tokens
  cost      691,200 / 1e6 x $1.00    = $0.69 for one ten-minute clip

  over 10,000 such clips             = $6,912

Nearly seven hundred thousand tokens for ten minutes of video is already past the context window of many models, so for a large part of the field this is not a cost problem but a feasibility one: the request does not fit and no amount of budget makes it fit. Deduplication is how you make the request possible before it is how you make it cheap.

The comparison, and the bug in it

The comparison itself is cheap. Compute a small perceptual hash per frame — a 64-bit difference hash is enough and costs microseconds — and drop a frame when its Hamming distance from a reference frame is at or below a threshold T. On 64-bit hashes, T around 3 to 5 is a conservative “visually identical” setting and 8 to 10 is aggressive.

The bug is in the choice of reference frame, and it is the single most common defect in hand-rolled implementations.

WRONG: compare each frame to the immediately previous frame

  f1  f2  f3  f4  f5 ... f60
  each pair differs by 1 bit -> every frame dropped except f1
  but f1 and f60 differ by 60 bits: completely different images

  a camera drifting slowly, a sunset, a slow zoom, a door opening
  over four seconds -- all of these drift past any threshold without
  any single step ever exceeding it. The entire event collapses to
  one frame and the model never sees it happen.

RIGHT: compare each frame to the last frame you KEPT

  keep f1. compare f2..fn against f1 until one exceeds T; keep that
  one; it becomes the new reference. Accumulated drift is now bounded
  by T rather than by T per step.

The difference is not subtle. Comparing against the previous frame bounds the change between neighbours; comparing against the last kept frame bounds the change between the frames you actually send, which is the property you wanted. The off-the-shelf implementation gets this right: FFmpeg’s mpdecimate filter, documented in the FFmpeg filters reference, drops frames that do not differ greatly from the previous non-dropped frame, with hi, lo and frac parameters controlling how much of the frame must differ and by how much. If you only need this at the command line, it is one filter and you do not need to write the loop at all.

The saving, derived

Here is the honest part. The number of frames you keep is not a function of the threshold. It is a function of the threshold and your footage, and no page can tell you the second one. So the derivation carries it as a named variable — the retention ratio r, the fraction of sampled frames that survive dedup — and the arithmetic is exact once you supply it.

tokens after dedup = frames x r x tokens_per_frame
                   = 1,200 x r x 576

  r = 0.50   600 frames   345,600 tokens   $0.35   50% saved
  r = 0.25   300 frames   172,800 tokens   $0.17   75% saved
  r = 0.15   180 frames   103,680 tokens   $0.10   85% saved
  r = 0.05    60 frames    34,560 tokens   $0.03   95% saved

  at r = 0.15, across 10,000 clips
    $6,912  ->  $1,037      saving $5,875

WHAT r DEPENDS ON -- all of these, none of them the threshold alone
  static camera, static subject      r very low   (a fixed security camera
                                                   at night may sit under 0.02)
  static camera, one moving subject  r low
  handheld or continuous camera move r high       (every frame differs;
                                                   dedup saves almost nothing)
  rapid intercutting                 r high
  screen recording, mostly still     r very low

The last two lines of that block are the warning. On continuously moving footage — a handheld walkthrough, a drone shot, a sports feed with a tracking camera — every consecutive frame genuinely differs andr approaches 1. Deduplication returns nearly nothing and you have added latency for no benefit. Know which kind of footage you have before you build the stage.

The maximum-gap rule

A pure similarity rule has an unbounded worst case in time: a camera watching an empty corridor for nine minutes and something happening in the tenth will keep one frame from the first nine minutes, which is correct, and may keep too few from the tenth if the event unfolds slowly, which is not. It also produces a frame sequence with no consistent time base, so the model has no way to know whether two adjacent frames are 0.5 seconds or 6 minutes apart.

Both problems are fixed by the same addition: an unconditional maximum gap. Always keep a frame at least every N seconds regardless of similarity, and attach its timestamp.

with a maximum gap of N seconds over a clip of length L seconds,
the kept-frame count has a floor:

  minimum kept = L / N
  minimum r    = (L / N) / (L x sample_rate) = 1 / (N x sample_rate)

  L = 600 s, sample rate 2 fps, N = 10 s
    minimum kept = 60 frames
    minimum r    = 1 / (10 x 2) = 0.05

so r can never fall below 0.05 here, and the token floor is
  60 x 576 = 34,560 tokens -- the price of never being blind for
  more than ten seconds.

Choosing N is choosing the longest interval you are willing to have no observation of. For a moderation pass that is a policy decision; for video question answering it is set by the shortest event a question might ask about. Pass the timestamps along with the frames either way — a model given nine frames with no time information cannot answer any question about duration or order reliably, and the timestamps cost a handful of text tokens against hundreds of visual ones.

Measuring your own retention ratio

Everything above resolves to one number you do not have, so get it before you spend anything. The measurement is cheap because deduplication is cheap: hashing runs at thousands of frames per second on a CPU and requires no model call at all.

1. Take a representative sample of your footage — twenty clips spanning the kinds of content you actually receive, not twenty clips of the easy kind.
2. Run the dedup pass at your intended sample rate and several thresholds, comparing against the last kept frame, and record the kept-frame count for each. Divide by the sampled count to get r at each threshold. No model has been called yet and nothing has been billed.
3. Plot r against threshold. The curve is usually flat at low thresholds and then falls off sharply; the knee is the setting where you are dropping real duplicates rather than real content.
4. Check the dropped frames by eye on two or three clips, particularly around any known slow event. This is the step that catches the previous-frame comparison bug and it takes minutes.
5. Multiply your measured r through the token arithmetic above with your provider’s current price to get the figure you can actually quote to somebody.

The related decision — which frames to keep when you want variety rather than merely non-redundancy — is a different algorithm and is covered in keyframe extraction. Deduplication removes what is the same; keyframe extraction chooses what is representative, and on a long library the second is what you want to store, as the cost derivation in storing and indexing a video library works through.