Video Summarization: Picking the Frames That Matter

Extractive video summarization is a constrained optimisation problem with a scoring model bolted on the front. Once you see it that way, the parts that matter separate cleanly: the segmentation decides what the atoms are, the scorer ranks them, and a knapsack solver picks a subset that fits the budget. Most of the published effort goes into the scorer, and most of the observable result comes from the other two.

Two different things called a summary

A storyboard is a set of still frames: cheap, browsable, suitable for a thumbnail strip, and produced by keyframe extraction. A video skim is a shorter video assembled from selected shots, with audio, playable as a trailer. They need different machinery: a storyboard can pick isolated frames anywhere, a skim must pick contiguous runs that do not cut mid-word and must sum to a duration.

A third thing also gets called summarization: producing a paragraph of text describing the video. That is a captioning problem downstream of this one, usually fed by the shots this pipeline selects. This page is about the skim, because it is the one with the budget constraint.

The three-stage pipeline

• Segment. Cut the video into shots. Almost every published method uses kernel temporal segmentation or a shot boundary detector, because a summary that starts and ends mid-shot looks broken to a viewer in a way that a slightly worse selection does not.
• Score. Assign every frame an importance in [0, 1], then average within each shot to get a shot score.
• Select. Choose a subset of shots maximising total importance subject to total duration not exceeding the budget. This is 0/1 knapsack, solved exactly by dynamic programming over quantised durations.

Scoring importance

Three families, and they encode genuinely different notions of “important”.

Representativeness asks which shots, if kept, best reconstruct the rest. Formally this is a facility-location or k-medoids objective over frame embeddings: pick a set S minimising the summed distance from every frame to its nearest member of S. It produces balanced coverage and it will happily include a long boring shot because a lot of the video looks like it.

Diversity penalises redundancy directly, usually with a determinantal point process or a repelling term that lowers a shot’s score in proportion to its similarity with shots already chosen. Good at removing near-duplicates, and prone to selecting anomalies — a lens flare is maximally diverse.

Learned interestingness trains a sequence model on human frame-level annotations, typically a bidirectional LSTM or a transformer over frame features producing one score per frame. It captures conventions that the first two cannot, such as human preference for shots containing faces, at the cost of inheriting whatever the annotators’ task told them to prefer.

A fourth signal is free and routinely ignored: the audio. Speech density, laughter, applause and sudden loudness changes correlate with importance far better than most visual features, they cost almost nothing to compute, and they are available for footage where the visual scorer has nothing to work with — a lecture recording of one static slide deck is visually uniform and acoustically highly structured.

The selection step, worked

Take a twenty-minute video — 1,200 seconds — and a ninety-second summary, a budget of 7.5% of the duration. Suppose segmentation gave shots with these durations and mean scores. The value of a shot is its mean score times its duration, because the objective is the sum of the frame scores you keep.

shot   duration (s)   mean score   value = d x m
S1          8            0.81           6.48
S2         45            0.74          33.30
S3         12            0.69           8.28
S4         30            0.66          19.80
S5         22            0.61          13.42
S6         19            0.55          10.45

budget: 90 s

greedy by value density (= mean score):
  S1  ->  8 s used, value  6.48
  S2  -> 53 s used, value 39.78
  S3  -> 65 s used, value 48.06
  S4  -> would be 95 s, exceeds budget, skip
  S5  -> 87 s used, value 61.48
  S6  -> would be 106 s, skip

selected: S1 S2 S3 S5   87 s   total value 61.48

Two things are worth extracting from that. First, value density equals the mean score exactly, because value is mean times duration and density is value over duration. So greedy-by-density and greedy-by-score are the same algorithm here, and the ranking of shots is entirely by mean importance — duration only decides what fits.

Second, greedy is not optimal in general. Shots are atomic, so the last three seconds of budget cannot be filled with three seconds of a good shot. In this instance the alternative subset S2 + S3 +  S4 uses 87 seconds for a value of 61.38, which loses to the greedy answer by 0.1 — a difference no viewer could perceive. That near-tie is not a coincidence, and it is the subject of the next section.

Two adjustments the arithmetic does not capture, both of which a viewer notices immediately. Selected shots must be re-ordered into chronological order before assembly, not left in score order, or the summary tells the story backwards. And the audio has to be trimmed at a different point from the video: cutting a shot at its exact boundary truncates the word being spoken across it, so the standard fix is to extend each selected segment to the nearest silence in the audio track, which spends a little of the budget on making the result listenable. Where that extension pushes you over the budget, drop the lowest-density selected shot rather than shortening several — shortening is what reintroduces the mid-word cut.

Why the benchmark barely measures this

The standard datasets are SumMe (Gygli and colleagues, ECCV 2014) and TVSum (Song and colleagues, CVPR 2015), and the standard protocol converts predicted frame scores to shot scores, runs exactly the knapsack above at a 15% budget, and computes F1 against human summaries.

Otani and colleagues showed in 2019 that summaries built from randomly assigned importance scores achieve F1 comparable to published methods under that protocol. The reason is structural: the shot segmentation and the duration constraint together restrict the output space so tightly that the scorer — the part every paper contributes — has little room to change the answer. Their paper is on arXiv and it proposes rank correlation, Kendall’s tau and Spearman’s rho, between predicted and human frame scores as a replacement, because that evaluates the scorer directly instead of through a lossy selection step.

If what your users actually want is a specific part of the video rather than a shortened version of all of it, summarization is the wrong tool and moment localization from a query is the right one: it optimises for one answer instead of coverage.