Video Moment Localization: Finding the Ten Seconds That Answer a Query

Given one untrimmed video and a sentence, return the start and end seconds of the segment the sentence describes. That is temporal grounding, and it is a different problem from searching a library for the right video — the video is already chosen, and the answer is a pair of numbers inside it.

The task, stated precisely

Input: a video of duration D, and a query such as “the woman puts the kettle down and turns to the window”. Output: a span (t_start, t_end) with 0 ≤ t_start < t_end ≤ D. There is exactly one correct span per query by construction of the datasets, and the prediction is scored by how much it overlaps it.

The datasets that define the task are worth knowing by name, because their construction determines what a reported number means. Charades-STA was built by Gao and colleagues from an existing activity dataset by decomposing its descriptions into sub-sentences with time stamps — the TALL paper introduces it. DiDeMo, from Hendricks and colleagues, takes a different route: it divides every video into five-second chunks and allows only contiguous runs of them as answers, so there are exactly 21 possible spans per video and the task becomes a 21-way classification. Their paper describes the design. ActivityNet Captions supplies long videos with many overlapping annotated segments.

Three ways to produce a span

• Propose then rank. Enumerate candidate spans, embed each alongside the query, score every pair and take the best. The enumeration is the cost: with the video split into T clips, the number of candidate (start, end) pairs with start ≤ end is T(T+1)/2, so T = 64 gives 2,080 candidates to score per query. 2D-TAN organises exactly that set as a two-dimensional map indexed by start and end, so a convolution over the map lets adjacent candidates share context — Zhang and colleagues describe it on arXiv.
• Regress the span directly. Fuse query and video features into one sequence and predict a start distribution and an end distribution over clip positions, exactly as extractive question answering predicts an answer span over tokens. No enumeration, so cost is linear in T, but the model has to commit to one answer and handles ambiguity poorly.
• Set prediction. Treat it like DETR: a fixed number of learned span queries, Hungarian matching between predictions and ground truth during training, and a loss combining L1 on the endpoints with a temporal generalized-IoU term. Moment-DETR, released with the QVHighlights dataset by Lei and colleagues, is the reference design — paper on arXiv.

Temporal IoU, worked

Everything is scored by intersection over union on the time axis, and the arithmetic is worth doing once because the threshold choice moves reported numbers more than most architectural changes do.

ground truth : [310.0, 321.0]      length 11.0 s
prediction   : [312.4, 319.8]      length  7.4 s

intersection = min(321.0, 319.8) - max(310.0, 312.4)
             = 319.8 - 312.4
             = 7.4 s

union        = max(321.0, 319.8) - min(310.0, 312.4)
             = 321.0 - 310.0
             = 11.0 s

IoU          = 7.4 / 11.0 = 0.673

That prediction is inside the right region, misses 3.6 seconds of it, and adds nothing wrong. It counts as a hit at IoU 0.5 and a miss at IoU 0.7. Papers report R@1 at several thresholds precisely because the same model looks strong or weak depending on which you read, and a system whose downstream use is “jump the player here” probably cares about the start time being close rather than about the union being right at all. If that is your use, score start-time error in seconds instead of IoU.

Note also what IoU does to short moments. A two-second ground-truth moment with a prediction offset by one second has an intersection of 1.0 and a union of 3.0, an IoU of 0.33 — a miss at every standard threshold, from an error a human would call correct. Long moments tolerate far larger absolute errors. Aggregate IoU across a dataset with mixed durations and you are mostly measuring performance on the long ones.

There is also a resolution floor you cannot argue past. Video features are computed over clips, so a model whose feature stride is two seconds cannot express a boundary finer than two seconds, and a prediction snapped to that grid can carry a second of error before the model has made any mistake at all. If your target moments are shorter than the stride, no amount of training fixes it; the stride has to come down first, and in the proposal-based designs above that cost is quadratic.

The blind baseline problem

The uncomfortable finding in this field is that on the standard datasets, a model that never looks at the video does surprisingly well. Moment start and end times are not uniformly distributed: annotators favour the beginning of a clip, moments cluster at conventional durations, and the query text correlates with position. A model can learn to map query language onto a prior over spans and score competitively without any visual grounding at all.

Otani and colleagues documented this in 2020 with baselines that ignore visual input, and their conclusion is a methodological one: their analysis is on arXiv. The practical instruction is short. Before believing any moment localization number, including one you produced, run two controls: a query-only model with no video, and a video-only model that predicts the same span for every query. If your model does not beat both by a comfortable margin, it has not learned grounding.

Making it work on your own video

On real footage — lectures, meetings, security recordings, support calls — a trained grounding model is often not the first thing to reach for, because two cheaper signals carry most of the answer.

• The transcript. If the moment is described in speech, retrieval over transcript segments gives you a time range directly, and the alignment problem is the one covered in aligning a transcript with visual events. This handles a large fraction of real queries at a fraction of the cost.
• Shot structure. Snapping predicted boundaries to the nearest shot boundary usually improves both the IoU and the perceived quality, since a moment rarely begins mid-shot in edited footage.
• Frame-level embedding search. Embed sampled frames and the query in a shared space, take the top-scoring frames, and merge runs of them into spans with a gap tolerance. Crude, requires no training, and gives a strong baseline — and it is the same indexing machinery described under multimodal retrieval.

Reach for a trained grounding model when the query describes something nobody says aloud and no single frame settles — a manoeuvre, an interaction, a sequence. That is the case the task was invented for, and it is narrower than the literature makes it look.