Video Question Answering Models, Explained

A video question answering model does not watch a video. It is handed a small set of still frames and a question, and everything it can possibly know about the footage has to be present in those frames. Most of the surprising behaviour of these systems follows from that one constraint.

What the model is actually given

The dominant architecture is a vision encoder feeding a language model. Frames are sampled from the clip, each is passed through an image encoder, the resulting visual features are projected into the language model’s embedding space, and the question is appended as text. The language model then generates an answer conditioned on that combined sequence.

The consequence worth internalising is that the visual input is a sequence of independent stills. There is no motion signal in it unless the architecture adds one. A model given frames at 0 s, 5 s and 10 s cannot see anything that happened between them, cannot see how fast anything moved, and knows the frames are ordered only because their position in the token sequence says so. Some architectures strengthen that with explicit temporal position embeddings or with a temporal pooling stage; some rely purely on sequence order.

Audio is usually absent unless it was separately transcribed and passed as text. If your question is “what did she say about the budget”, the frames will not answer it and the transcript will — which is why aligning the two is its own problem, covered in aligning transcripts with visual events.

The frame budget, worked

How many frames fit is a token-count question, and the arithmetic is fully determined by the vision encoder’s geometry. Take a ViT-style encoder operating on 336×336 pixel inputs with a patch size of 14 pixels, the configuration used by the larger CLIP image encoder described in OpenAI’s 2021 CLIP paper. The patch grid is 336 divided by 14, which is 24 patches on a side, so each frame becomes 24×24 = 576 visual tokens.

ASSUMPTIONS
  encoder input   336 x 336 px, patch 14   -> 24 x 24 = 576 tokens per frame
  context budget  reserve 100k tokens for visual input

  frames that fit  100,000 / 576  = 173 frames

  spread over a 10-minute clip
    600 s / 173 frames = one frame every 3.5 s

  spread over a 60-minute clip
    3600 s / 173      = one frame every 20.8 s

  the same clip at 30 fps holds
    600 s  x 30 = 18,000 frames   -> you are seeing 1 in 104
    3600 s x 30 = 108,000 frames  -> you are seeing 1 in 624

Two things fall out. First, the effective temporal resolution is set by clip length, not by anything about the model’s intelligence: the same model on an hour of footage is looking at one frame every twenty seconds, and any event shorter than that may be entirely absent from its input. Second, this is why token-reduction stages exist. Pooling the 576 tokens down to 64 or 144 per frame, or merging similar tokens across adjacent frames, multiplies the number of frames that fit by four to nine times, at some cost in fine spatial detail. Dropping near-identical frames before encoding does the same thing more bluntly and is worked out in frame deduplication before sending to a model.

Three questions, three requirements

Take a fixed forty-second clip: a person enters a kitchen, fills a kettle, opens a laptop at the table, checks a wristwatch, gets up when the kettle boils, checks the watch again, and pours.

• “What colour is the kettle?” Answerable from any single frame in which the kettle is visible. Frame count is almost irrelevant; spatial resolution is what matters, and pooling visual tokens too aggressively is what breaks it.
• “Did she open the laptop before or after filling the kettle?” Requires at least one frame from each event and requires the model to use their order. No single frame contains the answer, and a model that treats its frames as an unordered bag will answer at chance. This is the class where explicit temporal position information earns its cost.
• “How many times did she check her watch?” The hardest of the three and the one that fails most quietly. Counting requires that each occurrence be sampled at least once and that two samples of the same occurrence not be counted twice. At one frame every 3.5 seconds, a glance lasting one second has roughly a 3.5-to-1 chance against being sampled at all; at one frame every second, a three-second glance appears in three frames and invites a double count. Sampling rate sets both error directions and they move in opposite directions, so there is no rate that removes both.

The practical reading is that you should classify your questions before choosing a configuration. Attribute questions want resolution. Ordering questions want frames spread across the whole clip. Counting questions want dense sampling in the region where the events occur, which usually means finding that region first.

Why benchmarks overstate temporal ability

A large fraction of questions in early video QA datasets can be answered correctly from one randomly chosen frame, because the question is really about the scene rather than about what happened in it. This is why the benchmarks that matter are the ones built specifically to defeat that shortcut.

NExT-QA, introduced by Xiao and colleagues in their 2021 paper, is explicitly partitioned into causal, temporal and descriptive questions over 5,440 videos and about 52,000 annotated question-answer pairs, so a system’s score can be read separately on the split that a single frame cannot solve. EgoSchema, published by Mangalam and colleagues in 2023, goes further: over 5,000 multiple-choice questions spanning more than 250 hours of egocentric video, each over a three-minute clip, with a “temporal certificate” measure of how much of the video a human must actually watch to answer. The paper reports models with several billion parameters scoring under 33% against a 20% random baseline and human accuracy around 76% — a gap that is the point of the benchmark rather than a footnote to it.

Retrieve, then read

For anything longer than a few minutes, the productive structure is not to feed the model the whole video at a coarse rate. It is to find the relevant span first and sample it densely — the video equivalent of retrieval before generation.

That first stage is moment localization: embed frames or shots, embed the question, and retrieve the window whose similarity is highest. Restricting a 173-frame budget to a thirty-second window instead of an hour raises the effective sampling rate from one frame every twenty seconds to nearly six frames a second, which converts a counting question from impossible to routine. It also gives you something to show the user: the timestamp the answer came from, which is the difference between an answer and a citation. The same framing underlies video captioning, where the caption for a long video is normally assembled per shot rather than generated in one pass.