Video Captioning Models: How They Turn Frames Into a Sentence

A video captioning model is three components in a line: something that turns each sampled frame into vectors, something that combines those vectors across time, and something that decodes the result into words. Almost every practical property of the system — what it can see, what it costs, what it gets wrong — is decided by the first step and by how many frames you let it have.

The shape of the pipeline

The task is fixed: input an untrimmed or trimmed clip, output one or more sentences describing what happens in it. The architecture that solves it has been stable in outline since Venugopalan and colleagues published S2VT in 2015, which fed per-frame convolutional features into a recurrent decoder — see the S2VT paper on arXiv. What has changed since is every individual box: the frame encoder became a vision transformer, the recurrent aggregation became attention or a learned resampler, and the decoder became a language model that was pretrained on text and never saw a video during that pretraining.

Keeping the three boxes separate in your head is worth doing, because they fail differently. An encoder failure produces a caption about the wrong objects. An aggregation failure produces a caption that is correct about objects and wrong about what happened to them. A decoder failure produces fluent language that was never conditioned on the video at all.

The frame encoder

Each sampled frame is resized to the encoder’s input resolution — commonly 224×224 or 336×336 — and cut into fixed-size patches. A ViT-L/14 backbone at 224×224 produces a 16×16 grid, so 256 patch tokens per frame plus a class token. Those tokens are what the rest of the system sees. Everything outside the crop, and every detail finer than a patch after downscaling, is gone before any temporal reasoning starts.

This is why captioning models are poor at small text and small objects in wide shots. A road sign that occupies 30 pixels in a 1920-wide frame occupies about 3.5 pixels after the resize to 224, which is a quarter of one patch. No amount of temporal modelling recovers it; the fix is a higher-resolution encoder, a tiling scheme that runs the encoder over crops, or a separate scene text recogniser whose output is passed to the decoder as text.

Aggregating over time

Now you have N frames × 257 tokens and a decoder that wants a sequence. There are four families of answer, in rough order of how much temporal information survives.

• Mean pooling. Average the per-frame vectors. Cheap, and permutation-invariant: it returns the same representation for a clip and its time-reverse, which is a mathematical property of the mean rather than a shortcoming that better training fixes. Adequate for “a kitchen with a person and a kettle”, useless for “puts the kettle down”.
• Recurrent or temporal-attention aggregation. Run an LSTM over the frame sequence, or let the decoder attend over frames with a learned weight per step. Order is preserved, and the decoder can look at different frames while emitting different words.
• Spatio-temporal encoders. Skip the per-frame bottleneck and convolve or attend in three dimensions from the start — I3D, or TimeSformer’s divided space-time attention, which splits attention into a temporal pass over the same patch position across frames and a spatial pass within each frame. That split is a cost decision: joint attention over T·HW tokens is quadratic in the product, and dividing it makes the cost the sum of two much smaller quadratics. Bertasius and colleagues set this out in the TimeSformer paper.
• Learned resamplers. The dominant approach in current vision-language models: a small set of learned query vectors cross-attends to the full frame-token set and compresses it to a fixed budget regardless of input size. Flamingo’s Perceiver Resampler and BLIP-2’s Q-Former are the two canonical designs — Li and colleagues describe the Q-Former in the BLIP-2 paper. The output is a constant number of tokens, which is what makes a long video tractable for a language decoder at all.

The frame budget, worked

Do the arithmetic once and the design of every current system stops looking arbitrary. Take a ten-minute video sampled at one frame per second: 600 frames. At 256 patch tokens per frame that is 153,600 tokens before a single word is generated. Send all of them to a language decoder and you have spent a large context window on one video, at quadratic attention cost, for a two-sentence answer.

600 frames x 256 tokens          = 153,600 tokens   (raw ViT-L/14 patches)
600 frames x  64 tokens          =  38,400 tokens   (Perceiver-style resampler)
600 frames x  32 tokens          =  19,200 tokens   (Q-Former, 32 queries)
 16 frames x  32 tokens          =     512 tokens   (uniform sampling + resampler)

The last line is what most hosted models actually do, and it has a consequence worth stating plainly: sixteen frames spread uniformly over ten minutes is one frame every 37.5 seconds. Any event shorter than that interval is invisible with probability close to one. A model that captions such a video confidently is not lying about what it saw; it is describing the frames it was given, and the sampler discarded the event. If the thing you care about is brief, uniform sampling is the wrong sampler and content-aware keyframe selection is the fix.

Where captions go wrong

Three failure modes recur, and they are distinguishable if you know what to look for.

• Object hallucination. The caption names an object that is not present, usually one that co-occurs with the real scene in the training distribution — a laptop in an office, a ball on a pitch. This is a property of the language decoder’s priors overwhelming weak visual evidence, and it is the same phenomenon described for still images in vision model hallucination.
• Static description of a dynamic clip. The caption lists what is in the frame rather than what happens. Almost always an aggregation problem, and the diagnostic is to feed the clip reversed: if the caption is unchanged, the temporal path is not carrying information.
• Correct captions that score badly. n-gram metrics such as CIDEr compare against a handful of human references, so a caption that is accurate but phrased differently is penalised, and a generic caption that matches the reference vocabulary is rewarded. Vedantam and colleagues introduced CIDEr with exactly this consensus framing, which is a virtue for benchmark comparison and a trap if you use it as a product metric. Score against the task you actually have — retrieval accuracy, or human judgement on a sample.

If your downstream use is answering questions rather than producing prose, captioning may be the wrong intermediate representation entirely: a caption is a lossy summary chosen without knowing the question, which is the argument for asking the model directly or for indexing frames and doing multimodal retrieval instead.