Action Recognition in Video, Explained

Take a four-second clip at 25 fps — 100 frames of a hand, a table and a cup. Played forward it is “picking something up”. Played backward it is “putting something down”. A model that classifies each frame and averages the results returns the same answer for both, and no amount of extra training data changes that. Action recognition is the set of techniques that make the two answers different.

The time-reversal test

The reason the averaging model fails is arithmetic, not weakness. If the clip representation is the mean of per-frame features, then it is invariant to any permutation of the frames, because addition is commutative. Reversal is a permutation. So the representation is provably identical, the classifier sees identical input, and the two opposite actions are indistinguishable by construction. The same holds for max pooling and for a bag-of-visual-words histogram over frames.

This gives you a cheap and decisive diagnostic for any video model you did not build: feed it a clip and its reverse. If the output is unchanged, the temporal path carries no information, whatever the architecture diagram says. It is the same test that catches a captioning model whose aggregation step is a mean pool.

Why frame-level models score so well anyway

Here is the uncomfortable part. On the standard action-recognition benchmarks, single-frame models do far better than the time-reversal argument suggests they should. The reason is scene bias: many action classes are almost perfectly predicted by the setting. A basketball court implies “playing basketball”; a pool implies “swimming”; a kitchen and a knife imply “chopping”. The model is not recognising an action at all, it is recognising a place, and the benchmark rewards it identically.

The dataset built specifically to remove that shortcut is Something-Something, released by Goyal and colleagues in 2017. Its labels are templates with the object abstracted away — “Pushing something from left to right” and “Pushing something from right to left” are separate classes, and both contain every object. Appearance tells you nothing; the dataset paper on arXiv sets out the design. If you need to know whether a model does temporal reasoning, its behaviour on temporally-defined class pairs is the question to ask, not its headline accuracy on a scene-biased set like Kinetics, whose construction Kay and colleagues describe in the Kinetics dataset paper.

How motion actually gets modelled

• Two-stream networks. Simonyan and Zisserman’s 2014 design runs one network on RGB frames and a second on stacked optical flow, then fuses the predictions — the two-stream paper. Optical flow is an explicit, hand-computed motion field, which makes the temporal stream unable to cheat on appearance. Its cost is that flow computation often dominates the inference budget.
• 3D convolutions. Extend the kernel along time so a single filter spans several frames. I3D’s contribution was practical: inflate a 2D ImageNet-pretrained network into 3D by replicating each kernel along the temporal axis and dividing by the number of copies, which preserves the pretrained responses on a static input. Carreira and Zisserman describe it in the I3D paper.
• Factorised 3D. R(2+1)D replaces a 3×3×3 convolution with a 1×3×3 spatial convolution followed by a 3×1×1 temporal one. Fewer parameters, an extra non-linearity between the two, and an easier optimisation problem — Tran and colleagues on spatiotemporal convolutions.
• Two frame rates at once. SlowFast runs a low-frame-rate pathway with high channel capacity alongside a high-frame-rate pathway with few channels, on the argument that spatial semantics change slowly and motion changes quickly, so they deserve different sampling. Feichtenhofer and colleagues published it in 2018.

Sampling at training and at test time

A 3D network consumes a fixed clip length — 8, 16 or 32 frames is typical — which is a fraction of a second to a couple of seconds. A ten-second action does not fit. Two conventions handle this and they have different consequences.

Dense clips with multi-view testing. Sample several short clips at different offsets, and several spatial crops of each, run the network on all of them and average the softmax outputs. The common protocol on Kinetics is ten temporal clips by three spatial crops, which is thirty forward passes per video. This is the number people forget when they quote a model’s GFLOPs: the reported per-clip cost multiplied by thirty is the actual cost of the reported accuracy.

Sparse segment sampling. Temporal Segment Networks split the video into K equal segments and take one short snippet from each, so the sampled frames span the whole video at training time rather than a slice of it. Wang and colleagues introduced it in the TSN paper, and it remains the cheap default for long actions.

The choice interacts with frame rate in a way that catches people out. A 16-frame dense clip sampled at every second frame from 30 fps footage covers about one second of real time; the same 16 frames sampled every eighth frame cover four seconds, at the cost of aliasing any motion faster than four hertz. There is no default that is right for both a hand gesture and a golf swing, which is why the sampling stride belongs in your evaluation sweep alongside the learning rate rather than being inherited from whichever reference implementation you cloned.

Where it stops working

Trimmed-clip classification assumes somebody already found the clip. Real footage is untrimmed, and the task becomes temporal action localization: output start and end times as well as a label, scored by mean average precision at a temporal IoU threshold. That is a substantially harder problem, and it shares its geometry with language-queried moment localization — same IoU arithmetic, different query type.

Three other limits are worth planning around. Long-horizon actions — “assembling a shelf” — exceed any clip length and need a hierarchy or an explicit memory. Fine-grained distinctions within a family, such as which of two similar strokes a swimmer used, depend on detail that survives neither the resize nor aggressive compression. And multi-person scenes need the action attributed to a person, which means detection and tracking with identity underneath the classifier rather than a single label for the frame.