Video Understanding: Frames, Sampling and Cost

There is no video encoder in most of these models. There is an image encoder being called repeatedly, and a decision — yours or the provider’s default — about which frames to call it on. Get that decision right and video is affordable. Take the default on a long video and it is not.

Video is frames, and frames are images

The common design samples the video at some frames per second, encodes each sampled frame exactly as it would encode a still, and lays the resulting token runs end to end in the context. Google documents this for the Gemini API explicitly: as of August 2026 its docs describe video sampled at one frame per second by default, with each frame costing on the order of 258 tokens, and the audio track tokenised separately at roughly 32 tokens per second. Other vendors differ in the constants and in whether they downsample frames harder, but the structure is the same everywhere.

Two consequences fall straight out. Temporal resolution is capped by the sampling rate — at 1 fps a gesture lasting 200 ms happened between frames and does not exist. And motion is inferred from differences between stills rather than perceived; questions like “did the ball cross the line” are answered from whichever frames happened to be sampled.

It also means the honest framing of most video tasks is not “can the model understand video” but “did my sampler include the frame that contains the answer”. That reframing is useful because it moves the problem somewhere you have control. When a video question comes back wrong, the first thing to check is not the prompt and not the model — it is the extracted frames. Dump them to a folder and look. A surprising share of video failures turn out to be a sampler that captured the two seconds before and after the event and nothing in between.

The arithmetic that decides everything

tokens = duration_s * fps * tokens_per_frame
       + duration_s * tokens_per_second_of_audio

worked, at 258 tokens/frame and 32 tokens/s audio:

  60 s clip   @ 1 fps    60 * 258 =    15,480  + 1,920  =  17,400
  10 min      @ 1 fps   600 * 258 =   154,800  + 19,200 = 174,000
  60 min      @ 1 fps  3600 * 258 =   928,800  + 115,200 = 1,044,000
  60 min      @ 0.2 fps 720 * 258 =   185,760  + 115,200 =  300,960

at an input rate of $1.00 per million tokens:
  the 10-minute clip costs  $0.17
  the hour at 1 fps costs   $1.04
  the hour at 0.2 fps costs $0.30

The rate is a placeholder; substitute yours. The shape is the point. An hour of video at full default sampling is a million-token request, which is not merely expensive — it is at or past the context limit of most models, and it makes time-to-first-token minutes rather than seconds because all of it has to be prefilled.

Notice also that at low frame rates the audio track becomes the dominant term. Halving the frame rate on a talking-head video saves much less than you expect, because the words are what you were paying for.

Four sampling strategies

• Uniform. Every n-th frame. Trivial, predictable, and wasteful on static content — a lecture slide held for four minutes is encoded 240 times at 1 fps, and 239 of them are identical.
• Scene change. Sample when the picture actually changes. On screen recordings, security footage and edited video this cuts frame count by an order of magnitude with no loss of content, and it is the single highest-leverage change available.
• Two-pass, coarse to fine. Pass one at a very low rate over the whole video to locate the interesting interval; pass two at a high rate over the 30 seconds that matter. For “find the moment when X happens” this is dramatically cheaper than one dense pass, and it degrades gracefully — a missed interval costs one extra coarse pass, not a re-run of everything.
• Transcript-first. If the answer is in the speech, transcribe the audio, answer from the text, and send frames only for the timestamps the transcript points at. Speech-to-text over an hour of audio costs a small fraction of a million image tokens.

Extracting scene changes

ffmpeg does the scene detection, so this is a one-liner rather than a project:

# frames where the scene score exceeds 0.3, with timestamps
ffmpeg -i talk.mp4 \
  -vf "select='gt(scene,0.3)',showinfo,scale=768:-1" \
  -vsync vfr -q:v 3 frame_%04d.jpg

# a guaranteed floor: at least one frame every 30 s
ffmpeg -i talk.mp4 -vf "fps=1/30,scale=768:-1" keyframe_%04d.jpg

Run both and merge. The scene filter alone will skip a ten-minute static shot entirely, which is usually wrong — you want a heartbeat frame so the model knows nothing changed rather than knowing nothing. Tune the threshold on your own content: 0.3 is a reasonable start, lower for slow documentary footage, higher for fast cuts.

Keep the timestamps. Sending frames without them costs you every question that involves ordering or duration, and the fix is a text line before each image — [00:04:12] — which costs a handful of tokens and makes the model’s answers citable back to the video.

Two smaller details save real money at scale. Scale the frames down as part of the same ffmpeg invocation rather than after — the filter chain above already does it, and there is no reason to pay to transfer pixels you will discard. And write JPEG rather than PNG for photographic frames: it does not change the token count, but on a job producing tens of thousands of frames it changes disk and transfer by an order of magnitude.

Do not forget the audio track

For a substantial share of real video work — meetings, lectures, interviews, support calls — the visual channel is nearly redundant and the audio carries the content. Transcribing first and sampling frames sparsely is not a compromise on those; it is a better system, because the transcript is searchable, cacheable, cheap to re-query, and reviewable by a human, none of which is true of a million image tokens you have to re-send every time someone asks a follow-up question.

The general pattern this suggests is worth stating on its own, because it applies to every long video you will ever process: extract a durable, cheap intermediate once, and answer questions from that. A transcript with timestamps, a list of scene boundaries, and a one-paragraph description of each scene together make a searchable index of an hour of video that costs a few thousand tokens to hold in context. Questions are answered from the index; frames are fetched only when the index says which second to look at. This is the same architecture as retrieval over documents, applied to time, and it turns an unaffordable per-question cost into an affordable per-video one.