Video Input in the Gemini API: How Frames Become Tokens

Video is not charged by megabyte or by minute. Gemini samples frames at a fixed rate, charges each sampled frame as an image, charges the audio track separately per second, and adds the two. Once you know those three facts the cost of any clip is arithmetic—and the API will confirm it before you send.

How a video becomes tokens

The model does not watch video. It receives a sequence of stills and a transcribable audio stream. Google’s video understanding documentation documents the pipeline: the video is sampled at one frame per second by default, each sampled frame is tokenised as an image, and the audio track is tokenised at its own per-second rate. Timestamps are made available to the model so it can refer to moments in the clip.

Two consequences follow before any arithmetic. First, the frame rate of your source file is irrelevant to cost—a 60 fps recording and a 24 fps recording of the same length cost the same, because both are resampled to one frame per second. Second, and less comfortably, anything that happens between sampled frames is not seen. A gesture, a flash of text on screen, a single frame of a slide transition: if it falls between two one-second samples, it is not in the model’s input at all. “The model missed something obvious in the video” is very often the sampling rate rather than the model.

The documented rates

Google’s token-counting documentation gives the figures this page uses. Stated as assumptions, so that the arithmetic below is checkable:

• Sampling rate: 1 frame per second, by default.
• Per frame, default media resolution: 258 tokens — the same per-image figure Gemini uses for a standard still.
• Per second of audio: 32 tokens.
• Per frame, low media resolution: 66 tokens.

Adding the first three gives roughly 290 tokens per second of video with sound, which is consistent with the approximately-300-tokens- per-second figure Google quotes for video in its own summaries. That consistency is the only cross-check available on a page like this and it is worth stating rather than hiding.

A worked clip

Take a twelve-minute product demo with its soundtrack—720 seconds, an entirely ordinary length for the kind of clip people actually send.

Assumptions: 1 fps sampling, 258 tokens per frame at default
media resolution, 32 tokens per second of audio.

Frames
  720 seconds x 1 frame per second   = 720 frames
  720 frames x 258 tokens per frame  = 185,760 tokens

Audio
  720 seconds x 32 tokens per second =  23,040 tokens

Total input for the clip alone       = 208,800 tokens

Add the prompt, the system instruction and any conversation
history on top; the clip is not the whole request.

Two hundred thousand tokens for twelve minutes of video is the number worth internalising, because it reframes what video input is. It is not an attachment. A single twelve-minute clip is a fifth of a million-token context window, and asking five follow-up questions about it re-sends all 208,800 tokens five times unless the clip is behind an explicit context cache. Video is the workload where caching stops being an optimisation and becomes the difference between viable and not.

The same clip at low media resolution:

  720 frames x 66 tokens per frame   =  47,520 tokens
  720 seconds x 32 tokens per second =  23,040 tokens
                                       ------------------
                                        70,560 tokens

About a third of the default-resolution cost, for the same clip.

The two levers that change the number

Both are set per part rather than per request, which is what makes them usable—you can send one clip at high fidelity and another at low in the same call.

Sampling rate and clipping, via videoMetadata

A video part accepts a video_metadata object carrying a start offset, an end offset and a custom frames-per-second value. Sending only the relevant ninety seconds of a one-hour recording is a fortyfold reduction that no compression setting can match:

{
  "file_data": {
    "mime_type": "video/mp4",
    "file_uri": "https://generativelanguage.googleapis.com/v1beta/files/xyz789"
  },
  "video_metadata": {
    "start_offset": { "seconds": 1830 },
    "end_offset":   { "seconds": 1920 },
    "fps": 0.5
  }
}

fps below 1 undersamples: at 0.5 you get one frame every two seconds, halving the frame cost. That is the right setting for a static talking-head recording or a screencast of a slide deck, where nothing changes within two seconds. It is the wrong setting for sport, for driving footage, or for anything where the point is motion—and raising fps above 1 for those cases raises the cost in exact proportion.

Media resolution

generationConfig.mediaResolution selects the per-frame budget. Low resolution is the right default for clips whose content is large-scale motion or scene identification, and the wrong one for anything requiring reading text off the screen—a slide, a dashboard, a licence plate—where 66 tokens per frame does not carry enough detail.

The ceilings that apply before cost does

Cost is rarely the first thing that stops a video request. Four constraints sit in front of it, and each produces a different failure.

• Duration. Maximum clip length is bounded by the context window and the per-second rate together, which is why the documented maxima differ by model and by media resolution: a lower per-frame cost buys a longer clip out of the same window. The arithmetic earlier on this page is the arithmetic that produces those maxima, so you can work out your own ceiling for a given model rather than looking one up.
• Inline size. A video sent as inline_data counts against the total request size limit—20 MB in the documented case, before base64 inflation. In practice this means inline video is for short clips only, and everything else goes through the File API.
• Processing state. Uploaded video is processed asynchronously. The file object has a state that begins as PROCESSING and becomes ACTIVE, and a generation request naming a file that is still processing fails. This is the most common first-run error with video, and the fix is a poll on the file resource rather than a retry on the generation.
• Retention. Files uploaded through the File API are held for a documented 48 hours and then deleted. A stored URI in a database outlives its file, so treat the handle as ephemeral and keep the original.

YouTube URLs are the exception worth knowing about: a public YouTube link can be passed as a file_data.file_uri directly, with no upload step, subject to its own documented limits on how many may be used per request and per day. That is convenient and it is also a dependency on a video staying public, which is a different reliability profile from a file you uploaded.

Getting the real figure

Every number above is a derivation. The API will tell you the actual count for your file, your model and your settings, and it costs nothing but a round trip:

1. Upload the clip through the File API, which is required above the inline-request size limit anyway, and wait for its state to become ACTIVE. Video files are processed asynchronously and are not usable until then.
2. Call countTokens on the exact request body you intend to send, including the file part and any video_metadata.
3. Read totalTokens. Multiply by the current input price for your model. That is the real cost of the call.

POST .../v1beta/models/gemini-2.5-flash:countTokens

{
  "contents": [
    {
      "role": "user",
      "parts": [
        { "file_data": { "mime_type": "video/mp4", "file_uri": "https://generativelanguage.googleapis.com/v1beta/files/xyz789" } },
        { "text": "List every on-screen price shown in this demo, with its timestamp." }
      ]
    }
  ]
}

After the call, usageMetadata.promptTokenCount in the response reports what you were actually billed, and comparing it against yourcountTokens estimate is the check that catches a resolution setting or a metadata field that did not apply the way you expected.