Qwen2-VL's Image and Video Input: Dynamic Resolution

Most vision models resize every image to a fixed square and charge a fixed number of tokens for it. Qwen2-VL does not: it encodes an image at close to its native resolution and the token count varies with the image. That is a better model of the world and a harder cost model, and the two facts are the same fact.

Why the token count varies

The approach Alibaba calls naive dynamic resolution is documented in the Qwen2-VL paper (arXiv:2409.12191, September 2024) and on the model cards, for example Qwen/Qwen2-VL-7B-Instruct. The mechanism has three parts.

• The image is resized so that its height and width are each multiples of 28 pixels. Aspect ratio is approximately preserved; there is no crop to a square and no letterboxing.
• The vision transformer splits it into 14×14-pixel patches, as ViTs do.
• A patch merger then combines each 2×2 group of adjacent patches into one token before the tokens reach the language model. That is the step that makes the whole thing affordable: it divides the visual token count by four.

Combine the last two and each visual token covers a 28×28-pixel region of the resized image, which is where the 28 in the first step comes from. Around the sequence, the processor inserts <|vision_start|> and <|vision_end|> markers so the language model knows where the visual span begins and ends.

Position information is handled by M-RoPE, which decomposes the rotary position embedding into temporal, height and width components rather than flattening the image into one dimension. This is why the model can reason about spatial relationships and, for video, about time — and why the same architecture handles a still and a clip without a separate code path.

Working out the token count

The arithmetic below is derived from the documented constants — patch size 14, 2×2 merge, dimensions rounded to multiples of 28 — not measured from a running model. Take a 1024×1024 image:

round each side to a multiple of 28:
  1024 / 28 = 36.57  ->  37 blocks  ->  1036 px

visual tokens = 37 x 37 = 1369

each token covers 28 x 28 = 784 px of the resized image
1036 x 1036 = 1,073,296 px  ->  1,073,296 / 784 = 1369   (consistent)

1,369 tokens for one screenshot-sized image, before any of your text. Halve each dimension to 512×512 and you get 19×19 = 361 tokens: a quarter of the pixels is a quarter of the tokens, because the relationship is linear in area and therefore quadratic in the side length. That is the single most useful fact for controlling cost here. An image sent at twice the necessary width costs four times the necessary tokens.

min_pixels and max_pixels

The processor takes two bounds on the resized area, expressed in pixels. Images larger than max_pixels are scaled down until they fit; images smaller than min_pixels are scaled up. Because a token is 784 pixels, the model card writes the bounds as multiples of 28*28, which makes them read directly as token counts:

from transformers import AutoProcessor

processor = AutoProcessor.from_pretrained(
    "Qwen/Qwen2-VL-7B-Instruct",
    min_pixels=256 * 28 * 28,    # floor:   256 visual tokens
    max_pixels=1280 * 28 * 28,   # ceiling: 1280 visual tokens
)

Applying that ceiling to the 1024×1024 example above is worth following through, because the rounding does something you would not predict. 1,369 tokens exceeds the 1,280 cap, so the image is scaled down; but the result must still be a whole number of 28-pixel blocks, and 36×36 = 1,296 is still over the cap, so it lands at 35×35 = 1,225 tokens. The effective ceiling is not 1,280 but the largest square block count under it. Token counts from this pipeline come in a discrete lattice, not a continuum, which is why a small change to an input image can move the count by a step and then not at all.

The processor’s own built-in defaults are much wider than the card’s recommended pair, so if you set nothing you can find a single high-resolution photograph consuming several thousand tokens. Set both bounds explicitly. Setting them per request rather than globally is also legitimate and often correct: a diagram that must be read needs a high ceiling, a thumbnail being classified does not.

Video: frame sampling and temporal merging

Video is handled as a sequence of frames through the same encoder, with two mechanisms on top that decide how many frames there are.

The first is sampling rate. The helper package Alibaba ships, qwen-vl-utils, samples frames at a fixed frames-per-second rate rather than taking every frame, and exposes it as an fps field on the message content. Doubling it doubles the token cost of the clip. The package also enforces a floor and a ceiling on the number of sampled frames, so a long video is not sampled indefinitely — those bounds are module-level constants and have changed between releases, so read them from the version you have installed rather than assuming a value.

The second is temporal merging. The vision encoder applies a 3D convolution over pairs of consecutive frames, so two sampled frames produce one set of visual tokens rather than two. The practical effect is that the cost of a clip is roughly half what per-frame arithmetic suggests — and that a frame count should be even.

messages = [{
    "role": "user",
    "content": [
        {"type": "video", "video": "file:///path/to/clip.mp4",
         "fps": 1.0, "max_pixels": 360 * 28 * 28},
        {"type": "text", "text": "What happens after the door opens?"},
    ],
}]

Note the separate, much lower max_pixels on the video entry. This is the important lever: per-frame resolution multiplies by frame count, so a ceiling that is merely generous for a still image becomes ruinous across a hundred frames. Video work is nearly always a matter of trading spatial detail for temporal coverage, and the two knobs — fps and per-frame max_pixels — are how you spend the budget.

Budgeting visual tokens against the context

Visual tokens are context tokens. They come out of the same window as your text, and on the Qwen2-VL and Qwen2.5-VL checkpoints that window is the same 32,768-token native figure the text models carry, with the same YaRN caveat described in the context window page. Two consequences follow.

• Multi-image prompts run out fast. At the card’s recommended 1,280-token ceiling, twenty images is 25,600 tokens — most of a 32K window before a word of instruction. If you need many images in one request, lower the ceiling for all of them rather than accepting the default for a few.
• The failure mode is a hard error, not degradation. Exceeding the context with visual tokens produces a length error like any other, and the number that caused it is invisible in your request, which contains a file path rather than a token count. Compute the expected visual tokens before sending if you are anywhere near the limit — the arithmetic above is all you need.

For how another family answers the same design question, with a fixed tile cost rather than a continuous one, see Llama 3.2’s vision input and Gemini’s video token cost. The tile model is easier to budget and coarser; Qwen’s is harder to budget and preserves detail that tiling destroys. Which is better depends entirely on whether your images have small text in them.