Gemma 3's Image Input: Resolution, Tokens and Pan-and-Scan

Gemma 3 was the generation that made the family multimodal, but not uniformly: the 1B is text-only, and the sizes that do see images see them through a fixed-resolution encoder that turns each one into a fixed number of tokens. Both facts change how you budget context.

Which sizes see images

Gemma 3 shipped in March 2025 at 1B, 4B, 12B and 27B. The 4B, 12B and 27B checkpoints include a vision encoder and accept interleaved image and text input. The 1B does not: it is text-only, and it also carries a shorter 32K context against 128K on the larger three. Google states this on the Gemma 3 model cards, for example gemma-3-4b-it.

You can tell them apart without reading a card. A multimodal Gemma 3 checkpoint loads with an AutoProcessor that has an image processor attached, and its config carries a vision tower:

from transformers import AutoConfig

cfg = AutoConfig.from_pretrained("google/gemma-3-4b-it")
print(hasattr(cfg, "vision_config"))   # True for 4B / 12B / 27B

The encoder and the fixed resolution

Images go through a SigLIP vision encoder, roughly 400M parameters, shared across the multimodal sizes rather than scaled with the language model. That is a deliberate choice: the 4B and the 27B see images with the same eyes and differ in what they do with what they see.

The encoder takes a fixed square input, documented at 896 by 896 pixels. Whatever you supply is resized to that. There is no higher-detail mode you can request per image the way some hosted APIs offer, and there is no benefit to sending a 4000-pixel-wide photograph beyond what survives the downscale. The practical reading is that fine detail, small text in a screenshot or a thin line in a chart, is at risk, and that is what pan-and-scan exists to mitigate.

A shared encoder has a consequence that shows up in evaluation. Because perception is identical across the sizes, a task that fails on the 4B because the model cannot see the detail will fail on the 27B too, and no amount of moving up the family fixes it. A task that fails because the model saw the detail and reasoned about it badly is exactly the kind that a larger size can fix. Separating those two failure modes before you reach for a bigger checkpoint saves a great deal of GPU time, and the test is simple: crop the image to the region in question and ask again. If the small model gets it right on the crop, the problem is resolution, not capability.

What one image costs in tokens

The encoder’s output is pooled down to a fixed sequence of 256 tokens per image before it enters the language model. Fixed is the operative word: a simple diagram and a dense photograph cost the same, because the cost is set by the architecture and not by the content.

That makes budgeting arithmetic rather than guesswork. On a 128K window, images are cheap in relative terms:

# Rough context budget for a multi-image prompt on Gemma 3 4B/12B/27B.
images        = 8
tokens_image  = 256          # fixed, per image, before pan-and-scan crops
text_tokens   = 1500
reserved_out  = 1024

used = images * tokens_image + text_tokens + reserved_out
print(used, "of 131072")     # 4572

Compare that with the same eight images on a Gemma 2 style 8,192-token budget and the point of the Gemma 3 context extension becomes obvious: 2,048 tokens of images is a quarter of the older window and under two per cent of the newer one.

A fixed per-image cost is unusual enough to be worth contrasting with the alternative. Several hosted vision APIs bill images by tile count, so cost scales with resolution and you can trade money for detail per request. Gemma 3 makes that choice once, at the architecture level: no knob, no per-request detail parameter, and a token count you can put in a spreadsheet. The cost of the simplicity is that you cannot pay for a closer look at one difficult image, which is precisely the gap pan-and-scan fills, and it fills it by changing the number of images rather than the resolution of one.

Pan-and-scan, and when it fires

A fixed square encoder deals badly with a wide screenshot or a tall receipt, because squashing either into 896 by 896 destroys exactly the detail you sent it for. Gemma 3’s answer is an adaptive windowing scheme, described in Google’s documentation as pan-and-scan: the image is cropped into several square regions, each crop is encoded separately at the native resolution, and the crops are passed to the model together.

The consequence for your token budget is direct and it is the thing people miss. Each crop is another 256 tokens. An image that triggers three crops costs 768 tokens plus whatever the full-image pass costs, not 256. When a multi-image prompt comes in far heavier than your arithmetic predicted, this is usually why.

• It fires on aspect ratios far from square, and on images large enough that downscaling would lose meaningful detail.
• It is a processor-level behaviour, controlled by arguments to the image processor rather than by the model, so different serving stacks may default it differently.
• Turning it off makes cost perfectly predictable and makes wide screenshots noticeably worse. Turning it on does the reverse. There is no setting that avoids the trade.

Preparing images in practice

Use the processor that ships with the checkpoint rather than resizing yourself. It knows the target resolution, the normalisation constants and the placeholder token that marks where the image sits in the text stream, and all three have to agree with how the model was trained.

from transformers import AutoProcessor, Gemma3ForConditionalGeneration
from PIL import Image

proc = AutoProcessor.from_pretrained("google/gemma-3-4b-it")
model = Gemma3ForConditionalGeneration.from_pretrained("google/gemma-3-4b-it")

messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": Image.open("chart.png")},
        {"type": "text", "text": "Which quarter has the largest drop?"},
    ],
}]

inputs = proc.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True, return_tensors="pt"
)
print(inputs["input_ids"].shape)   # includes the image placeholder tokens
out = model.generate(**inputs, max_new_tokens=256)
print(proc.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Printing the input shape before generating is the habit worth keeping. It is the only place the real image token cost, crops included, is visible, and it turns a budgeting surprise into a number you can see in a log.

Two further practicalities. The placeholder token that marks an image’s position in the text stream is inserted by the processor, and the order of images relative to text is preserved, so you can interleave several images with commentary between them and the model will associate each with the text around it. Referring to them by position in your question, asking about the second image rather than about the image, works for the same reason and is worth doing whenever you send more than two. And image inputs go through the same start-of-turn template as text: an image lives inside a user turn, not in a separate channel. If you are building the prompt string by hand for a serving stack that takes raw text, this is the part that will not work, because the placeholder has to line up with pixel data the tokenizer never sees. Use the processor.

On the input side, send the image at a resolution at least as large as the encoder wants and let the processor downscale. Pre-shrinking a screenshot to save bandwidth throws away detail before pan-and-scan gets a chance to use it, and the model has no way to ask for it back. The opposite habit, sending very large files, costs upload time and nothing else, since everything above the encoder’s working resolution is discarded anyway.