Llama 3.2’s Vision Models: Which Sizes Support Image Input

Llama 3.2 shipped four sizes on 25 September 2024 and only two of them can see. The dividing line is not where you would guess from the parameter counts, because the vision models are not scaled-up text models — they are text models with a separately trained adapter attached.

The split

Model                Image input   Built on
Llama 3.2 1B         no            pruned/distilled from 3.1 8B
Llama 3.2 3B         no            pruned/distilled from 3.1 8B
Llama 3.2 11B Vision yes           Llama 3.1 8B + vision adapter
Llama 3.2 90B Vision yes           Llama 3.1 70B + vision adapter

The 1B and 3B models are text-only, full stop. There is no image mode to enable, no adapter to download, and no prompt that makes them see. They exist for on-device and edge use, where the point is to be small.

Note the parameter arithmetic on the other two, because it explains the otherwise strange sizes. 11B is roughly the 8B text model plus about 3B of vision encoder and cross-attention; 90B is roughly the 70B text model plus about 20B of the same. The vision models are therefore heavier than their text counterparts to serve, while being no better at text — the text weights are the same weights.

Meta’s model cards for all four are published with the weights and in the meta-llama/llama-models repository.

Why 11B and 90B and not 1B and 3B

Meta describes the vision models as built by training a separate image encoder and a set of cross-attention layers, then attaching them to a frozen Llama 3.1 text model. The text weights were not updated. That design choice is the reason for everything else on this page.

It means the vision models inherit the text model’s behaviour exactly — same tokenizer, same chat template, same context length of 131,072, same instruction tuning. It means an existing text-only integration continues to work against them unchanged. And it means vision was an additive project rather than a retraining, which is why the two sizes chosen were the ones with existing 3.1 text models to build on. There was no 3.1 model at 1B or 3B to attach an adapter to.

It also sets a real limitation: because the text model was frozen, the vision models are strong at describing and reasoning over an image and are not a different model in any other respect. Do not expect a 90B Vision model to outperform 3.1 70B on a pure text task. It is the same text model.

The inheritance runs the other way too, which is convenient. Because the text half is unchanged, everything documented for Llama 3.1 applies — the same 131,072-token window described in Llama 3’s context window across releases, the same special tokens, the same stop-token handling, the same instruction tuning. The text-only 1B and 3B models are the exception in this release rather than the vision models: they were derived by pruning and distillation rather than by addition, and their characteristics are covered in the 1B and 3B context window page.

The image token and the prompt format

Images enter the prompt through a reserved token, <|image|>, placed in the message where the image belongs. The image itself is supplied out of band alongside the text.

from transformers import AutoProcessor, MllamaForConditionalGeneration
from PIL import Image

model_id = "meta-llama/Llama-3.2-11B-Vision-Instruct"
processor = AutoProcessor.from_pretrained(model_id)
model = MllamaForConditionalGeneration.from_pretrained(
    model_id, torch_dtype="bfloat16", device_map="auto")

image = Image.open("invoice.png")
messages = [{"role": "user", "content": [
    {"type": "image"},
    {"type": "text", "text": "What is the total on this invoice?"},
]}]

prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(image, prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128)
print(processor.decode(out[0]))

Two things in that snippet are load-bearing. The architecture class is MllamaForConditionalGeneration, not LlamaForCausalLM — a vision checkpoint is a different architecture and will not load through the text class. And the processor, not the tokenizer, renders the template, because it has to align the image with the <|image|> token position.

The image token must precede the text in the message. The documented format places it at the start, and the model was trained that way; putting a question before the image tends to degrade the answer rather than error. Remember also that images consume context — a large image is not free against the 131,072-token budget, so the effective text capacity of a vision request is lower than the headline figure.

The EU clause

Capability and availability are different questions here, and this is the one Llama 3.2 fact most likely to affect whether you can ship. The Llama 3.2 Community License Agreement, published with the release on 25 September 2024, withholds the licence grant for the multimodal models from individuals domiciled in the European Union and from companies whose principal place of business is in the European Union. The text-only 1B and 3B models are not covered by that restriction.

The restriction is on the licence to use the models, not on end users of a product built with them, and Meta stated at the time that downstream products incorporating the models could still be offered to EU end users. That is a distinction with real consequences either way, and it is not one to resolve from a summary. Read Meta’s Llama 3.2 licence text and take advice.

What an image costs you

Because the vision models attach a separately trained encoder rather than tokenizing pixels into the text vocabulary, the cost of an image is not a number you can read off the tokenizer the way you can for text. It is set by the preprocessor: the image is resized and split into tiles, each tile is encoded, and the resulting representations enter through the cross-attention layers.

Two practical consequences follow, and both are worth checking against your own preprocessor config rather than taken from a blog post.

• Resolution is quantised, not continuous. The processor picks from a set of supported aspect ratios and tile counts, so a slightly larger image can cost a whole extra tile while another that is much larger costs the same as a small one. If you control the upload path, resizing to a supported shape before sending is free efficiency. The supported ratios are in preprocessor_config.json alongside the checkpoint.
• Legibility, not size, is what you are buying. The usual failure on documents and screenshots is text too small to read after downscaling, and the fix is cropping to the region of interest rather than sending a higher-resolution version of the whole page. Two cropped requests routinely beat one full-page one.

One more limitation belongs to the cross-attention design rather than to any config: the number of images a single request handles well is small, and support for interleaving several images between blocks of text varies by runtime even for the same weights. If your application needs many images per turn, verify it on the stack you will deploy on rather than on the reference implementation.

Telling them apart

If you have a checkpoint and are unsure, the config answers it without downloading weights:

python -c "import json;c=json.load(open('config.json'));\
print(c['architectures'], 'vision_config' in c)"

# text-only:  ['LlamaForCausalLM'] False
# vision:     ['MllamaForConditionalGeneration'] True

The vision_config key holds the image encoder’s own settings and is present only on the multimodal checkpoints. Naming is not reliable — repository names get changed and quantised re-uploads are inconsistently labelled — but the architecture field is.