Phi-3.5-Vision's Multimodal Input and Image Token Cost

Phi-3.5-vision charges for an image in tokens, and the count is not fixed — it scales with resolution in 336-pixel steps. Microsoft publishes the formula on the model card, and it is worth actually evaluating before you send a screenshot.

How an image enters the prompt

Images are referenced by a placeholder in the text of the turn, and supplied separately to the processor. The placeholders are numbered and one-indexed:

from PIL import Image
from transformers import AutoProcessor, AutoModelForCausalLM

proc = AutoProcessor.from_pretrained(
    "microsoft/Phi-3.5-vision-instruct", trust_remote_code=True
)

images = [Image.open("dashboard.png"), Image.open("legend.png")]
prompt = (
    "<|user|>\n"
    "<|image_1|>\n<|image_2|>\n"
    "Which series in the first chart is the one the legend calls 'p95'?<|end|>\n"
    "<|assistant|>\n"
)

inputs = proc(prompt, images, return_tensors="pt")
print(inputs["input_ids"].shape)   # total prompt length, images included

That last line is the honest way to get the cost: the processor expands each placeholder into the real run of image tokens, so the shape of input_ids is what the model will actually read. The formula below tells you what to expect before you run it.

The card documents multi-frame input — several images in one prompt, which is what makes the model usable for comparing screenshots or reading a short video as frames. Each one is billed separately, at the rate derived below.

The HD transform: crops plus a thumbnail

The vision encoder is a CLIP ViT-L/14 operating at 336×336 pixels. That is a fixed input size, so a naive implementation would downscale every image to 336 square and lose all fine detail — which is fatal for the documents, charts and screenshots this model is aimed at.

Phi-3.5-vision uses a dynamic-cropping scheme instead. The image is resized so both sides are multiples of 336, then split into a grid of 336×336 blocks, each encoded independently at full detail. Alongside the grid, a single downscaled copy of the whole image is encoded as a global view, so the model sees both the fine detail of each region and the overall layout.

Inside each block, ViT-L/14 produces a 24×24 grid of patches — 576 of them, since 336 ÷ 14 = 24. Those are pooled 2×2 before entering the language model, giving 12×12 = 144 tokens per block. That 144 is the number you will see in the formula, and knowing where it comes from is what makes the formula predictable instead of magic.

The documented token-count formula

Microsoft’s Phi-3.5-vision-instruct model card gives the token count for an image of height h and width w, after resizing, as:

tokens = ((h // 336) * (w // 336) + 1) * 144
         + 1
         + ((h // 336) + 1) * 12

where:
  (h // 336) * (w // 336)   number of full-detail crops
  + 1                       the global thumbnail view
  * 144                     tokens per encoded 336x336 block
  + 1                       a leading separator token
  ((h // 336) + 1) * 12     row separators, one set per grid row

The structure is worth reading rather than just the result. The count is dominated by the crop grid, which grows with the area of the image — double both dimensions and you quadruple the crop count. The separators are a rounding term that never matters much.

Three images, worked through

Assumptions for all three: the image has already been resized to exact multiples of 336, no cropping strategy other than the documented one applies, and the text of the prompt is excluded.

A 672 x 672 image  (a 2x2 grid)
  crops       = 2 * 2 = 4
  (4 + 1) * 144         = 720
  + 1                   =   1
  (2 + 1) * 12          =  36
  total                 = 757 tokens

A 1344 x 1344 image  (a 4x4 grid)
  crops       = 4 * 4 = 16
  (16 + 1) * 144        = 2448
  + 1                   =    1
  (4 + 1) * 12          =   60
  total                 = 2509 tokens

A 1344 x 672 landscape screenshot  (a 2 high x 4 wide grid)
  crops       = 2 * 4 = 8
  (8 + 1) * 144         = 1296
  + 1                   =    1
  (2 + 1) * 12          =   36
  total                 = 1333 tokens

The middle case is the one to remember. A 1344-pixel-square image — not a large image by any modern standard — is roughly 2,500 tokens, which is more than most of the text prompts anyone sends this model. Quadrupling the pixel count from the first case took the cost from 757 to 2,509, a little over 3×, exactly as the area term predicts.

What this does to a 128K window

Phi-3.5-vision is documented at a 128K context length, which sounds like room for an unlimited number of images and is not. At the 1344-square rate above, fifty images is about 125,000 tokens — the whole window, with nothing left for the question or the answer. At the 672-square rate the same fifty images cost about 38,000, which leaves room to work.

The lever is resolution, and it is a sharp one. Three practical consequences:

• Downscale deliberately. If the task is “what kind of chart is this”, 672 square is plenty and costs a third of what 1344 does. If the task is reading 9-point text in a screenshot, it is not, and the cost is the price of the task.
• Crop before sending. Sending the relevant region at high resolution beats sending the full page at high resolution, and usually beats sending the full page downscaled, because the resolution that matters is preserved where it is needed.
• Remember the KV cache. Image tokens occupy the same cache as text tokens. The per-token memory arithmetic in the Phi-3 context window page applies unchanged, so 2,500 image tokens is 2,500 tokens of cache on top of your prompt.

Multi-frame input and video

The model card documents multi-frame input, and this is where the arithmetic stops being academic. There is no video decoder in the model: video means frames you extracted yourself, each one an image billed at the rate derived above. The sampling rate is entirely your decision, and it is the single largest cost lever in a video pipeline.

Work an example with the assumptions stated. A 60-second clip sampled at one frame per second is 60 images. At 672 square that is 60 × 757 = 45,420 tokens, which fits in the 128K window with room for a question. At 1344 square the same clip is 60 × 2,509 = 150,540 tokens, which does not fit at all. Sampling the same clip at one frame every five seconds gives 12 frames: 9,084 tokens at the lower resolution, 30,108 at the higher. The choice between those four numbers is made by two parameters, and neither of them is a model setting.

Practical consequences for anything frame-based:

• Sample on change, not on a clock. A fixed interval spends the budget evenly across a video that is mostly static. Extracting frames on scene change, or on a perceptual-hash difference threshold, puts the tokens where the information is.
• Resolution per frame, not per video. A high-detail frame where text needs reading and low-detail frames for continuity costs far less than uniform high detail, and reads better than uniform low detail.
• Order is only what you say it is. The placeholders are numbered, so frames arrive as an ordered list — but nothing tells the model how far apart they are in time. If elapsed time matters, put the timestamps in the text of the turn.
• Check the documented maximum. The card states an upper bound on images per prompt for the configuration it was evaluated at. Exceeding it may work and is outside what was measured.