Quantizing a Vision-Language Model: What's Different

A vision-language model is not one model with extra inputs. It is a vision encoder, a projector and a language model welded together, each with a different parameter budget and a different tolerance for rounding — and quantizing it as though it were homogeneous is the mistake that produces a model which talks fluently about images it is not really seeing.

Three parts, one checkpoint

• The vision tower. Usually a ViT-family encoder that turns an image into a grid of patch embeddings. Typically a few hundred million parameters, and often a fraction of a percent to a few percent of the total checkpoint.
• The projector. A small module — sometimes a single linear layer, sometimes a two-layer MLP, sometimes a resampler with learned queries — that maps patch embeddings into the language model’s token embedding space. Often only a few million parameters, and the narrowest bottleneck in the architecture: every bit of visual information reaches the language model through it.
• The language backbone. The transformer that consumes projected image tokens alongside text tokens. Almost always the overwhelming majority of the parameters.

The asymmetry in those three sizes is what determines the whole strategy, and it is worth computing rather than asserting.

Why the tower is skipped: the arithmetic

Take a VLM with a 7B language backbone, a 400M vision tower and a 20M projector — a plausible shape for this class of model, though the exact split varies. Quantize to 4 bits and compare the two policies:

FP16 sizes
  backbone   7.00e9 * 2 = 14.00 GB
  tower      4.00e8 * 2 =  0.80 GB
  projector  2.00e7 * 2 =  0.04 GB
  total                 = 14.84 GB

policy A: quantize everything to 4 bits
  7.42e9 * 4 / 8        =  3.71 GB

policy B: quantize the backbone only
  backbone  7.00e9 * 4 / 8 = 3.50 GB
  tower + projector fp16   = 0.84 GB
  total                    = 4.34 GB

cost of policy B          = 0.63 GB  (+17% over A,
                                      still -71% vs FP16)

Six hundred megabytes to leave the entire visual pathway at full precision. Against the 11.1 GB the backbone quantization saved, that is a rounding error — and it removes every question about whether the patch embeddings are faithful. This is why the standard recipes for multimodal quantization exclude the vision components rather than debating bit widths for them, and it is the clearest instance of the general principle in mixed-precision quantization: promote what is cheap to promote and structurally sensitive.

The ratio is what makes the argument, so check it for your model before assuming. A VLM with a 2B backbone and a 1B vision tower is a different calculation, and there the tower is a real share of the memory rather than a rounding error.

Calibration has to include images

This is the part most easily got wrong, because it fails silently. The language backbone is being quantized, and calibration decides which of its rounding errors are affordable — so the question is what activations flow through it during the calibration pass.

Feed it text-only calibration data and the image tokens never appear. Projected visual embeddings occupy a region of the embedding space that text tokens do not: they are produced by a different module, they are not drawn from the vocabulary, and their per-channel statistics differ from text embeddings’. The Hessian that GPTQ builds, or the per-channel magnitudes AWQ collects, therefore contain no information about the channels that carry visual content, and the algorithm concludes — correctly, given its inputs — that those channels are cheap to damage.

The result is a model whose text quality is unchanged and whose grounding in the image has quietly degraded. It still describes pictures; it describes them less accurately, and no text benchmark will show it. The general mechanism is in what a calibration dataset actually does; this is its sharpest practical case.

Practically, the calibration set must be image-text pairs processed through the model’s own processor — the same resize, the same patching, the same chat template with image placeholder tokens in their real positions — because all of that determines how many image tokens there are and where they sit.

The ignore list in practice

Every quantization library expresses this as an exclusion list. The names differ — ignore, modules_to_not_convert, skip_modules — and they take module name patterns. Transformers’ own documentation describes modules_to_not_convert as being for exactly this: modules that must be left in their original precision, naming a Llava encoder as an example alongside a Whisper encoder and Mixtral gate layers.

The vLLM project’s llm-compressor documents the multimodal pattern as a regex ignore list of the form:

ignore = [
    "re:.*lm_head",
    "re:.*vision_tower.*",
    "re:.*multi_modal_projector.*",
]

recipe = GPTQModifier(
    targets="Linear",
    scheme="W4A16",
    sequential_targets=["MistralDecoderLayer"],
    ignore=ignore,
)

Three things in that snippet are the general lesson rather than the specific strings. The head is excluded alongside the vision parts, for the reasons in mixed-precision quantization. The patterns are regexes over module paths, so they must match your model’s naming. And sequential_targets names the decoder layer class, which controls how granular the sequential quantization is and therefore how much memory the run needs.

What breaks, and how it looks

The failures here are distinctive because they are asymmetric — the language half is fine and the visual half is not — and knowing the signature saves a lot of guessing.

• Plausible but ungrounded descriptions. The model produces fluent, well-formed answers about the image that are correct in genre and wrong in detail — the right kind of object, the wrong count; a chart described in the right shape with the wrong numbers. This is the signature of a backbone calibrated without images.
• OCR and fine detail degrade first. Reading small text out of an image depends on precision surviving the whole pathway and is the task most sensitive to any of it being rounded. If OCR quality falls while general description holds up, check the projector and tower were actually excluded.
• A silently unmatched ignore pattern. A regex that matches nothing does not error — it just quantizes the tower. Verify by inspecting the produced checkpoint for which modules carry quantization metadata, not by trusting the config you passed in.
• Gated weights. Many strong VLM checkpoints require accepting a licence on the model page before download. That gate is the licence being enforced and it is the route to use; a redistributed copy that skips it may also have been converted by someone whose calibration choices you cannot inspect.