AWQ’s “Only Supports Batch Size 1” Assertion Error

A single prompt works. Two prompts in one call raise an assertion, and no batching flag on the server changes it. The restriction is compiled into the kernel your checkpoint was packed for, and the fix is at the checkpoint end.

Where the assertion fires

The traceback ends inside AutoAWQ’s quantized linear layer — the GEMV variant, in awq/modules/linear/gemv.py — on an assert guarding the leading dimension of the input. Reports and the project’s own documentation describe the constraint the same way: the GEMV path only supports batch size 1.

The trigger is a tensor of shape (B, S, D) arriving with B > 1. That happens the first time you pass a list of prompts to generate, the first time an evaluation harness batches examples, or the first time a serving layer decides two queued requests can share a forward pass. It is why the error so often appears on the day something is moved from a notebook into a server.

GEMV and GEMM are different kernels

AWQ ships more than one kernel for the same 4-bit weights, and they are not interchangeable at runtime because the weights are physically packed differently for each.

• GEMM — a general matrix-matrix multiply. It handles an arbitrary number of rows, which means arbitrary batch and arbitrary prompt length. AutoAWQ’s documentation describes it as substantially faster than fp16 at small batch sizes and as the variant that behaves well with long contexts.
• GEMV — a matrix-vector multiply. One row of activations, by construction. Because it never has to tile across rows it can be specialised harder, and the project documents it as roughly 20% faster than GEMM — but only at batch size 1, and it is described as a poor fit for large contexts.

That is the whole mechanism. GEMV is the decode-step kernel taken to its logical conclusion: single-token generation is genuinely a matrix-vector product, so a kernel that assumes it can be faster. The assertion is not a missing feature, it is the assumption being checked.

It is worth being precise about where the advantage comes from, because it explains why the restriction is not going to be lifted. Decoding one token at a time is bound by memory bandwidth: the dominant cost is reading the weights, and the arithmetic per weight read is tiny. A kernel that knows there is exactly one activation vector can keep that vector in registers for the whole pass, skip all the tiling machinery a general matmul needs, and spend its entire budget on streaming weights. Add a second row and the tiling has to come back, at which point you have written GEMM. The two kernels are not two settings of one implementation; they are two implementations, and the weights are packed for one of them at quantization time. This is the same shape of constraint discussed in batching generally — batch size one never saturates a modern GPU, so a kernel that gives up batching has to earn it back somewhere.

Checking which one your checkpoint has

The packing choice is recorded in the checkpoint, not chosen at load time. Read it from config.json before changing anything:

python - <<'PY'
import json
cfg = json.load(open("config.json"))
print(cfg.get("quantization_config"))
PY

# e.g. {'quant_method': 'awq', 'bits': 4, 'group_size': 128,
#       'version': 'gemv', 'zero_point': True}

There is a second reason to look here before doing anything else. The same field is what a serving engine reads to decide which layer class to instantiate, so it is not merely documentation — it is the input to the decision that produced your assertion. A model card that says nothing about packing and a config.json that says gemv are not in conflict; the card is describing the quantization method and the config is describing the kernel layout, and only the second one has an opinion about batching.

The version field is the answer. gemv or gemv_fast means you have the batch-size-one packing; gemm means you do not, and a batch assertion in that case is coming from somewhere else. Some repositories publish both packings as separate branches or separate repositories, and the difference is rarely visible in the model name.

What actually changes it

1. Use a GEMM-packed copy of the same model. Cheapest by a wide margin if one exists. Check the model card and the other branches of the repository before doing anything else.
2. Re-quantize with the GEMM version. If you own the quantization step, pass version as GEMM in the quant config. This re-runs quantization against a calibration set, so it is a GPU job, not a repack.
3. Serve it through vLLM instead. vLLM implements its own AWQ layers rather than calling AutoAWQ’s, and its AWQ path is built for continuous batching. This is the usual answer for anything that has to serve concurrent traffic.
4. Keep batch size one deliberately. Legitimate for a single-user local assistant, where GEMV’s speed advantage is real and there is no second request to batch with. Set the harness to one prompt per call rather than letting it discover the assertion.

What does not work: server-side batching flags, padding the batch, reshaping the input to (1, B*S, D), or disabling continuous batching. The first two do not reach the kernel’s assumption and the third silently changes the attention semantics.

The wider situation with AutoAWQ

AutoAWQ has been retired upstream. Its repository states that the project is deprecated and no longer maintained, and points at the vLLM project’s llm-compressor for the recommended quantization workflow. That matters for this error in two ways: a GEMM re-quantize through AutoAWQ is running an unmaintained code path, and the AWQ checkpoints you can still load through vLLM will outlive the library that produced them.

If you are choosing a quantization format now rather than debugging one you inherited, the relevant question is not which kernel is fastest at batch one but which format your serving engine supports without a special case — choosing a quantization format covers that trade properly.