Quantisation for Edge Devices: What int8 and int4 Do to a Weight

Quantisation replaces each weight with the nearest value on a coarse grid and stores the grid index instead of the number. That is the whole mechanism. Everything difficult about it follows from one fact: the grid is shared across many weights, so a single outlier decides how coarse it is for all of them.

What quantisation does to one weight

Integer quantisation is an affine map from a range of real values onto a set of integers, plus the inverse map used at compute time:

quantise:    q = clamp( round(w / s) + z,  q_min,  q_max )
dequantise:  w' = s × (q − z)

s  scale       — the width of one grid step
z  zero point  — the integer that represents exactly 0.0
q  the stored integer, in [q_min, q_max]

For a symmetric int8 scheme, z is 0 and the integers run from −127 to 127, so the scale is set by the largest magnitude in the group:

s = max(|w|) / 127

Worked: a group of weights whose largest magnitude is 0.42

  s = 0.42 / 127 = 0.003307

  w = 0.1234  →  q = round(37.32) = 37  →  w' = 0.12236
  error = 0.00104,  bounded by s/2 = 0.00165

The error on any single weight is at most half a step. What matters is how big a step is, and that is where bit width enters:

Per-tensor, per-channel, per-group

If one scale covers an entire weight matrix, one unusually large weight stretches the range and every other weight in the matrix is represented on a needlessly coarse grid. The fix is to share scales across fewer weights:

• Per-tensor. One scale for the whole matrix. Cheapest metadata, worst accuracy, and the reason early quantisation results were so poor.
• Per-channel. One scale per output channel or row. Now a single wild channel damages only itself. This is the default for int8 and it is close to free.
• Per-group. One scale per contiguous block of 32, 64 or 128 weights. This is what makes 4-bit viable at all: the group is small enough that its dynamic range is narrow, so 15 levels are enough to cover it.

Granularity costs storage, which is the trade-off nobody states plainly. A 4-bit scheme with a group size of 32 and an fp16 scale per group actually stores about 4.6 bits per weight — the metadata is nearly fifteen per cent on top of the packed weights. Halving the group size to 16 improves accuracy and pushes you towards 5.2 bits, at which point a well-tuned 5-bit scheme may simply be better. Always compute the real bytes-per-weight from the scheme rather than the nominal bit width; the arithmetic is worked through in the phone memory budget.

Weight-only versus full integer

These are routinely conflated and they buy completely different things.

For a language model generating tokens one at a time, weight-only quantisation captures most of the win, because the bottleneck is moving weights from memory rather than the arithmetic itself. For a vision model running on an NPU, weight-only is close to useless — the accelerator cannot execute it, and you will get a silent fallback to CPU of exactly the kind the Android capability probe exists to catch.

Full-integer quantisation needs a calibration set: a few hundred representative inputs, run through the model to observe the range each activation actually takes, so activation scales can be fixed ahead of time. Calibration data that does not match production data produces a model that clips activations it never saw, and the symptom is a model that is fine on your test set and poor in the field.

Why the cost is not spread evenly

This is the section the vendor documentation does not write, and it is the reason a single published “int4 costs 1.2 points” figure tells you nothing about your model.

Outliers concentrate in particular places

In transformer models, a small number of channels carry activations orders of magnitude larger than the rest, and they are not randomly distributed — they recur in the same positions across inputs. A per-tensor activation scale set by those channels leaves everything else on a grid with a handful of usable levels. This is why the successful low-bit methods all do something targeted: keep the outlier channels in higher precision, or apply a per-channel rescaling that moves the difficulty from activations into weights where per-channel scales can absorb it.

Error accumulates through depth

Each layer’s quantisation error becomes part of the next layer’s input. In a residual architecture, errors added into the stream persist through every subsequent block. The practical consequence is that early layers matter more than their parameter count suggests, and it is why quantisation recipes commonly leave the embedding table and the output projection at higher precision even though those are often the largest single tensors.

Some capabilities are near a decision boundary and some are not

Aggregate metrics hide the failure mode that actually bites. A quantised model typically loses very little on tasks where the correct answer is robustly the most likely one, and loses a great deal on tasks where it was already marginal — long arithmetic chains, exact formatting, rarely-seen entity names, the last steps of a multi-step instruction. Averaged over a broad benchmark this looks like a small decline. In a product where one of those marginal capabilities is the feature, it looks like the feature stopped working.

Which is why the only number that means anything is the one produced by your own evaluation on your own task, and why the last section of this page is a protocol rather than a table.

Hardware with no float changes the decision

On a server GPU the question is which quantisation to choose. On many edge accelerators, and on essentially all microcontrollers, there is no question: the unit executes integer arithmetic and nothing else. That inverts the reasoning in three ways.

• Quantisation stops being an optimisation. An unquantised model does not run slower on the accelerator; it does not run on the accelerator. The comparison is not int8 against fp16, it is int8 on the NPU against fp32 on the CPU, and the gap is far larger than any accuracy delta.
• Mixed precision may not be available. Recipes that keep sensitive layers in float assume a device that can execute float layers cheaply. Where it cannot, keeping one layer in float means partitioning the graph and paying a handoff each way — which can cost more than the accuracy it saved.
• The supported operator set is narrower. Integer-only hardware supports the operators its vendor implemented in integer form. Anything else is a partition boundary. Design the model within that set rather than quantising a model that was never going to fit it.

The evaluation protocol

Run this before and after quantisation, on the same inputs, and treat the comparison as the release gate.

1. Assemble a task set, not a benchmark. Two to five hundred examples drawn from what your feature actually receives, with the correct output for each. This is the expensive part and it is the whole exercise; a borrowed benchmark measures somebody else’s task.
2. Include the marginal cases deliberately. Deliberately over-sample the inputs your unquantised model gets right but not confidently. That is where the loss will be, and a uniformly sampled set will average it away.
3. Score decisions, not distances. Exact match, label agreement, schema validity — whatever your product depends on. Cosine similarity between logits is not a product metric.
4. Report per-category, not in aggregate. Break the score down by input type. A three-point average drop that is entirely concentrated in one category is a different decision from a three-point drop spread evenly.
5. Measure the win in the same run. Bytes resident, median latency and sustained latency, on the target device. A quality cost is only assessable against the resource saving that bought it.
6. Re-run whenever anything changes. A new runtime version, a new delegate, a new calibration set. Quantised numerics are a property of the whole pipeline, not of the checkpoint.

If the drop is unacceptable, the ladder is: increase granularity (smaller groups), then keep the most sensitive tensors at higher precision, then move to quantisation-aware training, which costs a fine-tuning run and usually recovers most of the gap because the model learns weights that survive the grid. The same ordering applies to server-side deployments, described in quantisation for inference; the difference on a device is only that you have less room to give up.