Quantisation Trade-off Planner

need(bits) = (P × bits/8 + KV) × (1 + overhead) ≤ VRAM

The table above is the easy half of the decision and it is completely determined: weights scale linearly with bits per weight, the KV cache does not move at all when you requantise the weights, and the sum either fits or it does not. That is worth seeing laid out, because it shows how quickly the KV term stops being a rounding error. At long context or any real batch size, dropping from 8 bits to 4 buys much less than half, since half of what you are carrying was never weights.

The hard half is what the compression costs you, and no calculator can answer it. Degradation is not uniform: it varies by model family, by how much redundancy the weights had to begin with, and above all by task. Quantisation that leaves conversational fluency untouched can quietly damage arithmetic, long-context retrieval, structured-output adherence or a low-resource language, and the loss shows up in exactly the places a casual test will not look.

So use the fit table to shortlist, then run a real comparison: fifty examples from your own workload, the same prompts at two adjacent levels, graded the way you actually care about. If two levels both fit and you cannot tell them apart on your own eval, take the smaller one and spend the memory on context or batch instead. If you can tell them apart, you have your answer and it cost you an afternoon.