Which Quant to Download in LM Studio's Model Picker

The picker shows you six or eight files with names like Q4_K_M and IQ4_XS, a size in gigabytes for each, and a badge saying whether it fits. The size is the honest number and the name is the confusing one, so the useful move is to work backwards from the size to what the name actually costs you.

What the letters mean

Every entry in that list is the same model with its weights stored at different precision. The naming is llama.cpp’s, and it decomposes cleanly once you know the parts.

• The number is the nominal bits per weight. Q4 is roughly four bits per stored weight, Q8 roughly eight. It is never exactly that, for reasons the next section makes arithmetic.
• _K means a k-quant. These store weights in super-blocks with their own scales held at higher precision than the weights themselves, which is why a Q4_K is both larger and better than a flat Q4_0.
• _S, _M, _L are the mix. Small, medium and large refer to how many tensors are kept at higher precision than the headline number. Q4_K_M keeps more of the attention tensors above four bits than Q4_K_S does, and is correspondingly larger.
• I means an importance-matrix quant. IQ4_XS and its relatives use a calibration pass to decide which weights can be crushed hardest. They get more quality per byte at the same nominal bit count, at the cost of being newer and slower to compute on some backends.
• MLX entries are a different format entirely. On Apple Silicon the picker also offers MLX builds, labelled by bit depth (4bit, 8bit) rather than by quant name. They are not GGUF, and the GPU offload control does not apply to them, because unified memory means there is nothing to offload.

Deriving bits per weight from the file size

You do not have to trust the label. The file size and the parameter count give you the real figure, and the derivation below is one you can repeat for any model in the picker.

Start with a quant whose layout is known exactly. A Q8_0 block stores 32 weights as one byte each plus a 16-bit scale for the block: 34 bytes for 32 weights, which is 34 x 8 / 32 = 8.5 bits per weight. Now take a published file size. The lmstudio-community GGUF repository for Meta Llama 3.1 8B Instruct lists its Q8_0 at 8.54 GB. Assuming GB means 10⁹ bytes, as Hugging Face displays it:

parameters = bytes x 8 / bits_per_weight
           = 8.54e9 x 8 / 8.5
           = 8.04e9

8.04 billion, for a model called 8B. The method checks out, so run it the other way on the rest of the list, dividing each published size by that parameter count:

Q3_K_L   4.32 GB  ->  4.32e9 x 8 / 8.04e9  =  4.30 bits/weight
IQ4_XS   4.45 GB  ->                            =  4.43
Q4_K_M   4.92 GB  ->                            =  4.90
Q5_K_M   5.73 GB  ->                            =  5.70
Q6_K     6.60 GB  ->                            =  6.57
Q8_0     8.54 GB  ->                            =  8.50  (as derived)

Assumptions, stated: the file sizes are the ones published on that model card at the time of writing, GB is 10⁹ bytes, and the parameter count is derived rather than taken from Meta — the official meta-llama repository is gated behind a licence acceptance, so its configuration is not readable without that, and this derivation deliberately does not need it.

What the table shows is that the labels understate. Q4_K_M is not four bits per weight, it is 4.9, so the file is about 22% larger than the name suggests. That gap is the difference between fitting on an 8 GB card and not, and sizing from the label rather than from the number in front of you is the common mistake here.

What the fit badge does not include

Next to each entry LM Studio shows an estimate of whether that file can run with everything on the GPU on this machine. It is a genuinely useful signal — it reads your actual hardware rather than making you look it up — and it is an estimate about the weights.

Things it does not know about: the context length you are going to ask for, whether you will run more than one model at once, and what else on the machine is holding video memory right now. A file the badge calls comfortable can fail to load later the same day because a browser is open, and the same file marked green will still spill to the CPU if you raise the context far enough.

Treat it as a filter rather than a decision. It correctly rules out the entries that cannot work; among the ones it allows, the choice is still yours, and the two sections below are how to make it.

The part the file size never covers

Weights are fixed; the cache is not, and it is what turns a comfortable-looking choice into a tight one. For a model with 32 layers, 8 key/value heads and a head dimension of 128 — the Llama 3.1 8B shape, whose configuration is published on Hugging Face — a 16-bit KV cache costs 2 x 32 x 8 x 128 x 2 = 131,072 bytes per token, or 128 KiB. So:

 4,096 tokens  ->  0.5 GiB of cache
16,384 tokens  ->  2.0 GiB
32,768 tokens  ->  4.0 GiB

Put that beside the quant table. On a 12 GB card, Q4_K_M at 4.92 GB leaves plenty of room for a 32k context; Q8_0 at 8.54 GB leaves about three gigabytes, which is a 16k context and nothing else on the card. The upgrade from a 4-bit to an 8-bit quant is frequently paid for in context rather than in money, and that is usually the wrong trade for a task that has to read a long document.

If the cache is what is squeezing you, the lever is its precision rather than the model’s: an 8-bit KV cache halves every number above. The mechanism is in the KV cache, and the general quality question in choosing a quantization.

Choosing

The rule that follows from the arithmetic is short. Take the largest quant whose file size, plus the cache for the context you actually need, fits in your VRAM with a gigabyte or so of headroom — and if nothing does, take a smaller model rather than a smaller quant.

That last clause is the one worth arguing for. Below roughly four bits per weight, quality degrades noticeably and unevenly, because the damage concentrates in outlier features that a small number of weights carry disproportionate responsibility for; averaging error across a block hurts most exactly where the distribution has a long tail. A large model crushed to two bits is generally a worse answer than a smaller model at four, and it is also slower, since it is still moving more bytes per token. The case where the trade goes the other way is when the smaller model simply lacks the capability at all.

One thing this page cannot tell you is how much quality any of these costs on your task, because that has not been measured here and published perplexity figures are a proxy rather than an answer. The measurement that decides it is yours: download two adjacent quants, run your actual prompts through both, and keep the smaller one if you cannot tell them apart. Once you have chosen, how many layers reach the card is the next dial — see the GPU offload slider.