QLoRA and Fine-Tuning on One Consumer GPU

LoRA removes the optimiser state. QLoRA removes most of the weights as well, by keeping the frozen base in 4 bits and training the adapter in 16. The combination is what put fine-tuning on a single desktop card, and the budget is worth deriving rather than trusting.

The idea in one line

Quantise the frozen base model to 4 bits, keep the LoRA adapters in bf16, and dequantise each weight tile back to bf16 only for the moment it is used in a matmul. The base is never updated, so quantisation error never compounds through training — it is a fixed distortion of a fixed model. Dettmers et al. (2023, QLoRA: Efficient Finetuning of Quantized LLMs, arXiv 2305.14314) introduced it along with three specific mechanisms.

That “never updated” clause is doing more work than it looks. Quantising weights you intend to train is a genuinely bad idea: gradients are small relative to the quantisation step, so updates round away to nothing, and the errors that survive compound over thousands of steps. Quantising weights you will never touch is a different operation entirely — the distortion is applied once, measured once, and the trainable parameters sitting beside it are at full precision and can learn to compensate for it. QLoRA works because those two facts are separable, not because 4-bit training was solved.

NF4 and double quantisation

• 4-bit NormalFloat (NF4). An information-theoretically optimal data type for values that are normally distributed, which trained neural network weights approximately are. Sixteen quantisation levels placed at the quantiles of a normal distribution rather than evenly, so the levels are dense where the weights are dense.
• Double quantisation. Blockwise quantisation stores one fp32 absmax constant per block. At block size 64 that is 32 bits per 64 weights — an extra 0.5 bits per parameter, which is 12.5% on top of a 4-bit budget. Double quantisation quantises those constants too, and the paper reports the saving at about 0.37 bits per parameter.
• Paged optimisers. NVIDIA unified memory paging for optimiser state, so a gradient-checkpointing memory spike evicts to host RAM instead of raising CUDA out of memory. This is what turns an intermittent crash on the longest sequence in the dataset into a slow step.

The 0.37 bits is not a rounding detail at scale. On a 65B model that is 0.37 × 65e9 / 8 / 1e9 ≈ 3.0 GB, which is the difference between fitting on a 48 GB card and not.

Where 24 GB goes

Take a 7B model, sequence length 2048, batch size 1, gradient checkpointing on, LoRA r = 16 on attention and MLP projections. Every term:

A. frozen base in NF4
     6.74e9 x 0.5 bytes                     =  3.37 GB
B. quantisation constants (after double q)
     6.74e9 x 0.127 bits / 8                =  0.11 GB
C. LoRA adapters, bf16   (~20M params)
     2.0e7 x 2 bytes                        =  0.04 GB
D. adapter grads + Adam m,v (fp32)
     2.0e7 x 12 bytes                       =  0.24 GB
E. checkpointed layer boundaries
     batch 1 x seq 2048 x hidden 4096
     x 2 bytes x 32 layers                  =  0.54 GB
F. recompute buffer for one layer
     (attention + MLP intermediates)        =  0.3–0.8 GB
G. logits, bf16
     2048 x 128256 x 2 bytes                =  0.53 GB
H. logits upcast to fp32 for cross-entropy
     2048 x 128256 x 4 bytes                =  1.05 GB
                                              ----------
     subtotal                                  ~6.2–6.7 GB
     + CUDA context, fragmentation, allocator
       caching                                  1–2 GB

Comfortably inside 24 GB — which is the point, and also why people are surprised when it fails. Note which terms scale with what. A, B, C and D are fixed once you pick a model and a rank. E, F, G and H all scale with batch × sequence length, and G and H scale with vocabulary size too.

Term E is worth understanding rather than accepting, because it is what gradient checkpointing buys. Without checkpointing, the backward pass needs every intermediate activation from the forward pass — attention scores, MLP hidden states, layer norms — for all 32 layers at once, which for these shapes is tens of gigabytes. Checkpointing stores only the input to each layer (term E) and recomputes that layer’s internals on demand during the backward pass (term F, one layer at a time rather than 32). You pay roughly one extra forward pass in time — commonly around 30% slower steps — and you get an activation budget that no longer scales with depth. On a 24 GB card this is not an optimisation, it is a precondition.

The tensor that actually OOMs you

Terms G and H are the ones that catch people, because they are the only ones that depend on the vocabulary and vocabularies have grown. A model with a 128k vocabulary at sequence length 4096 and batch size 4:

logits  4 x 4096 x 128256 x 2 bytes (bf16) =  4.2 GB
        + fp32 copy for cross-entropy      =  8.4 GB
                                             --------
                                              12.6 GB

...for a model whose weights are 3.4 GB.

This is why an out-of-memory error often arrives at step 340 rather than step 1: the dataset was not length-sorted and step 340 held the longest example. The diagnostic is that the reported allocation in the error is large and round and the crash correlates with sequence length, not with time.

The fixes, in order of how much they cost you: cap max_seq_length and drop or truncate the tail of the length distribution; reduce the micro-batch and raise gradient accumulation to keep the effective batch size the same; use a fused or chunked cross-entropy that never materialises the full fp32 logits tensor. Length-group the sampler so long sequences batch with long ones instead of padding short ones up to them.

What fits and what does not

Base weights only, NF4, before any of the terms above:

The pattern to take away: 4-bit quantisation solves the weight term and nothing else. Once the weights fit, your remaining ceiling is sequence length times batch size times vocabulary, and no amount of further weight quantisation touches it.

The 34B row is where the paged optimiser earns its place. With 17 GB of weights on a 24 GB card there are about 7 GB left for everything else, and the peak — not the average — is what raises CUDA out of memory. Paging lets the transient spike spill to host memory over PCIe rather than killing the run. It is much slower for the steps where it engages, which is the correct trade: a run that is occasionally slow finishes, and a run that crashes at step 900 does not. If you find paging engaging on most steps rather than a few, that is the signal to reduce sequence length rather than to wait it out.

What the paper reports

Rather than paraphrase: the QLoRA authors report finetuning a 65B parameter model on a single 48 GB GPU — bringing the requirement down from over 780 GB of GPU memory — while preserving full 16-bit fine-tuning task performance. Their Guanaco model family came out of exactly that setup, and the paper is explicit that the 4-bit base plus 16-bit adapters combination is what recovers the quality that naive 4-bit training loses.

Two things the paper is careful about that get dropped in retellings. The quality claim is about the tasks they evaluated, not a universal equivalence. And 4-bit training is slower per step than bf16 training — dequantisation is real work — so QLoRA buys you feasibility on small hardware, not throughput. If you have the memory for plain LoRA in bf16, use it.

One consequence that survives past the training run and catches people at deployment: your adapter was trained against a specific quantisation of a specific base. Loading it against the full-precision base, or against the same base quantised by a different method, gives you a model you did not evaluate. Merging is worse — folding the adapter into 4-bit weights requires dequantising, adding and requantising, and the result drifts from the artefact your evaluation blessed. The safe sequence is to merge into the full-precision base, quantise the merged model with whatever method you serve, and then re-run the evaluation on that exact file. Treat any evaluation performed on a different quantisation as advisory.