LoRA Explained: Fine-Tuning Without the GPU Bill

LoRA is one idea: freeze the weights, and learn a low-rank correction to them instead. The idea is a paragraph. The reason it changed who can fine-tune is arithmetic, and the arithmetic is worth doing on paper once.

What full fine-tuning costs

Training a weight in mixed precision with Adam does not cost one number per parameter. It costs about eight, and standard accounting for a bf16 forward pass with an fp32 master copy comes to roughly 16 bytes per trainable parameter:

per trainable parameter, mixed-precision Adam
  bf16 weight ................  2 bytes
  bf16 gradient ..............  2 bytes
  fp32 master weight .........  4 bytes
  fp32 Adam m (momentum) .....  4 bytes
  fp32 Adam v (variance) .....  4 bytes
                               --------
                               16 bytes

7B model, all parameters trainable
  6.74e9 x 16 bytes = 107.8 GB      (before activations)

That is why full fine-tuning a 7B model does not fit on a 24 GB card, or on an 80 GB one, and needs sharding across several. The weights themselves are the small part; the optimiser state is three quarters of it. LoRA attacks that number rather than the weights.

The arithmetic, on actual matrices

Take one attention projection from a 7B-class model — a query projection, shape 4096 × 4096. Full fine-tuning learns an update ΔW with the same shape as W:

full update
  ΔW : 4096 x 4096  =  16,777,216 trainable parameters

LoRA update, rank r = 8
  A  : 8 x 4096     =      32,768
  B  : 4096 x 8     =      32,768
                       -----------
  ΔW = B·A          =      65,536 trainable parameters

ratio  16,777,216 / 65,536 = 256x fewer
share  65,536 / 16,777,216 = 0.39% of the full update

The product B·A is still a 4096 × 4096 matrix — it has to be, it is added to W — but it can only ever have rank 8. The bet LoRA makes is that the update a fine-tune needs is intrinsically low-rank even though the weight it corrects is not. Hu et al. (2021, LoRA: Low-Rank Adaptation of Large Language Models, arXiv 2106.09685) is the paper that made that bet and measured it holding.

The general form: a d × k weight costs d·k parameters to update fully and r·(d + k) under LoRA. For a square d × d matrix that is d² against 2·d·r, so the break-even rank is d/2 — at d = 4096, rank 2048. Every rank below that is a saving, and the ranks in practice are 8 to 64.

Two details from the paper that matter in practice. A is initialised from a random Gaussian and B is initialised to zero, so B·A = 0 at step 0 and the adapted model starts exactly equal to the base — training begins from the base model’s behaviour, not from noise. And the update is scaled by α/r, which is why raising the rank without raising alpha changes the effective learning rate as well as the capacity.

Scaling it to a whole model

One matrix is not the interesting number. Apply the same rank to the query and value projections of every layer of a 32-layer, 4096-hidden model:

r = 8, targets = {q_proj, v_proj}, 32 layers, d = 4096

  per matrix   2 x 8 x 4096          =      65,536
  per layer    2 matrices            =     131,072
  whole model  x 32 layers           =   4,194,304   (~4.2M)

  as a share of 6.74e9 total         =       0.062%

  optimiser + gradient memory
    4.19e6 x 16 bytes                =        67 MB
  versus full fine-tuning
    6.74e9 x 16 bytes                =    107.8 GB

  adapter file, bf16
    4.19e6 x 2 bytes                 =       8.4 MB

Sixty-seven megabytes of trainable state instead of a hundred gigabytes. The frozen base weights still have to be resident — 13.5 GB in bf16 for a 7B model — but they need no gradients, no momentum and no fp32 master copy, so they are a fixed cost rather than a multiplied one. The LoRA paper reports a roughly 3× reduction in GPU memory for GPT-3 175B and a 10,000× reduction in trainable parameters.

A caveat on the shapes: on models using grouped-query attention, the key and value projections are not square. A model with 32 query heads and 8 KV heads has k_proj and v_proj at 1024 × 4096, so the same rank buys a different share there. Redo the arithmetic against the config of the model you are actually adapting.

Choosing the rank

Rank is capacity. The commonly reported pattern — and this is folklore confirmed repeatedly rather than a theorem — is that style, format and tone tasks saturate at low rank, while tasks that ask the model to do something structurally new benefit from more.

• r = 4–8. Format compliance, tone, a fixed schema, a label vocabulary. Adapters of a few megabytes.
• r = 16–32. The usual default for instruction tuning on a few thousand examples. Common practice pairs it with α = 2r.
• r = 64–128. Larger behavioural shifts, or where you are targeting all linear layers rather than just attention. At this point you are approaching the memory profile you were avoiding.

Which modules you target matters at least as much as the rank. The original paper adapted attention projections only; later practice frequently targets every linear layer including the MLP blocks, which multiplies the trainable count by roughly three to four and is often worth it. Increasing coverage tends to help more per parameter than increasing rank on a narrow target set.

Two published variants are worth knowing about before you spend a week sweeping rank. rsLoRA (Kalajdzievski, 2023, arXiv 2312.03732) argues the conventional α/r scaling is what makes high ranks underperform — the factor shrinks as rank grows, damping the update exactly when you added capacity — and proposes scaling by α/√r instead, which is why a “rank-stabilised” flag exists in most training libraries. DoRA (Liu et al., 2024, arXiv 2402.09353) decomposes the weight into a magnitude and a direction and applies the low-rank update only to the direction, reporting improvements over LoRA at equal rank. Neither changes the arithmetic above; both change how much you get per trainable parameter.

Merging, and why inference is free

The property that made LoRA a serving story and not just a training story: because the adaptation is additive, you can fold it back in.

serving unmerged   y = W·x + (α/r)·B·(A·x)     two extra matmuls
serving merged     W' = W + (α/r)·B·A          done once, offline
                   y = W'·x                    identical to the base

A merged model is bit-for-bit an ordinary model of the same architecture. No extra latency, no special runtime, no adapter plumbing. Adapter-style serving trades a small per-token overhead for the ability to host many adapters against one copy of the base, which is a different and often better deal.

The named trap here: merging into a quantised base is not lossless. If you trained against a 4-bit base, merging requires dequantising, adding, and requantising, and the result is not the model you evaluated. Merge into the full-precision base and quantise afterwards, then re-run your evaluation on the artefact you are actually going to serve.

Where it does not help

• It does not shrink the base. You still need the whole model resident to train against it. LoRA removes optimiser state, not weights — that is what quantisation is for, and combining the two is QLoRA.
• It does not make a small model into a big one. A rank-16 correction cannot add capability the base does not have. It redirects existing capability.
• It is a weak instrument for large distribution shifts. Adapting to a new language or a genuinely new domain is a pretraining-scale problem; a low-rank correction on instruction data is not the tool.
• Adapters are pinned to a base. An adapter trained against one checkpoint is meaningless against another, including a point release of the same family. This is the maintenance cost that outlives the training cost.