Applying a LoRA at Inference Time Without Merging It

A merged model and a runtime-applied adapter compute the same thing. The difference is whether BA was added into W once, offline, or is added to the activations on every forward pass — and the second costs far less than people assume.

Two ways to apply the same matrices

LoRA, as described in Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models” (2021), replaces an update to a weight matrix with the product of two thin ones. For a projection W of shape d_out × d_in, the adapter stores A of shape r × d_in and B of shape d_out × r, with r typically 8 to 64, and the adapted weight is W + (alpha/r) · BA.

Merging computes that sum once and writes a new checkpoint. Runtime application keeps the three matrices separate and evaluates Wx + B(Ax) instead. The parenthesisation is the whole trick: form Ax first, giving a vector of length r, then expand it with B. Nobody ever materialises BA, which would be a full-size matrix and would defeat the point.

Deriving the extra arithmetic

Count multiply-accumulates per token, for one adapted projection. Assume batch size one and ignore bias terms; both simplifications affect the base and the adapter equally, so the ratio survives them.

base projection:   d_out * d_in            MACs
adapter path:      r * d_in  (for Ax)
                 + d_out * r  (for B(Ax))
                 = r * (d_in + d_out)

ratio = r * (d_in + d_out) / (d_out * d_in)

worked, for a square 4096 x 4096 projection at r = 16:
  base    = 4096 * 4096      = 16,777,216
  adapter = 16 * (4096+4096) =    131,072
  ratio   = 131,072 / 16,777,216 = 0.0078  ->  0.78%

Under 1% of the arithmetic of the layer it adapts. The ratio scales linearly in r and inversely in the layer dimension, so a rank-64 adapter on the same projection is about 3.1%, and the same rank-16 adapter on a 2048-wide model is about 1.6%. It never approaches the cost of the base, because the base is quadratic in the dimension and the adapter is linear.

Deriving the extra memory

Same shapes, now counting parameters. Take the published configuration of Llama 3 8B — hidden size 4096, 32 layers, 32 attention heads and 8 key/value heads, giving a head dimension of 128 and therefore a v_proj of 4096 in and 1024 out. Adapt q_proj and v_proj, the common default, at rank 16:

q_proj  A: 16 x 4096 = 65,536   B: 4096 x 16 = 65,536   -> 131,072
v_proj  A: 16 x 4096 = 65,536   B: 1024 x 16 = 16,384   ->  81,920
per layer                                                  212,992
x 32 layers                                              6,815,744 params

at f16 (2 bytes):  6,815,744 * 2 = 13,631,488 bytes  ~= 13.0 MiB

Thirteen megabytes of resident weights against a base of several gigabytes. The activation overhead is smaller still: the intermediate Ax is r values per adapted projection per token, so 16 floats where the layer is already carrying 4096. Neither number is large enough to change a hardware decision, which is the useful conclusion — adapter overhead is not why you are short of VRAM.

Why the cost is smaller than the FLOP count

The 0.78% figure is an upper bound on the latency effect, and in practice single-stream decoding does not pay even that. Generating one token at a time is memory-bandwidth bound, not arithmetic bound: the hardware spends its time streaming weights out of memory and has spare arithmetic capacity sitting idle. The adapter adds 13 MiB to the bytes that must be read and a rounding error to the arithmetic, so the extra work partly hides in time the machine was already spending waiting.

This is the same asymmetry that makes KV caching worth so much, and it reverses under batching: with a large batch the arithmetic units saturate and the adapter’s extra matmuls start to cost roughly what the FLOP ratio says. Anyone serving many concurrent requests through one adapter should treat the ratio as real; anyone running a single local stream should expect it to disappear into noise. Neither claim is a measurement — measure your own configuration before spending anything on the difference.

What separation buys you

The cost side of this trade is small enough to be uninteresting. The benefit side is where the decision actually lives, and it comes down to three things a merged file cannot do.

The first is that the scale becomes a runtime parameter. Because the adapter contributes (alpha/r) · B(Ax) and that coefficient is read at load rather than baked into W, multiplying it by 0.5 or 2.0 costs nothing — llama.cpp exposes it as --lora-scaled and its server exposes it per request. A merged model has one strength, chosen at merge time by somebody who could not see your prompts. Being able to dial an over-eager fine-tune down to half without retraining is worth more in practice than most people expect, and it is a knob that simply does not exist on the merged side.

The second is composition. Two adapters over one base sum: the forward pass becomes Wx + B1(A1x) + B2(A2x), and each keeps its own scale. Whether the combination is any good is an empirical question — adapters trained independently can interfere, and there is no guarantee that a tone adapter and a domain adapter compose the way you want — but the arithmetic permits it and merging forecloses it. Once B1A1 has been folded into W, adding a second adapter means adding it to an already-modified base, which is not the same operation.

The third is that one copy of the base serves many behaviours. Weights dominate memory; a 13 MiB adapter does not. A server holding one quantized base resident and switching between a dozen adapters uses roughly the memory of one model, where a dozen merged models would need a dozen copies. This is the entire premise of multi-adapter serving, and it only works because the merge never happened.

When merging is still right

• The runtime cannot take adapters. Plenty of deployment paths — a converted ONNX graph, an embedded runtime, a hardware-vendor toolchain — accept only a single set of weights. Merge, and treat the result as a new model.
• You are shipping the model to somebody else. A merged checkpoint has no pairing to get wrong. An adapter plus a base is two artefacts and one implicit contract, and that contract is not enforced anywhere.
• You want to quantize after adapting. Quantizing a merged fp16 model lets the quantizer see the adapted weights. Quantizing the base and adding an f16 adapter on top gives a different, usually slightly better result for the adapted part, but you cannot then hand somebody one file.

What you give up by merging is everything the separation bought: adjusting the scale without a rewrite, stacking two adapters, and keeping one base resident for several behaviours. You also give up the ability to recover the adapter later — extracting one from a merged model requires the original base and returns an approximation.