Merging a LoRA or Keeping It Separate: Choosing for Local Deployment

A trained LoRA can be folded into the base weights to produce one self-contained model, or kept as a separate file the runtime applies on load. The choice looks like a packaging detail and is actually a decision about disk, about how many variants you can serve, and about where quantization loss lands. What a LoRA is is assumed here.

How small an adapter actually is

The asymmetry that drives everything else is worth deriving rather than asserting. A LoRA replaces an update to a weight matrix of shape d_out x d_in with the product of two matrices of shape d_out x r and r x d_in, so it stores r x (d_in + d_out) parameters instead of d_in x d_out.

Take one query projection in Llama 3.1 8B: 4096 x 4096, which is 16,777,216 parameters. At rank 16 the adapter for that tensor is 16 x (4096 + 4096) = 131,072 parameters — 0.78% of what it modifies. Applied across every linear projection in all 32 layers, with the wider gate, up and down projections included, a rank-16 adapter over an 8B model lands in the tens of megabytes at fp16, against several gigabytes for the base.

That ratio is the entire argument for keeping adapters separate, and it is also the reason merging is tempting: something two orders of magnitude smaller than the thing it modifies is easy to lose track of, and a merged file cannot be run with the wrong adapter or with none.

Disk: linear in variants, or not

Merging produces a full model file per variant. Three fine-tunes of an 8B at Q4_K_M, at roughly 4.58 GiB each, is about 13.7 GiB and it grows linearly with every new variant. Keeping them separate is one base file plus three small adapters — roughly 4.58 GiB plus a few hundred megabytes total.

The difference is not just disk. It is also what you re-download when the base model is updated, what you re-verify when you check integrity, and what a backup costs. A workflow that produces a fine-tune a week is a workflow where merging turns into a storage problem within a couple of months, and local model disk budgets fill faster than anybody expects.

Against that: a merged model is one artifact with one checksum. There is no way to run it against a mismatched base, no scaling parameter to get wrong, and no second file to forget when you copy it to another machine. For anything shipped to a machine you do not control, that matters more than the gigabytes.

Memory and the runtime cost of not merging

At inference time a merged model is indistinguishable from any other model: the weights are the weights and nothing extra is resident.

An adapter applied at runtime costs more than its file size suggests, and the reason is worth understanding. llama.cpp normally maps the GGUF file so that weight pages are shared with the page cache and never copied. Applying an adapter means modifying weights, which means those pages can no longer be a read-only mapping of the file — loading a LoRA disables mmap. The practical effect is that the model is read into anonymous memory rather than mapped, which raises host memory use and makes the load slower, in exchange for the flexibility.

The flexibility is real and specific. llama-server can hold several adapters at once and expose their scales, so a single running process with one copy of the base weights in VRAM can serve several fine-tunes by varying which adapter is active at what strength. That is the deployment merging cannot do at all: with merged files, three variants means three model loads and three times the VRAM, and on a single consumer card that usually means it is not possible. Serving multiple LoRAs covers that pattern in general.

# Runtime application, llama.cpp
llama-server -m base-Q4_K_M.gguf \
  --lora ./adapters/support-tone.gguf \
  --lora-scaled ./adapters/sql-style.gguf 0.6

# Merging into a single file instead
llama-export-lora \
  -m base-f16.gguf \
  --lora ./adapters/support-tone.gguf \
  -o base-with-tone-f16.gguf

The quantization order problem

This is the part that decides the answer more often than disk does, and it is easy to get wrong.

The adapter was trained against weights at some precision. Merging it into an already quantized base means adding a full-precision update to weights whose values have been snapped to a quantization grid, then snapping the result back — two lossy operations stacked, with the second one applied to a model whose error budget has already been spent. The llama.cpp project has fielded reports of exactly this presenting as a merged model that appears to have lost its training. The --allow-requantize flag on llama-quantize carries an explicit warning in its own documentation that requantizing “can severely reduce quality compared to quantizing from 16bit or 32bit”.

The order that works is: merge into the full-precision base first, then quantize the merged result once. That requires you to have the fp16 or bf16 base on disk, which for a 70B is a large ask and is itself an argument for keeping the adapter separate on a machine that only ever held the quantized file. See what requantizing costs for the general case.

Which one, for which deployment

• One fine-tune, shipped to machines you do not control — merge, from the full-precision base, then quantize. One artifact, one checksum, no way to run it wrong.
• Several fine-tunes on one card — keep them separate. This is the case merging cannot serve at all, because merged variants do not share VRAM.
• Still iterating on the fine-tune — keep them separate. A merge is minutes of compute and a full file write per attempt; swapping an adapter is a restart.
• You only have the quantized base — keep them separate, and accept that you are applying an adapter to quantized weights. Merging here means requantizing, which is the outcome you were trying to avoid.
• Air-gapped or tightly audited deployment — merge. Two files with a scaling parameter between them is two things to verify and one more way for a deployment to be subtly wrong. See air-gapped deployment.

One thing that is not a factor either way: licensing. The base model’s licence governs the merged artifact exactly as it governs the base, because the merged weights are a derivative of it. Merging does not launder a restriction and separating does not create one. Check the base licence for redistribution terms before you publish either form.