LoRAs for Image Models: Style Without Retraining

A LoRA is a pair of thin matrices per adapted layer whose product is added to the original weight matrix. The file is small because the matrices are thin, and the thinness is also the capacity ceiling. Those are the same fact, and almost everything about what LoRAs can do follows from it.

The short answer

For a weight matrix W of shape (d_out × d_in), low-rank adaptation learns two matrices, A of shape (r × d_in) and B of shape (d_out × r), and uses W + BA in place of W. The rank r is small — 4 to 128 in practice — so the added parameters are a fraction of a per cent of the original.

The base model is frozen. A LoRA does not change what the model knows; it applies a small, structured shift to how existing features are combined. The general theory is covered in LoRA explained; this page is about what it means for image models specifically.

Deriving the file size

Parameters added per adapted matrix:

params(A) = r × d_in
params(B) = d_out × r
total     = r × (d_in + d_out)

Compare against the original matrix, d_out × d_in:

    A 640 × 640 projection, r = 16
        original  = 409,600 parameters
        LoRA      = 16 × (640 + 640) = 20,480 parameters
        ratio     = 5.0%

    The same matrix at r = 64
        LoRA      = 64 × 1280 = 81,920 parameters
        ratio     = 20.0%

Now scale it to a whole model. The adapted set is usually the attention projections — query, key, value and output — in every attention block, and sometimes the cross-attention only.

Assumptions, all stated:
  A1  100 adapted matrices  (four projections across ~25 attention blocks)
  A2  average width d_in = d_out = 768
  A3  weights stored in fp16, 2 bytes each

    per matrix at r = 16   :  16 × (768 + 768)  =   24,576 params
    across 100 matrices    :                       2,457,600 params
    on disk in fp16        :  × 2 bytes         =   4.9 MB

    the same at r = 32     :                       9.8 MB
    the same at r = 64     :                      19.7 MB
    the same at r = 128    :                      39.3 MB

Add the text-encoder adapters, which are commonly included, and a
rank-64-to-128 character or style LoRA lands in the 20-60 MB range.
That is where "a 50 MB file" comes from: a few million parameters at
two bytes each.

Two useful checks fall out of this. A LoRA file much larger than the arithmetic predicts is either high-rank, stored at full precision, or adapting more of the network than the attention projections. A LoRA that is a few hundred kilobytes is very low rank and will only be able to express a narrow adjustment.

Rank, alpha, and the weight you apply

The update is not applied raw. Implementations scale it:

W_effective = W  +  (alpha / r) · B·A · user_weight

  alpha        a constant fixed at training time
  r            the rank
  user_weight  the slider in your UI, usually 0 to 1 (or beyond)

The alpha / r factor exists so that changing the rank does not change the effective magnitude of the update, which would otherwise make every rank need its own learning rate. The practical consequences:

• A LoRA trained with alpha equal to r is at full trained strength at user weight 1.0. That is the common convention, and it is why 1.0 is the sensible default.
• A LoRA trained with alpha = r/2 is at half strength at weight 1.0, so it will appear weak and people push the slider to 2.0 to compensate. Nothing is wrong; the metadata is just not being read.
• Weights above about 1.2 usually degrade the base model. You are scaling a learned update beyond the magnitude it was validated at, and the symptoms — saturation, texture collapse, the LoRA’s training backgrounds appearing — look like overfitting but are an application-time error.

What a low-rank update can teach

The constraint is precise: for each adapted matrix, the update BA has rank at most r. It can shift that layer’s behaviour along at most r directions. That is a lot if the change you want is a consistent adjustment applied everywhere, and almost nothing if the change is different for every input.

• A style. Rendering choices — palette, brush behaviour, line weight, film stock, lighting convention — are exactly a consistent shift applied to everything. This is the best case, and it is why style LoRAs work at low rank and on small datasets.
• A specific subject. A face, a product, a character. The model already knows how to render a person; the LoRA associates a trigger with a particular point in that space. Needs more rank than a style, typically 16 to 32, and more varied data.
• A concept the model half-knows. If the base has seen something rarely, a LoRA can sharpen and name it. If it has never seen it at all, the LoRA has no features to reweight.
• A compositional bias. Always shoot from below, always centre the subject, always this framing. Cheap to teach and frequently taught by accident, which is why a LoRA trained on twenty portraits in the same pose produces that pose forever.

What it cannot

• Fix text rendering. The limit there is partly the autoencoder, which the LoRA does not touch at all, and partly the text encoder’s representation. A rank-32 shift in the denoiser’s attention layers cannot add spatial resolution the latent does not have. See why image models struggle with text.
• Fix anatomy in general. It can bias towards better-formed hands in the specific contexts it was trained on. It cannot install a counting mechanism.
• Extend the resolution range. The buckets a model was trained on are a property of the base. A LoRA trained at 1024 does not make a 512-base model work at 1024.
• Add spatial control. A LoRA has no per-pixel input. If you need to say where, that is structural conditioning, a different mechanism entirely.
• Teach something the base has no features for. This is the honest limit of the whole approach. Low-rank adaptation reweights an existing basis; if the basis lacks the direction you need, no rank you can afford will create it. That is the point at which continued training of the base becomes the answer.

Stacking, merging and the inference cost

Updates are additive, which makes stacking arithmetically simple and practically hazardous.

Two LoRAs applied together:

    W_eff = W  +  s1·(alpha1/r1)·B1·A1  +  s2·(alpha2/r2)·B2·A2

The two updates were each validated alone. Their sum was validated by
nobody. If each was trained to full strength, applying both at 1.0 applies
roughly twice the magnitude of shift the base was ever tested against.

A rule that follows from the arithmetic rather than from taste: when
stacking n LoRAs, start each at about 1/n of its solo weight and raise
from there.

Merging computes W + (alpha/r)BA once and writes the result into the weights. The adapter is then gone: no per-layer extra work at inference, and no ability to change the weight or remove it. Right for a fixed production configuration, wrong during iteration.

Unmerged inference cost is small and derivable. For one adapted matrix, the extra work per position is two thin matrix multiplications alongside the wide one:

original matmul   :  2 · d_in · d_out          FLOPs per position
LoRA extra        :  2 · r · (d_in + d_out)   FLOPs per position

At d_in = d_out = 768, r = 16:
    original  =  2 × 768 × 768        = 1,179,648
    extra     =  2 × 16 × 1536        =    49,152
    overhead  =  4.2%

At r = 64:   extra = 196,608  →  16.7% overhead on the adapted matrices.

Only the adapted matrices pay this, so the whole-model overhead is lower
again. Low-rank adapters are cheap to run; the reason to merge is
deployment simplicity, not speed.

The variants and what each one changes