Multi-LoRA Serving: Many Adapters, One Base Model

A merged fine-tune occupies a GPU. An unmerged adapter occupies a few megabytes. That ratio — roughly a thousand to one — is what makes per-customer fine-tuning a product rather than a pilot.

The idea

From the LoRA arithmetic, the adapted forward pass is y = W·x + (α/r)·B·(A·x). The base weight W is shared by everyone; only A and B are per-customer, and they are tiny. So keep one copy of the base resident, keep many adapters resident beside it, and select the right pair per request.

The consequence is a different unit economics. Instead of provisioning capacity per fine-tune, you provision capacity for aggregate throughput and the number of distinct fine-tunes becomes almost free. A customer with 200 requests a day gets their own model without getting their own GPU.

The memory arithmetic

Take an 80 GB card serving a 7B base with a fleet of rank-16 adapters targeting attention and MLP projections — around 20M parameters each, 40 MB in bf16.

base model, bf16
  6.74e9 x 2 bytes                        =  13.5 GB

adapters, r=16 all-linear, bf16, 40 MB each
    100 adapters                          =   4.0 GB
    500 adapters                          =  20.0 GB
  1,000 adapters                          =  40.0 GB

with r = 8, attention only (8.4 MB each)
  1,000 adapters                          =   8.4 GB

remaining for KV cache on an 80 GB card
  80 − 13.5 − 8.4  (r=8 fleet)            =  58 GB
  80 − 13.5 − 40.0 (r=16 fleet)           =  26 GB

The line that matters is the last pair. KV cache is what determines how many concurrent requests you can serve, so adapter memory competes directly with throughput. A thousand rank-16 all-linear adapters is affordable in the sense that it fits and expensive in the sense that it costs you more than half your concurrency.

This is why adapter rank is a serving decision and not only a training decision. If you are running a per-customer fleet, the difference between r = 8 on attention only and r = 32 on every linear layer is roughly a factor of ten in resident adapter memory, and you will feel it as a concurrency limit rather than as an out-of-memory error.

The batching problem

Memory is the easy half. The hard half is that throughput on a GPU comes from batching, and batching normally means many requests going through the same weights at once. Here, request 1 needs adapter A, request 2 needs adapter B, and naively you can only batch requests that share an adapter — which destroys the batch size exactly when you have many customers, which is the case you built this for.

The resolution: the base matmul is shared and can be batched normally, because W is the same for everyone. Only the low-rank term is per-request. So the batch splits into one large dense operation over all requests plus a gather of small per-request low-rank operations, and the problem reduces to making that gather efficient — a grouped or segmented matrix-vector product over heterogeneous small matrices.

What the systems papers do

• Punica (Chen et al., 2023, arXiv 2310.18547) introduced a CUDA kernel for exactly the gather described above — segmented gather matrix-vector multiplication — so that a single batch can carry requests bound to different adapters without serialising them.
• S-LoRA (Sheng et al., 2023, arXiv 2311.03285) addresses the memory management side: adapters are held in host memory and paged into a unified pool alongside the KV cache, with custom kernels for batched heterogeneous ranks. The paper’s framing is serving thousands of concurrent adapters from a single machine.
• Production inference servers have since absorbed the approach; several open-source serving stacks expose multi-adapter loading as a first-class feature, and hosted vendors offer per-tenant adapters on shared bases. Check the specific limits of whichever you use — maximum concurrent adapters, maximum rank and whether ranks may differ across the fleet vary by implementation and by version.

The paging idea in S-LoRA is the one worth understanding even if you never read the kernels. Adapter usage is bursty and long-tailed: a handful of customers are active at any moment, and the rest are idle. Treating GPU adapter memory as a cache over host memory rather than as permanent residence is what lets the fleet be much larger than the card.

Binding a request to an adapter

The mechanism is usually a field in the request naming the adapter, and that innocuous design has two consequences worth deciding deliberately rather than inheriting.

The first is a security boundary. If the adapter identifier comes from the client, a client can name somebody else’s adapter. A fine-tuned adapter is trained on a customer’s data and can reproduce fragments of it, so serving tenant A’s adapter to tenant B is a data-exposure incident, not a routing bug. The adapter identifier should be derived server-side from the authenticated principal, and if it must be client-supplied it needs authorisation against the caller’s identity on every request — the same treatment you would give a database tenant id.

The second is what happens on a miss. A request naming an adapter that does not exist, has been deleted, or failed to load has two possible behaviours: fail the request, or silently serve the base model. Silent fallback is the more attractive default and the more dangerous one, because the customer keeps getting plausible answers from the wrong model and nobody finds out for weeks. Fail loudly, and make the adapter identity part of the response metadata so that “which weights produced this?” is answerable from a log line rather than from a reconstruction.

Related: version adapters explicitly rather than overwriting them in place. An adapter file swapped under a stable name means the answer to that question changes retroactively for every request already logged, and a rollback becomes a restore-from-backup instead of a pointer change.

Where it stops working

• Heterogeneous ranks cost you. Kernels are most efficient when every adapter in a batch has the same rank. A fleet with ranks scattered from 4 to 128 batches worse than a uniform one. Standardising the rank across your fleet is a real throughput decision.
• Cold adapters add latency. If an adapter is not resident, the first request pays a load. Tens of megabytes over PCIe is milliseconds rather than seconds, but it lands on the tail latency, and per-customer traffic is exactly the pattern that keeps adapters cold.
• Only one base. Every adapter in the fleet must target the same base checkpoint. When you migrate the base you migrate every customer’s adapter at once, retraining and re-evaluating the whole fleet.
• Some fine-tunes cannot be adapters. Full fine-tuning, a resized embedding table for an extended tokeniser, or continued pretraining all produce a different base rather than an adapter, and none of them can share memory with anything.
• Isolation is logical, not physical. Every customer’s adapter is in the same process and the same GPU memory as everyone else’s. If your compliance story requires physical separation, this architecture does not provide it.