Serving an Open Model to Your Whole Team

One person on a laptop and ten people on a server are different engineering problems, and the second one is mostly about memory bookkeeping. It is entirely calculable in advance, which is fortunate, because discovering it at 10am on a Monday is not pleasant.

Why a batching server, not a bigger laptop

Single-user runtimes process one request at a time. Two people asking at once means the second waits for the first to finish generating, which is a poor experience and a terrible use of the hardware.

Continuous batching solves this. Because generation is bound by streaming weights out of memory rather than by arithmetic, the weights fetched to produce one token can produce tokens for many sequences at once. A server that admits new requests into the running batch at each step — vLLM, SGLang and TGI all do this — turns ten concurrent users from a tenfold slowdown into a modest one, with aggregate throughput several times the single-stream rate.

The cost of that is memory. Every sequence in flight owns a KV cache, and that is the constraint you are about to compute.

The capacity model

vram = weights + kv_total + activation_overhead

weights  = params * bits_per_weight / 8
kv_token = 2 * layers * kv_heads * head_dim * elem_bytes
kv_total = kv_token * avg_context * max_concurrent_sequences

# rearranged, which is the form you actually want:
max_concurrent = (vram - weights - overhead) / (kv_token * avg_context)

Note max_concurrent_sequences, not users. Users are not continuously generating: they read, think and type. The ratio between headcount and simultaneous in-flight requests is the number that decides your sizing, and for interactive chat it is heavily in your favour — a team of ten produces far fewer than ten concurrent generations, most of the time.

Paged attention makes this better still. Servers that allocate the cache in fixed blocks rather than reserving the full context per sequence charge you for tokens actually generated, so a conversation that stops at 800 tokens does not hold a 32k reservation. Your avg_context should therefore be an average, not the maximum.

Sizing for ten people

Take the 8B-class model from the arithmetic page — 32 layers, 8 KV heads, head dimension 128 — on a single 24 GB card, served at fp16 rather than a GGUF quant because that is what a batching server wants.

weights, fp16   8e9 * 16 / 8 = 16.0 GB      -> 14.9 GiB
overhead        CUDA context, activations   ->  ~2.0 GiB
left for KV     24 - 14.9 - 2.0             ->   7.1 GiB

kv_token        2 * 32 * 8 * 128 * 2        = 128 KiB
avg_context     4,000 tokens (measure yours)
kv_per_seq      128 KiB * 4000              = 500 MiB

max_concurrent  7.1 GiB / 500 MiB           = 14 sequences

Fourteen simultaneous generations comfortably serves a team of ten doing interactive work. Now watch what one decision does to it:

• Raise average context to 16k and kv_per_seq becomes 2 GiB — capacity falls to three. Long documents in the prompt are the single most common reason a deployment that was fine stops being fine.
• Serve the same model with four-bit GPU-native weights and you free roughly 11 GiB for cache, taking concurrency past 35 at 4k context. On a memory-constrained box, quantising the weights buys concurrency, not just fit.
• Move to an 8-bit KV cache and every figure above doubles.
• Choose a mixture-of-experts model and the weights term follows total parameters while throughput follows active ones — good compute economics, hard memory economics on one card.

The serving configuration

vllm serve org/model-8b-instruct \
  --max-model-len 8192 \            # hard cap; also caps kv_per_seq
  --gpu-memory-utilization 0.90 \   # leave room for the driver
  --max-num-seqs 16 \               # your computed max_concurrent
  --api-key "$TEAM_KEY" \
  --host 0.0.0.0 --port 8000

# two cards, one model too large for either:
#   --tensor-parallel-size 2
# quantised weights to buy KV headroom:
#   --quantization awq          (weights must be an AWQ checkpoint)

--max-model-len is the important one and it is a policy decision, not a technical default. It bounds the worst-case cache a single request can take, which is what stops one person pasting a novel and evicting everybody else. Set it to the smallest value your real work needs.

Put an authenticating reverse proxy in front. The server’s own key is a single shared secret with no per-user attribution, and the moment more than a handful of people use this you will want to know who sent what, be able to revoke one of them, and apply a rate limit that stops a runaway script from consuming the batch.

Checking the model against reality

The calculation is a prediction. Test it before people depend on it, with the concurrency you predicted and prompts shaped like your real ones:

# vLLM ships a serving benchmark; drive it at your predicted concurrency
vllm bench serve \
  --model org/model-8b-instruct \
  --base-url http://localhost:8000 \
  --dataset-name random \
  --random-input-len 3000 --random-output-len 400 \
  --max-concurrency 14 --num-prompts 200

# read: p99 time to first token, output tok/s per request,
#       and whether any request was queued rather than admitted.

Then repeat at double the concurrency, because what you need to know is the shape of the failure. A batching server under pressure queues rather than crashing: time to first token climbs, per-request generation rate falls, aggregate throughput stays roughly flat. That is a graceful degradation, and knowing the concurrency at which p99 time-to-first-token crosses what your users will tolerate is the real capacity number. Watch nvidia-smi alongside it — if memory is near the ceiling at your target concurrency, your average context estimate was optimistic.

What actually breaks

• Context creep. Someone adds retrieval, average prompt length triples, concurrency collapses. Alert on mean prompt tokens, not just on request count.
• One user, one script. A batch job pointed at the team endpoint saturates the batch. Per-key rate limits at the proxy, not politeness.
• Cold starts. Loading tens of gigabytes of weights takes real time. Do not let an orchestrator restart the container on a liveness check tuned for a web service.
• The single point of failure. One box means one outage. Decide in advance whether the fallback is a hosted endpoint behind the same OpenAI-compatible interface, or an accepted downtime — but decide, rather than discovering.
• Upgrades. Serving-engine releases move quickly and occasionally change behaviour around templates, tool calling or quantisation support. Pin versions, and re-run your eval set after an upgrade rather than after a complaint.