Running Multiple Models at Once in Ollama

Ollama has served more than one model at a time for a long while, so the question is not whether it can but what it costs and what it does when the cost exceeds the card. Both answers are more specific than the documentation makes them look, and both are worth knowing before you point a router at a single Ollama host.

Two limits, doing different jobs

These get conflated constantly, and they control unrelated things. Ollama’s FAQ defines both.

• OLLAMA_MAX_LOADED_MODELS is the maximum number of different models resident concurrently, provided they fit in available memory. The documented default is 3 times the number of GPUs, or 3 for CPU inference. This is the one that governs “can I serve a chat model and an embedding model from one host”.
• OLLAMA_NUM_PARALLEL is the maximum number of parallel requests each model will process at the same time, documented as defaulting to 1. This is the one that governs whether two users of the same model wait for each other.
• OLLAMA_MAX_QUEUE is the backstop: the maximum number of requests Ollama will queue when busy before rejecting additional ones, default 512. Requests beyond that are refused rather than held.

The important interaction is that these multiply. Three loaded models each with four parallel slots is twelve concurrent sequences, and every one of those slots owns its own KV cache. Raising OLLAMA_NUM_PARALLEL to improve throughput while leaving the model count alone is the usual way to run a machine out of VRAM without touching any obviously memory-related setting.

What each resident model costs

The budget for one model is weights plus cache, and only the first part is fixed.

Weights are the size of the file on disk, near enough. A quantized GGUF is loaded essentially as-is, so the number in the registry listing or on the model card is the number to plan with.

Cache is bytes-per-token times context times parallel slots, and bytes-per-token is a property of the architecture. Working it out for one model — the derivation is on Ollama’s default context length — is enough to see the shape:

per model =  weights
          + (bytes_per_token x num_ctx x OLLAMA_NUM_PARALLEL)
          + a per-model runtime overhead

total     =  sum over resident models
          +  OLLAMA_GPU_OVERHEAD (whatever you reserved)
          +  whatever else is on the card

Two terms in that are easy to forget. The first is that the desktop itself holds VRAM — a compositor and a browser can account for a couple of gigabytes on a workstation, and the card does not offer them back. The second is OLLAMA_GPU_OVERHEAD, documented as reserving a portion of VRAM per GPU in bytes; it exists precisely because the scheduler’s estimate of free memory is an estimate, and setting it is how you stop a load that fits on paper from failing in practice.

The asymmetry worth planning around is that a small model is disproportionately cheap. An embedding model is often a few hundred megabytes with a short context, so keeping one resident next to a chat model costs a rounding error. Two 8B chat models at a long context, on the other hand, can each cost more in cache than a small model costs in total.

What happens when it does not fit

Not an error. The FAQ describes the sequence directly: when there is not enough memory to load a new model, all new requests are queued until the new model can be loaded, and as prior models become idle, one or more will be unloaded to make room.

Read the word idle. Eviction is not immediate and it is not preemptive: a model that is mid-generation is not thrown out, and a model that is merely loaded but not currently working is a candidate. The practical consequence is a latency spike rather than a failure. A request for model B arrives, model A is busy, B’s request waits for A to finish, A is unloaded, B is loaded from disk, and only then does generation start. On a large model read from a spinning disk or a network mount, that load is seconds — sometimes tens of them.

This is why alternating between two models that do not co-resident-fit is the worst pattern available. Each request pays a full load, and throughput collapses to something far below either model’s standalone rate, with no error anywhere to explain it. If you see a bimodal latency distribution — most requests fast, a regular minority enormously slow — thrashing between models is the first hypothesis.

OLLAMA_LOAD_TIMEOUT, documented as how long to allow model loads to stall before giving up with a default of 5 minutes, is the eventual backstop. If you are hitting it, the answer is fewer resident models rather than a longer timeout.

Measuring what is actually loaded

Do not reason about this from configuration. Ask the server:

ollama ps

# or the machine-readable form
curl -s http://localhost:11434/api/ps

Each entry names the model and reports size, size_vram, context_length and expires_at. Those four answer the questions you actually have. size against size_vram tells you whether a model is fully on the card or partly in system RAM — if size_vram is smaller, the difference is being carried by the CPU and that model is slow. context_length tells you what the cache was sized for. expires_at tells you when the keep-alive timer will unload it, which is how you distinguish “the model is resident” from “the model is about to stop being resident”.

Watch the sum of size_vram across entries against your card’s total rather than watching a single model. Card-level usage from nvidia-smi is a useful cross-check but includes everything else on the GPU, which is the point of checking it.

Arrangements that work

• One large model plus small specialists. A chat model, an embedding model and possibly a reranker co-resident is the arrangement the defaults are built for, and it rarely needs tuning.
• Pin the hot model, let the rest expire. OLLAMA_KEEP_ALIVE is documented as defaulting to five minutes, and a negative value keeps a model loaded indefinitely. Setting a long or infinite keep-alive on the model that serves interactive traffic and a short one on batch models gets you predictable latency where users can feel it. See how keep_alive works.
• Raise parallelism only after measuring the cache. Going from one slot to four quadruples cache memory for that model. On a machine that is already near the limit this converts a working setup into a thrashing one. The num_parallel setting covers the throughput side of that trade.
• Separate hosts beat clever scheduling. If two large models both need to be hot, the honest answer is usually two servers or two cards, with OLLAMA_SCHED_SPREAD — documented as always scheduling a model across all GPUs — as the alternative when one model is too large for a single card rather than as a way to fit two.