Running Two Local Models Side by Side as a Fallback Pair

A fallback pair is worth having for one reason: the failure you are covering is usually not a crash, it is the primary model being wrong or running out of context. That changes both what you load and what triggers the switch.

Two servers, not one server with two models

llama.cpp’s server holds one model per process. If you want two resident simultaneously you run two processes on two ports, and the routing lives above them. This is a feature rather than a limitation — each has its own context size, its own slot count and its own thread allocation, and restarting one to change a flag does not disturb the other.

Ollama is the opposite arrangement: one server, several models loaded concurrently, selected by name per request. Its documentation states the default limit as three times the number of GPUs, or three for CPU inference, controlled by OLLAMA_MAX_LOADED_MODELS. Which model answers is a field in the request body rather than a port number.

Both are fine. The choice is whether you want per-model process isolation and explicit flags, or one endpoint and automatic load-and-unload. What neither gives you for free is a policy about when to use the second model, which is the part that actually matters and the part at the end of this page.

One decision to make before either: the pair should differ in something that matters, not just in size. A primary and a fallback at the same capability is two copies of one failure. The useful pairings are a fast general model behind a stronger specialist, or a heavily quantized model behind a higher-precision one of the same family — for which which level suits which task is the deciding argument.

The memory budget, derived

Two models resident means two sets of weights and two KV caches. The weights you can read off the repository listing. The KV cache you derive from the model’s own published config.json:

KV bytes per token = 2 (K and V)
                   x num_hidden_layers
                   x num_key_value_heads
                   x head_dim
                   x bytes per element (2 for fp16)

Take a concrete pair: Llama-3.1-8B-Instruct as the fast primary and Qwen2.5-Coder-14B-Instruct as the stronger fallback for code. Their published configurations give 32 layers, 8 key-value heads and a head dimension of 128 for the 8B, and 48 layers, 8 key-value heads and a head dimension of 128 for the 14B. So:

Llama-3.1-8B:      2 x 32 x 8 x 128 x 2 = 131,072 B/token = 128 KiB
Qwen2.5-Coder-14B: 2 x 48 x 8 x 128 x 2 = 196,608 B/token = 192 KiB

At 8,192 tokens of context each:
  8B  KV:  128 KiB x 8192 =  1.00 GiB
  14B KV:  192 KiB x 8192 =  1.50 GiB

Weights (Q4_K_M, published file sizes):
  8B:   4.58 GiB
  14B:  8.37 GiB
                          ----------
  total weights + KV:      15.45 GiB

On a 24 GiB card that leaves roughly 8.5 GiB for compute buffers, the CUDA context of two processes, and whatever the desktop is using — so it fits, with room to raise one context window. On a 16 GiB card it does not, and the honest conclusion is a smaller fallback rather than a smaller context.

Two ways to shrink the total if it does not fit. Quantize the KV cache, which llama.cpp supports and which halves those figures at fp8. Or give the fallback a much smaller context than the primary, on the reasoning that it handles a minority of requests and you can afford to make those the short ones.

The llama.cpp arrangement

Two servers, two ports, distinct aliases so the model id in OpenAI-compatible responses tells you which one answered:

# primary: fast general model
llama-server \
  -m ./models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
  --alias primary-8b \
  --host 127.0.0.1 --port 8080 \
  -c 8192 -ngl all -np 2

# fallback: stronger coding model
llama-server \
  -m ./models/Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf \
  --alias fallback-14b \
  --host 127.0.0.1 --port 8081 \
  -c 8192 -ngl all -np 1

A few of those flags earn their place. --host 127.0.0.1 is the default and should stay that way unless you have deliberately decided otherwise; the server has no authentication of its own. -np sets the number of server slots, and the context you asked for is divided among them — two slots on a 8192-token context is two conversations of 4096, not two of 8192. -ngl all puts every layer on the GPU, which is what you want when the whole thing fits and is the difference between the derived speed and something much worse.

The Ollama arrangement

One server, both models held resident. The two settings that matter are the concurrent-model limit and the idle timeout, because the default five-minute unload will evict your fallback exactly when it has not been needed for a while — which is the moment before it is:

OLLAMA_MAX_LOADED_MODELS=2 OLLAMA_KEEP_ALIVE=-1 ollama serve

# hold each one in memory explicitly
curl http://localhost:11434/api/generate \
  -d '{"model": "llama3.1:8b", "keep_alive": -1}'
curl http://localhost:11434/api/generate \
  -d '{"model": "qwen2.5-coder:14b", "keep_alive": -1}'

A request with an empty prompt and keep_alive: -1 loads a model and pins it without generating anything, which is the documented way to preload. Ollama’s own documentation is clear that if a new model will not fit alongside the loaded ones, requests queue until something idles out and is unloaded — so an under-budgeted pair does not fail loudly, it just gets slow in a way that looks like the model being slow rather than the arrangement being wrong.

Also worth knowing: OLLAMA_NUM_PARALLEL defaults to 1, and raising it multiplies the context allocation. Four parallel requests against a 8192-token context allocates as if for 32,768 tokens. That is the same slot arithmetic as llama.cpp wearing different names.

What should actually trigger the fallback

The setup is the easy part; the policy is where these arrangements usually go wrong. A fallback that fires on connection errors covers a failure that almost never happens on a local process. The failures worth covering are these:

• Context exhaustion. The primary has a smaller window and the request does not fit. This is deterministic, detectable before you send anything by counting tokens, and the single most common real reason to route elsewhere.
• Output that fails validation. JSON that will not parse, a schema violation, code that does not compile. Retrying the same model rarely helps; escalating to a stronger one sometimes does. This is the case where a fallback genuinely earns its memory.
• A refusal or an empty completion on a task the second model handles. Check the finish reason rather than the text.
• Genuine process failure. The server exited, the GPU fell over, the port is closed. Real, but rare, and covered by any connection-level retry.

Whatever the trigger, make the escalation visible. A pair that silently routes half its traffic to the slow model is indistinguishable from a slow primary until you log which one answered — which is the argument for the distinct --alias values above, since the model id comes back in every response. Record the escalation rate alongside each model’s own tokens per second, because a fallback firing on a third of requests is not a fallback, it is the wrong primary.