Skip to content
Multigrid

Local inference errors, string by string

The exact errors local runtimes emit, what each one means, and the fix — starting from the text you pasted into search.

A hosted model fails in one place, through one API, with a documented error code. A local model fails in a stack you assembled: a driver, a toolkit, a compiled kernel, a quantized file, a wrapper, and a launcher that hides most of them. The message you get back is usually printed by whichever layer noticed first, which is rarely the layer that is wrong. “Failed to load model” is true of a file that downloaded halfway, a file for an architecture your build predates, and a file that is perfectly fine on a machine with four more gigabytes of RAM.

These pages are organised the way the problem actually arrives: by the string. Each one opens on the literal text as the runtime prints it, names the specific mechanism that produces it, enumerates the distinct causes that share it, and gives you something to look at that tells them apart — a second line in the log, a digest, a flag that never appeared. Where a flag name or an environment variable is likely to be renamed in the next release, that is marked rather than asserted.

Ollama: “model requires more system memory than is available”

What the two numbers in Ollama's memory refusal are actually counting, and which of the four levers moves the one you can change.

9 min read

llama.cpp: “failed to allocate compute buffers”

Why this failure comes after the weights loaded successfully, and why batch size rather than model size is what fixes it.

9 min read

CUDA Out of Memory the Instant a Model Starts Loading

Why an OOM that lands during weight loading points at your offload setting rather than at your workload.

9 min read

Hugging Face 403 When Downloading a Gated Model

The three distinct states a gated-repo 403 covers, and how to tell which one you are in before you start changing tokens.

8 min read

LM Studio: “Failed to Load Model”

How to read the llama.cpp line under LM Studio's banner and separate a memory failure from a corrupt file from an unsupported architecture.

8 min read

Ollama: “address already in use” on Port 11434

Why the port is usually held by an Ollama you did not know was running, and how to find and release it on each platform.

8 min read

“Illegal Instruction” Running llama.cpp on an Older CPU

Why a binary built for AVX2 dies instantly on a CPU without it, how to confirm that is what happened, and the build flags that produce a binary that runs.

9 min read

whisper.cpp: “failed to load model”

The three distinct load failures whisper.cpp reports, separated by the line above the failure rather than by guesswork.

8 min read

MLX: No Metal Device Available

Why MLX refuses to start without a GPU it can see, and the four environments that hide one from it.

8 min read

Running Out of Memory During Quantization, Not Inference

Why producing a 4-bit model needs far more memory than running one, and which stage of which method is actually consuming it.

9 min read

Ollama: “digest mismatch, file must be downloaded again”

What a content-addressed digest mismatch proves, the one digest value that identifies an empty file, and how to clear the partial blob.

8 min read

“No Kernel Image Is Available for Execution on the Device”

Why this is a compile-target mismatch rather than a driver problem, and how to find the compute capability your build is missing.

9 min read

A GGUF Model Loads but Outputs Garbage

Two kinds of nonsense, four causes, and a triage order that tells a template mistake from a rope mistake from a broken backend.

10 min read

Ollama: “pull model manifest: file does not exist”

Why this error is about a name rather than a network, and the four naming mistakes that produce it.

8 min read

llama.cpp Server's “Slot Unavailable” Error Under Load

Why llama-server refuses a request when every slot is busy, and how to work out from your KV budget how many slots the machine can actually afford.

10 min read

vLLM’s “No Available Memory for the Cache Blocks” Error

Why vLLM refuses to start when the KV block allocator gets nothing, and why the message's own suggestion is sometimes the wrong move.

10 min read

“Failed Building Wheel for llama-cpp-python”

Why pip tries to compile a C++ project at all, which piece of the toolchain is missing on each platform, and how to skip the build entirely.

11 min read

AWQ’s “Only Supports Batch Size 1” Assertion Error

Why an AWQ checkpoint packed for the GEMV kernel refuses any batch above one, and which setting on the checkpoint — not on the server — decides it.

9 min read

ExLlama’s “Expected Scalar Type Half” Error

Why ExLlama's kernels refuse a float32 tensor outright instead of casting it, and the four places the wrong dtype usually enters.

9 min read

GPTQ’s “Exllama Kernel Not Installed” Warning

What the warning means about which matmul is running, what ignoring it costs, and why a pip-installed wheel is usually the reason.

9 min read

Ollama Modelfile’s “Invalid File Magic” Error

What Ollama reads in the first four bytes of the file your FROM line points at, and the six things that are commonly there instead.

9 min read

Jan.ai’s Model Stuck on Loading

The spinner that never resolves has two usual causes with opposite fixes, and the log file that says which one you have.

9 min read

text-generation-webui’s “CUDA Extension Not Installed” Warning

Which loader prints this on the way in, what it silently falls back to, and why switching loaders is a better fix than rebuilding.

9 min read

KoboldCpp’s “Context Length Setting Exceeds the Trained Maximum” Warning

Three different context limits are in play at once, this warning is about the gap between two of them, and going past it degrades output in a specific way.

10 min read

“Torch Not Compiled With CUDA Enabled”

The assertion means the torch in your environment has no CUDA in it at all, which is a build you installed rather than anything about your GPU.

9 min read

MLX Getting Silently Killed Loading a Large Model

A process that vanishes with two words and no traceback was killed by macOS, and the exit code tells you that definitively.

10 min read

faster-whisper’s “Could Not Load Library libcudnn” Error

CTranslate2 opens cuDNN at the first GPU inference rather than at import, and a version split between 4.4 and 4.5 decides which file it looks for.

9 min read

LM Studio Stuck at “Preparing Model”

Why the loading dialog sits at the preparing stage forever, and how to tell a truncated download from an architecture the bundled runtime cannot build a graph for.

9 min read

Ollama’s Docker Container Can’t See the GPU

Why a containerised Ollama falls back to CPU with no error, and the two host-side pieces — the NVIDIA Container Toolkit and the runtime flag — that most guides omit.

9 min read

A “GGML_ASSERT” Crash in llama.cpp

How to read the assertion line, why an out-of-date build produces most of them, and what the shape-comparison asserts are really telling you.

10 min read

“CUDA Error: Device-Side Assert Triggered” Running a Local Model

Why this error names no cause, how to make it name one, and why a token ID outside the embedding table is the usual culprit in local inference.

10 min read

Ollama’s “Llama Runner Process Has Terminated” Error

The message reports that a subprocess died, not why — here is how to read its exit status and recover the real error from the server log.

9 min read

“libomp.so: Cannot Open Shared Object File” Running llama.cpp

Why a binary that built successfully cannot start, which OpenMP runtime your build actually needs, and the package name for it on each major distribution.

9 min read

A GGUF Model Returns an Empty Response Every Time

Why a model that loads cleanly generates zero tokens, and how a chat-template mismatch makes a stop token the very first prediction.

10 min read

“Cannot Import Name ‘AutoGPTQForCausalLM’”

Which module the error names decides the cause: a broken CUDA extension, a transformers version that removed a function auto-gptq imports, or a package that no longer has that class.

9 min read

MLX’s “Received Parameters Not in Model” Error

Why MLX refuses a checkpoint whose weight names do not match the module tree, and how to read the listed parameter names to find which mismatch you have.

9 min read

A Model Downloads in LM Studio but Won’t Appear in the List

Why the file is on disk and the app does not list it: the two-level folder layout the indexer requires, and the cases where only a restart or a re-index will help.

8 min read

“Unknown (Magic, Version) Combination” Loading an Older GGML File

What the two hex numbers in the message mean, why current builds reject the file, and what your options are now that the conversion scripts are gone too.

9 min read

whisper.cpp Hallucinating Text on Silence

Why silence produces confident invented sentences, why the built-in thresholds do not catch them, and which whisper.cpp flags actually reduce it.

10 min read

An Ollama Model Works via CLI but Not via the API

Everything the CLI fills in for you that an HTTP client does not: streaming defaults, chat templating, exact model tags and the host it is talking to.

10 min read

Other topics