A “GGML_ASSERT” Crash in llama.cpp

The process prints a line naming a source file, a line number and a boolean expression, then aborts. It looks like an internal error you cannot act on. It is the opposite: the expression is the most specific diagnostic in the whole system.

The anatomy of the line

GGML_ASSERT is a macro, and reading its definition tells you how to read its output. In current ggml it expands to an abort with a formatted message:

#define GGML_ABORT(...) ggml_abort(__FILE__, __LINE__, __VA_ARGS__)
#define GGML_ASSERT(x) if (!(x)) GGML_ABORT("GGML_ASSERT(%s) failed", #x)

So a modern crash reads like /src/ggml/src/ggml.c:4231: GGML_ASSERT(ne02 == ne12) failed, followed by a backtrace if one could be produced. Older builds used a different format — GGML_ASSERT: ggml.c:4014: false — because the macro used to print through fprintf and call abort() directly. Both spellings are in the wild and both mean the same thing: an invariant the code requires was not true, and the author chose to stop rather than compute nonsense. The definition is in ggml.h in the ggml repository.

Three things in the line carry information. The file says which layer failed: ggml.c or ggml-cpu is the core tensor library, ggml-cuda.cu or ggml-metal or ggml-vulkan is a backend, ggml-alloc.c is the graph allocator, llama.cpp itself is model-level logic. The expression says what was violated. The line number is only useful together with the exact commit you built, which is why every bug report asks for it.

Cause 1: a build older than the model

The single most common origin is a binary that predates the model file. A new architecture arrives with new tensor layouts, new metadata keys and sometimes a new quantisation type. An older build may get far enough to start constructing a graph and then hit an assertion that encodes an assumption the new format breaks — a head count that must divide evenly, a tensor that must exist, a type that must be F16.

This is distinguishable from a genuine bug by one test: build from the current master and try again. If you are using a wrapper, the wrapper pins its own vendored copy, and that pin is usually weeks behind. Ollama embeds a llama.cpp snapshot; llama-cpp-python builds against a pinned submodule; LM Studio ships runtimes it versions separately. An assert coming out of any of those tells you about their vendored copy, not about upstream, so reproduce with an upstream llama-cli before concluding anything.

The reverse also happens, less often: a build newer than the GGUF conversion. Quantised files produced by an old convert script can carry metadata that current code has stopped accommodating. If the file predates a format change, requantising from the original weights is the clean path, and the pre-GGUF format page covers the case where the file is old enough to be rejected outright.

Cause 2: a shape invariant, and what ne means

Many asserts compare dimensions, and they use ggml’s naming convention. A tensor carries ne[0..3], the number of elements along each of four axes. In a binary operation the operands are src0 and src1, so ne02 is axis 2 of the first operand and ne12 is axis 2 of the second. An assert like ne02 == ne12 is therefore “these two tensors must agree on their third dimension and they do not” — typically a head-count or batch mismatch reaching a matrix multiply.

Once you can read that, the assert stops being generic. A comparison ofne values means shapes disagreed, which points at model metadata: a head count, a KV head count, an expert count or a sliding-window parameter read from the GGUF that does not match the weights present. Public issues on llama.cpp include exactly this shape of failure, where a config value was written into GGUF with the wrong type and produced a tensor-shape mismatch downstream. A type assert — src0->type == GGML_TYPE_F16 — means a kernel was handed a quantisation it has no path for, which is a backend limitation rather than a corrupt file.

An assert inside ggml-alloc.c about a buffer or a buffer_id is a third category again: the graph allocator could not place a tensor. That is usually downstream of memory, not of shape, and it is treated in the compute-buffer allocation page.

Cause 3: a backend-specific assert

If the file in the message names CUDA, Metal, Vulkan or SYCL, the invariant belongs to that backend and the same model may run fine on another. This is worth knowing because it gives you a one-command bisect: run with the GPU disabled and see whether the assert survives.

# force CPU only: no layers offloaded
llama-cli -m model.gguf -ngl 0 -p "hello" -n 16

# then narrow: half the layers
llama-cli -m model.gguf -ngl 20 -p "hello" -n 16

If -ngl 0 is clean and any offload asserts, you have a backend kernel that does not support something in this model — an unusual head dimension, a quantisation type without a kernel, an MoE routing tensor. Options that change how work is split across devices are worth eliminating too: split modes and tensor-split settings have their own assertions, and several reported crashes disappear when the split mode is changed. If the assert survives -ngl 0, it is core ggml or model logic, and the backend is a red herring.

Flash attention is the other high-yield toggle. It has its own kernels with their own supported head dimensions, so disabling it changes which code path runs and can make a backend assert vanish. That is a diagnostic, not a fix — it tells you which kernel to name in the report.

Getting a report that can be acted on

An assert that survives an up-to-date upstream build is a real bug, and the maintainers need four things. Collecting them takes a minute and is the difference between a fixed bug and a closed one:

1. The exact version. llama-cli --version prints the build number and commit; paste it verbatim rather than saying “latest”.
2. The full assertion line, with file, line number and expression, plus everything printed before it — the model load log names the architecture and the quantisation, and that is half the diagnosis.
3. The exact model file, by repository and filename. “A 7B Q4” is not reproducible; a specific GGUF from a specific repository is.
4. The command line in full, including -ngl, split mode, context size and flash-attention state, and whether -ngl 0 reproduces it.

A debug build makes the backtrace usable — cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo keeps optimisation while retaining symbols, so the trace names functions rather than addresses. For the wider picture of which runtime to build and which flags matter on which hardware, see the survey of local inference tools.