The Flash Attention Flag in llama.cpp and What It Saves

Flash attention does not change what attention computes. It changes whether the intermediate scores are ever written to memory, and at long context that intermediate is one of the largest single allocations llama.cpp makes.

The flag has three values

It is -fa or --flash-attn, it takes on, off or auto, and it defaults to auto. It is no longer a boolean, so -fa 1 from an older guide is the wrong spelling, and llama-bench carries the same three-valued flag with the same default so that a benchmark and a server run mean the same thing by it.

auto means llama.cpp decides per run: it uses the fused kernel where the backend and the model support it, and takes the ordinary path where they do not. That is why the flag often appears to do nothing — on a modern build with a supported backend, on and auto are frequently the same run. off is the setting with a guaranteed effect, and it is mostly useful for isolating a suspected kernel bug.

What the non-flash path allocates

The two paths sit next to each other in src/llama-graph.cpp. Without flash attention, the graph multiplies keys by queries into a tensor the code calls kq, applies the mask and the softmax to it, then multiplies by the values:

ggml_tensor * kq = ggml_mul_mat(ctx0, k, q);
ggml_mul_mat_set_prec(kq, GGML_PREC_F32);   // "this op tends to require high floating point range"
kq = ggml_soft_max_ext(ctx0, kq, kq_mask, kq_scale, ...);
ggml_tensor * kqv = ggml_mul_mat(ctx0, v, kq);

The flash path replaces all four lines with one call to ggml_flash_attn_ext, which tiles the same computation and keeps each tile in registers and shared memory. The scores are never assembled as a whole tensor, so they never need space.

kq is the allocation in question. Its shape is the number of cached keys by the number of tokens in the current micro-batch by the number of attention heads, and the code explicitly asks for F32 precision on it, because the operation needs the range.

The arithmetic at a given context

Every input below is something you can read off your own run, and the result is only as good as the assumptions, so they are stated rather than hidden. Take a model with 32 attention heads — llama.cpp prints n_head when it loads the model — the default micro-batch of 512 from -ub, a single sequence, and a KV cache that has filled to 32,768 entries. Then:

bytes(kq) = n_kv x n_ubatch x n_head x 4          (F32)
          = 32768 x 512 x 32 x 4
          = 2,147,483,648 bytes
          = 2 GiB

Two gigabytes of compute buffer for a tensor that exists only inside one attention operation. With flash attention that term disappears; the fused kernel’s output is the attended values, shaped head dimension by heads by tokens, which at a head dimension of 128 is 128 x 32 x 512 x 4 = 8 MiB. The gap between 2 GiB and 8 MiB is the whole of what the flag saves.

Three qualifications keep that honest. The buffer is reused across layers rather than allocated per layer, so this is one 2 GiB, not thirty-two of them. It only reaches full size when the cache is full and a large micro-batch is being processed, which is prefill of a long prompt — during single-token generation the middle term is 1 and the same expression gives 4 MiB. And it scales linearly in both n_kv and n_ubatch, which is why lowering -ub is the standard remedy for an out-of-memory error during prefill when you cannot turn flash attention on: dropping the micro-batch from 512 to 128 turns that 2 GiB into 512 MiB.

What it does not save

The flag does not shrink the KV cache. The cache holds keys and values for every position and both paths read it; its size is set by the context length, the number of key-value heads and the cache types you chose with -ctk and -ctv. If your memory problem grows with conversation length rather than with prompt length, flash attention is not the lever — the context-size arithmetic is, and what the KV cache stores explains why the two grow differently.

It also cannot always be used. The graph asserts that flash attention does not support a KQ bias, so architectures that add a positional bias matrix to the scores take the ordinary path regardless of your flag, and the fused kernel has to exist for your backend and your head dimension. This is not an error you will see; it is a fallback, and it is the reason a memory figure that improved on one model does not improve on another.

Nor is it a correctness switch. The fused kernel computes the same function, and llama.cpp asks it for F32 precision on its output, so the two paths agree to within floating-point reassociation. They are not bit-identical, because the tiled algorithm accumulates in a different order — which matters only if you are diffing two runs token by token and are startled that a sampled sequence diverged. That is the same class of effect as changing the batch size, not evidence that one path is wrong.

Why the default is auto

Because the kernel matrix is uneven. Support differs by backend, by head dimension and by KV cache type, and the combination that is fastest on one is unavailable on another — quantised KV caches in particular are only worth having where a fused kernel handles that type, and where it does not, the runtime pays to dequantise on every attention step. auto encodes that decision so that a single command line behaves sensibly on a Mac, a CUDA box and a Vulkan build.

Which leaves a short rule. Leave it at auto. Set it to on when you want a run to fail loudly rather than silently fall back — for instance when the quantised KV cache you configured only pays off on the fused path. Set it to off only to test whether a numerical oddity is coming from the fused kernel, and expect the compute buffer to grow by roughly the expression above when you do.