GPTQ’s “Exllama Kernel Not Installed” Warning

Nothing fails. The model loads, generates correct text, and prints a warning on the way in that it is not using the kernel it wanted. The warning is worth acting on, because what it is describing is the difference between a fused 4-bit matmul and a much more roundabout route to the same numbers.

The warning, and what I can verify of it

The line begins Exllama kernel is not installed, and continues into an explanation that this usually happens when the package was installed from a prebuilt wheel on Windows, in which the exllama kernels are not compiled, together with the advice to reinstall from source. It is frequently accompanied by a second, separate line:

CUDA extension not installed.

Both lines are emitted at import or load time and both are warnings rather than errors, which is why they get scrolled past. There is no later symptom except speed.

Why the kernel is missing

A GPTQ checkpoint stores weights as 4-bit integers packed into 32-bit words, with a scale and a zero point per group. Multiplying by them requires code that understands that packing. That code is a compiled CUDA extension, built at install time, and it is absent for one of four reasons:

• You installed a prebuilt wheel that does not contain it. The dominant case, and the one the warning itself names. Wheels are built for a specific torch and CUDA combination; the kernels are frequently omitted from the Windows builds.
• The build was told to skip it. BUILD_CUDA_EXT=0 at install time is documented upstream as producing exactly this, and it is set by more than one convenience installer.
• There is no CUDA to build against. A CPU-only torch, or a machine with no nvcc. If torch.cuda.is_available() is False, this is your cause and the kernel warning is a symptom — see the CPU-only torch page.
• The extension exists but will not import. Built against a different torch or CUDA version than the one now installed, so the shared object fails to load and the library catches that and warns. This one is invisible in a package listing, because the package is installed.

You can tell the last case from the others by trying the import directly. If it raises rather than reporting that the module does not exist, you have a version mismatch, not a missing build.

python -c "import torch; print(torch.__version__, torch.version.cuda)"
python -c "import exllama_kernels" 2>&1 | tail -2
python -c "import autogptq_cuda_256" 2>&1 | tail -2

Both messages are printed, not raised, and that is a deliberate choice by the library rather than an oversight: a model that runs slowly is more useful than a model that will not load, and on a CPU-only machine the fallback is the only path there is. The cost of that choice is that the warning is easy to lose in a wall of startup output, and many people never see it at all because a launcher script swallows stdout.

What the fallback actually does

The fallback is not a slower version of the same kernel. It takes a different route to the same answer, and knowing which route explains the size of the gap.

The exllama kernel is fused: it reads packed 4-bit weights straight from memory, unpacks and dequantizes them in registers, and does the multiply-accumulate against fp16 activations without ever writing a dequantized weight matrix anywhere. Because weight-bound decoding is limited by memory bandwidth rather than arithmetic, moving 4-bit data instead of 16-bit is most of the reason quantization is fast at all.

The generic fallback path reconstructs the full-precision weight matrix using ordinary torch operations, then calls a standard matmul on it. That means allocating and writing a dequantized tensor for every layer on every forward pass, and then reading it back — several times the memory traffic of the fused path, plus allocator pressure that was not supposed to exist. AutoGPTQ’s own documentation says missing CUDA kernels result in very slow inference speed, and this is the reason why.

No figure is quoted here because the ratio depends on the model, the group size and the card, and the honest way to get yours is to time the same prompt before and after the fix on your own machine. What is structural, and true everywhere, is that the gap is large enough to be obvious without a stopwatch.

Three ways out

1. Switch loader to ExLlamaV2 directly. Usually the fastest route in both senses. ExLlamaV2 ships its own compiled extension and reads GPTQ checkpoints, so you bypass the library that is warning at you rather than repairing it. In a front end this is a dropdown, not a reinstall.
2. Rebuild from source against your torch. Install with the CUDA extension enabled, in the same environment as the torch you will run with, with nvcc on PATH. Verify by importing the kernel module afterwards rather than by the absence of the warning, which can also be absent for the wrong reasons.
3. Move to a GGUF build instead. If the machine is hostile to compiling — Windows without build tools, a slim container, an unusual Python — a llama.cpp quantization ships precompiled and sidesteps the whole class of problem. Same model, different container format; see the llama.cpp guide.