text-generation-webui’s “CUDA Extension Not Installed” Warning

It scrolls past during startup, nothing fails, and generation is somewhere between disappointing and unusable. The line does not come from text-generation-webui at all — it comes from a quantization library it imported, and that tells you which loader to stop using.

The warning, and who prints it

CUDA extension not installed.

That exact string is AutoGPTQ’s, printed when its compiled CUDA module cannot be imported. It appears verbatim as the title of upstream reports including AutoGPTQ issue 246, and it is reported against text-generation-webui in issue 1289. An older, separate GPTQ implementation used in early builds prints a near-identical line for the same reason.

It is a print, not an exception, and the library continues. That is the whole difficulty with it: the interface gives no further indication that anything is degraded, and a user with no baseline has no way to know the speed they are getting is not the speed the model can do.

Which loader you are on

text-generation-webui is a front end over several backends and only some of them can produce this. Check the Model tab’s loader dropdown against the list:

• AutoGPTQ — the source of this warning. If it is selected, you have found your answer.
• ExLlamav2 and ExLlamav2_HF — separate compiled kernels of their own. They can fail to load too, but they say so differently, usually with an import error rather than a polite notice.
• llama.cpp — a compiled binary or a Python binding around one, not a torch extension. It does not print this; its GPU problem shows up as layers not being offloaded.
• Transformers — loading full-precision or bitsandbytes weights. It will print the warning if AutoGPTQ was imported anywhere in the process, which is a red herring: nothing you are running uses it.

That last case matters. The warning is emitted at import, so it can appear on a startup where no GPTQ model is ever loaded. If you are on llama.cpp or Transformers, the line is noise and your slowness has another cause.

What runs instead

When the extension is present, a GPTQ layer’s matmul is a fused kernel: packed 4-bit weights are read from memory, dequantized in registers, and multiplied against fp16 activations without a full weight matrix ever being written out. Since single-stream decoding is bound by memory bandwidth rather than arithmetic, moving 4-bit data instead of 16-bit is most of the point.

Without it, the library falls back to reconstructing the weight matrix with ordinary torch operations and calling a standard matmul. Every layer, every forward pass, allocates and writes a dequantized tensor and then reads it back. The arithmetic result is the same; the memory traffic is several times larger, and there is allocator churn that the fused path does not have. AutoGPTQ’s documentation states plainly that missing CUDA kernels result in very slow inference speed.

No ratio is quoted here, because it depends on the model, the group size and the card, and the only honest number is one you measure on your own machine by timing the same prompt before and after. What is structural is that the difference is large enough that you will not need a stopwatch to see it.

There is a second, quieter symptom that confirms you are on the fallback rather than merely warned about it: VRAM use goes up during generation rather than staying flat after load. The fused kernel never materialises a dequantized weight matrix, so its memory profile is essentially constant once the model is resident. The fallback allocates one per layer per forward pass, which shows in nvidia-smi as a moving figure. If you are unsure whether the warning applies to the model you are actually running, that is the tell.

Why the extension is missing

The usual reasons, in rough order of frequency: the wheel you installed was built without the extension, common for Windows builds; the install was run with the CUDA extension explicitly disabled, which some convenience installers do; there is no CUDA-enabled torch in the environment at all; or the extension exists but was built against a different torch or CUDA version and fails to import, which the library catches and turns into this print.

Distinguish the last from the others by importing directly — a module-not-found result is a missing build, while a symbol or version error is a mismatch:

python -c "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.is_available())"
python -c "import autogptq_cuda_256" 2>&1 | tail -2

If torch.cuda.is_available() prints False, that is the root cause and the kernel warning is downstream of it — the CPU-only torch page is where to go next.

Switching loader beats rebuilding

1. Select ExLlamav2 for the same checkpoint. It reads GPTQ weights and brings its own kernels, so you sidestep the library that is warning rather than repairing it. This is a dropdown change and a reload, which is why it is first.
2. Or use a GGUF quantization instead. The llama.cpp loader ships precompiled and asks nothing of your build environment, which is the right answer on a machine that cannot compile CUDA extensions comfortably.
3. Rebuild from source only if you must. In the same environment as your torch, with nvcc on PATH and the CUDA extension enabled. Verify by importing the kernel module, not by the warning being absent.

One habit worth adopting whichever route you take: record a baseline before you change anything. Time the same prompt with the same settings, note the tokens per second the interface reports, then make one change and time it again. Without that, a loader switch and a rebuild done on the same afternoon leave you unable to say which helped — and this is a class of problem where a change that appears to work sometimes only worked because a different model was selected.

The one-click installers deserve a specific mention here, because they are how most people arrive at this warning and how most people fail to leave it. They create their own environment, install their own torch, and are entirely capable of installing a CPU torch on a machine with a working GPU, or of putting a CUDA extension next to a torch it was not built for. If the diagnostic commands above disagree with what you believe you installed, trust the commands, and prefer re-running the installer’s own update script over hand-patching the environment it manages — a hand-patched managed environment tends to be reverted by the next update.