Running llama.cpp on AMD and Intel GPUs With the Vulkan Backend

Vulkan is a graphics API, and llama.cpp uses it as a compute backend that works on AMD, Intel and Nvidia hardware from the same build. On a non-Nvidia GPU it is usually the shortest path from a checked-out repository to a token, because it does not require a vendor compute runtime to be installed and matched to a kernel.

Why Vulkan instead of the vendor stack

The alternatives on non-Nvidia hardware are ROCm on AMD and SYCL on Intel. Both are the vendor’s own compute stack, both are capable of more, and both have the same practical problem: they are large installs with specific supported-hardware lists and kernel-driver version coupling. An older or consumer AMD card can be outside the officially supported ROCm matrix while its Vulkan driver is perfectly current, because the graphics driver is the one the vendor ships to everybody who plays games.

Vulkan trades peak performance for that reach. The compute shaders are hand-written in llama.cpp rather than delegated to a vendor BLAS library, so a well-supported card on its native stack will generally be faster. Whether the gap matters depends on your card and your model, and the honest way to find out is to build both and run llama-bench — which is a weekend, whereas the Vulkan build is an afternoon.

It is also the only backend that works uniformly across an integrated Intel GPU, a discrete AMD card and an Nvidia card in the same codebase, which matters if you are shipping something other people will run on hardware you do not control.

What you actually need installed

Three things: a Vulkan loader and driver, a shader compiler, and SPIR-V headers. llama.cpp’s build documentation gives the package names, and the third is the one that trips people up.

• Debian or Ubuntu — sudo apt-get install libvulkan-dev glslc spirv-headers. The docs note explicitly that SPIR-V headers are required and are not always pulled in by the Vulkan loader dev package alone.
• Fedora or openSUSE — the headers package is called spirv-headers-devel.
• Windows — install the LunarG Vulkan SDK; its Include directory already contains the headers. Either w64devkit or MSVC with CMake works.
• macOS — Vulkan runs over MoltenVK here, but there is no reason to: the Metal backend is native and better supported.

Before building, run vulkaninfo. If it does not list your GPU, nothing llama.cpp does will find it either, and you have a driver problem rather than a build problem. On Linux the usual cause is a missing Mesa Vulkan driver package — mesa-vulkan-drivers on Debian-family distributions — or a user not in the render group, which blocks access to /dev/dri/renderD128.

Building it

1. Clone the repository: git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp.
2. Configure with the Vulkan backend enabled. This is one CMake variable:

   cmake -B build -DGGML_VULKAN=ON
   cmake --build build --config Release -j

3. If configuration fails on a missing spirv/unified1/spirv.hpp, that is the SPIR-V headers package and not a llama.cpp bug. Install it and re-run the configure step.
4. If it fails compiling shaders, your glslc is missing or too old. It ships with the Vulkan SDK and as shaderc in most distributions.
5. Binaries land in build/bin on Linux and macOS and build/bin/Release on Windows with MSVC.

There is also a maintained Docker path if you would rather not install the SDK at all: the repository ships .devops/vulkan.Dockerfile, and the container needs the render device passed through with --device /dev/dri/renderD128.

Verifying the right device was found

The Vulkan backend announces itself on startup, and this single line is worth reading carefully because it tells you four things at once. llama.cpp’s build documentation gives this example of its shape:

ggml_vulkan: Using Intel(R) Graphics (ADL GT2) | uma: 1 | fp16: 1 | warp size: 32

uma: 1 means unified memory — an integrated GPU sharing system RAM, so “VRAM” is a slice of your RAM and offloading does not buy you the bandwidth jump a discrete card would. fp16: 1 means half-precision shaders are available, and a 0 there means a significant slowdown. If the name is not the GPU you meant — a common outcome on a laptop with both integrated and discrete graphics — list what is available and choose explicitly:

llama-server --list-devices
llama-server -m model.gguf -dev Vulkan1 -ngl 99

-dev/--device takes a comma-separated list, and the special value none disables offloading altogether, which is a cleaner way to force a CPU comparison than setting -ngl 0.

Serving tokens

./build/bin/llama-server \
  -m ./models/qwen2.5-7b-instruct-q4_k_m.gguf \
  -ngl 99 -c 8192 \
  --host 127.0.0.1 --port 8080

Then confirm it end to end with the same OpenAI-shaped call any other llama.cpp server takes:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"local","messages":[{"role":"user","content":"One sentence: what is Vulkan?"}]}'

Check the offload line in the server log, not just that a response came back — offloaded 33/33 layers to GPU is the confirmation that the build is doing what you installed it for. The rest of the server surface is identical across backends and is covered in the server page.

What you give up

• Allocation limits. Some Vulkan drivers cap a single buffer well below total VRAM, and a large model can fail to allocate despite the memory existing. llama.cpp exposes GGML_VK_FORCE_MAX_ALLOCATION_SIZE and GGML_VK_FORCE_MAX_BUFFER_SIZE to work around driver-reported limits.
• Silent system-memory fallback. GGML_VK_ALLOW_SYSMEM_FALLBACK controls whether allocations spill to host memory instead of failing. Spilling turns an out-of-memory error into a mysteriously slow run, which is worse; if throughput collapses at a context that used to work, this is the first thing to check.
• Cooperative-matrix paths vary by driver. The fastest matrix-multiply shaders need driver extensions, and llama.cpp has environment switches such as GGML_VK_DISABLE_COOPMAT to turn them off when a driver reports support it does not deliver. A driver update changing performance in either direction is normal here.
• Newest quantization types arrive last. Each type needs a hand-written shader per backend, so an exotic IQ type may be unimplemented or slow on Vulkan while it is fast on CUDA. Q4_K_M and Q8_0 are the safe choices.