Ollama’s Docker Container Can’t See the GPU

The container is running, the model answers, and it answers at roughly a tenth of the speed you expected. In the server log, among lines about inference, sits no compatible GPUs were discovered — or nothing at all about GPUs, which means the same thing.

What the log actually says

Ollama probes for accelerators at startup and records what it found. On a working setup the log names the device and its memory. On a broken one it reports that no compatible GPU was discovered and continues on CPU. Get the log first, because “it feels slow” and “there is no GPU” are different problems:

docker logs ollama 2>&1 | grep -i -e gpu -e cuda -e "inference compute"

# and with detail:
docker run -d --gpus=all -e OLLAMA_DEBUG=1 -v ollama:/root/.ollama \
  -p 11434:11434 --name ollama ollama/ollama

OLLAMA_DEBUG=1 makes the discovery path verbose, and the verbose form is what tells you whether the CUDA driver library was found and failed to initialise, or was never present in the container at all. Those two lead to different fixes.

Why it degrades instead of failing

This is the part that costs people an afternoon. Ollama is designed to run on machines without a GPU, so the absence of one is a supported configuration rather than an error. A container that cannot see any accelerator therefore starts normally, serves requests normally, and differs only in throughput. Nothing in the API response says which backend produced the tokens.

Docker behaves the same way one layer down. Passing --gpus=all on a daemon that has no NVIDIA runtime registered does not always error; depending on the Docker version and how the request is interpreted, the container can come up with no devices injected. The result is a stack in which two layers both consider a GPU optional, so a missing one produces silence at every level.

The consequence for debugging is that you must test the layers separately. Ollama is the wrong place to start.

The missing piece: the NVIDIA Container Toolkit

A container does not get GPU access from the driver on the host by itself. The host driver exposes device nodes and user-space libraries; something has to inject those into the container’s mount namespace at creation time. That something is the NVIDIA Container Toolkit, which installs a runtime hook that Docker calls when a container requests devices. Without it, the container has the kernel driver’s device files at best and none of the user-space CUDA libraries, which is exactly what “discovered no GPUs” looks like from inside.

Test that layer with no Ollama involved at all:

docker run --rm --gpus=all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

If that prints a device table, Docker’s GPU passthrough works and your problem is in Ollama’s configuration. If it errors — commonly with an nvidia-container-cli message about being unable to find libnvidia-ml.so, or with Docker reporting that it could not select a device driver with capabilities gpu — the toolkit is missing or not registered, and no amount of changing the Ollama command will help. Install it from NVIDIA’s container toolkit install guide, then run nvidia-ctk runtime configure --runtime=docker and restart the Docker daemon. The configure step is the one people skip: installing the package does not register the runtime in /etc/docker/daemon.json.

The flag, and why compose needs a different one

On the command line, --gpus=all is the request. In a Compose file it is not a valid key, and this is where a working docker run quietly becomes a CPU-only stack when somebody converts it. Compose expresses the same thing as a device reservation:

services:
  ollama:
    image: ollama/ollama
    ports: ["11434:11434"]
    volumes: ["ollama:/root/.ollama"]
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
volumes:
  ollama:

On some hosts, particularly NVIDIA’s Jetson images and older daemon configurations, the device-request path is not wired up and the toolkit’s runtime must be selected explicitly with --runtime=nvidia plus -e NVIDIA_VISIBLE_DEVICES=all -e NVIDIA_DRIVER_CAPABILITIES=compute,utility. Those environment variables are what the legacy hook reads; the newer CDI-based path sets them for you. If --gpus=all yields nothing and --runtime=nvidia works, that is the mechanism you are seeing, not a bug in Ollama.

When the toolkit is installed and it still cannot see it

• Compute capability below 5.0. Ollama’s CUDA builds target Maxwell and newer. A Kepler-era card is visible to nvidia-smi and still unusable, and the log will say the device was skipped rather than not found.
• CUDA_VISIBLE_DEVICES set to an empty string or a bad index. Set anywhere in the container environment, including inherited from the host through Compose, this hides every GPU from every CUDA process. An empty value is the trap: it looks unset and means “none”.
• A driver newer or older than the container’s CUDA runtime. The user-space CUDA runtime inside the image must be no newer than the host driver supports. A mismatch shows up as a driver-initialisation failure in debug logs rather than as an absent device.
• Rootless Docker or a restrictive seccomp profile can block the device injection while everything else works. Confirm with the plain nvidia-smi container above under the same daemon.

Once the GPU is visible, confirm it is actually being used rather than merely present: run a generation and check ollama ps, which reports how much of the model is resident on GPU versus CPU. A model that is half offloaded because it does not fit will still be slow, and that is a memory problem rather than a detection problem. For the wider picture of running inference in containers, see packaging an inference service in Docker.