Using the NVIDIA Container Toolkit With Docker

A container cannot contain a GPU driver. The driver is a kernel module and the container shares the host kernel, so what has to happen is that the host’s user-space driver libraries and device nodes get injected into the container at start. The NVIDIA Container Toolkit is what does the injecting.

What the toolkit actually does

When you run a container with --gpus all, a hook runs before the container’s process starts. It bind-mounts the device nodes under /dev/nvidia* into the container, injects the matching user-space driver libraries — libcuda.so, libnvidia-ml.so and their companions — and puts the nvidia-smi binary in place. The container image supplies the CUDA runtime; the host supplies the driver.

That split explains nearly every confusing symptom in this area. It is why installing a driver inside your image is wrong and will conflict with what is injected. It is why an image built against a newer CUDA runtime than the host driver supports fails with CUDA driver version is insufficient for CUDA runtime version. And it is why the same image runs on one machine and not another with no change to the image at all.

Prerequisites

• An NVIDIA driver on the host, installed and loaded. Run nvidia-smi on the host, not in a container. If it does not print a table there, nothing below will help — fix the host driver first.
• A container engine. Docker Engine on Linux. The toolkit also configures containerd and CRI-O, and on Kubernetes you generally want the device plugin or the GPU Operator to do this for you rather than configuring nodes by hand.
• Not Docker Desktop on macOS. There is no NVIDIA GPU to pass through. On Windows this works through WSL2 with an NVIDIA driver installed on the Windows side rather than inside the distribution.

Note the driver version and the maximum CUDA version printed in the nvidia-smi header before you go further. That number is the ceiling on what CUDA runtime your images may use, and knowing it now saves a debugging session later.

Install the toolkit

NVIDIA’s installation guide documents adding the libnvidia-container repository and then installing the packages. On a Debian or Ubuntu host:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

NVIDIA’s guide pins an explicit version on the install line, and on any host you will rebuild you should do the same — the documented example uses a NVIDIA_CONTAINER_TOOLKIT_VERSION variable applied to nvidia-container-toolkit, nvidia-container-toolkit-base, libnvidia-container-tools and libnvidia-container1 together. Pinning some of those four and not others is a supported way to produce a broken install, because they are versioned in lockstep.

Configure the Docker runtime

Installing the packages does not wire them into Docker. The nvidia-ctk command edits /etc/docker/daemon.json to register the NVIDIA runtime, and the daemon must be restarted to read it:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Rootless Docker needs a different target and two extra steps: point nvidia-ctk runtime configure at the user’s own daemon.json with --config, restart with systemctl --user restart docker, and set nvidia-container-cli.no-cgroups in /etc/nvidia-container-runtime/config.toml, which NVIDIA documents doing with sudo nvidia-ctk config --set ... --in-place. Skipping the cgroups setting in rootless mode is the usual cause of a container that fails to start with a cgroup permission error.

Verify with nvidia-smi

1. Confirm Docker knows about the runtime: docker info | grep -i runtimes. The output should list nvidia alongside runc. If it does not, the daemon did not restart or the configure command edited a different file.
2. Run the check NVIDIA documents, which is deliberately a plain base image with no CUDA in it: sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi. The binary is injected by the toolkit, which is exactly what makes this a valid test — if it prints a table, the injection works.
3. Read the table. It must show the same GPUs, the same driver version and the same memory totals as the host. A container that sees fewer GPUs than expected usually has a narrower --gpus argument than all, or a CUDA_VISIBLE_DEVICES set in the image.
4. Test with a CUDA runtime present, since nvidia-smi alone does not prove the runtime works: docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi, then your own image running your framework’s device check.
5. Restrict devices when you want to: --gpus '"device=0,1"' exposes only those indices. The quoting is awkward and the error when you get it wrong is unhelpful, so copy it exactly.

When nvidia-smi works but your framework does not

This combination is common and it narrows the problem usefully: injection is working, so the toolkit is installed correctly and the fault is in the image.

• Runtime newer than driver. CUDA driver version is insufficient for CUDA runtime version means the image’s CUDA is beyond the ceiling in the nvidia-smi header. Rebuild on an older CUDA tag, or update the host driver.
• A CPU-only framework build. If torch.cuda.is_available() is False while nvidia-smi works, check torch.version.cuda. If it is None, pip installed the CPU wheel — a genuinely common outcome when a requirements file pins a version whose CUDA build is not on the default index.
• Missing shared memory. Multi-process data loaders and tensor-parallel inference use shared memory, and Docker’s default /dev/shm is small. vLLM’s documentation directs you to run with --ipc=host or to raise --shm-size for this reason. The failure is a bus error or a hang, not a CUDA message.
• The flag was dropped. Compose files, CI wrappers and systemd units all lose --gpus easily. Inside the container, ls /dev/nvidia* returning nothing tells you the hook never ran.