Running Ollama in Docker

The container is the easy part. What separates a working Ollama container from one that quietly runs on the CPU is a runtime shim on the host that has nothing to do with Docker’s own configuration, and a volume decision that costs you a second download if you get it wrong.

The image, the port and the volume

The official image is ollama/ollama. It has three moving parts worth naming before you run it.

• Port 11434. The server listens there inside the container and the image already binds it to all interfaces, so -p 11434:11434 is all the publishing you need. You do not set OLLAMA_HOST to 0.0.0.0 for this — that variable is about which interface the process binds inside its own namespace, and the image handles it.
• /root/.ollama. Everything durable lives here: the model blobs, the manifests, and the keypair the server generates on first start. Mount it or lose all three when the container is replaced.
• Tags. latest is the CPU and NVIDIA image; rocm is the AMD one, and it is a separate image rather than a flag because the runtime libraries differ.

Making --gpus actually reach the card

--gpus=all is not self-contained. It is a request to a container runtime hook that has to be installed and wired into Docker separately, and if it is absent Docker will either error on the flag or — depending on version — start a container with no devices in it. That hook is the NVIDIA Container Toolkit. Per the image’s own documentation on Docker Hub, the two commands that matter after installing the package are:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

The first writes the NVIDIA runtime into /etc/docker/daemon.json; the second is what makes the daemon read it. Skipping the restart is the single most common reason --gpus=all appears to be ignored immediately after a successful toolkit install.

For AMD the shape is different: no toolkit, but the two device nodes have to be passed in explicitly with --device /dev/kfd --device /dev/dri alongside the ollama/ollama:rocm tag.

On a multi-GPU host, --gpus=all is usually not what you want, because it hands every card to one container and a second container started later will fight it for memory. Narrow it at the Docker layer with --gpus '"device=0,1"', which is preferable to narrowing it inside the container with CUDA_VISIBLE_DEVICES: the Docker form means the other devices are genuinely absent from the container’s namespace rather than merely hidden from one process. Ollama’s GPU documentation notes that numeric device IDs may be ordered differently than you expect and that UUIDs are more reliable, which matters exactly when you are splitting cards between containers and the assignment silently swaps after a reboot.

Bringing it up

1. Verify the host can see the card outside Docker: nvidia-smi. If this fails, nothing below will work.
2. Verify the toolkit is wired in, using a throwaway container rather than Ollama, so a failure here is unambiguous:

   docker run --rm --gpus=all ubuntu nvidia-smi

   A device table means the hook is working. An error mentioning an unknown runtime or missing device means it is not, and you go back to the previous section.
3. Start the server with a named volume so the blobs survive:

   docker run -d --gpus=all \
     -v ollama:/root/.ollama \
     -p 11434:11434 \
     --name ollama ollama/ollama

4. Pull a model into the running container: docker exec -it ollama ollama pull llama3.2:3b. The pull happens inside the container and lands in the volume.
5. Serve a request from the host, against the API rather than the CLI, because the API is what you are actually deploying:

   curl http://localhost:11434/api/generate -d '{
     "model": "llama3.2:3b",
     "prompt": "Reply with the single word: ok",
     "stream": false
   }'

6. Confirm the card is carrying the model rather than the CPU: docker exec -it ollama ollama ps. The PROCESSOR column should read 100% GPU for a model this size. Anything with CPU in it means the layers did not all fit or the devices did not arrive.

Where the blobs live

A named volume, as above, is the right default: Docker owns the lifecycle, the permissions are correct without you thinking about them, and docker rm does not destroy the models. The alternative is a bind mount of a host directory, which you want when the models are already on disk from a host install and you would rather not download twenty gigabytes again.

The trap in the bind-mount case is ownership. The host’s Ollama service typically owns its store as the ollama user, and the container runs as root, so the container writing to that directory leaves files the host service cannot read afterwards. If you are going to share a store between a host install and a container, decide which one writes to it and make the other read-only, rather than discovering the split later as a corrupted-looking manifest.

Layout inside the mount is the same as any other install — content-addressed blobs plus manifests — and is worth understanding before you go pruning it by hand; where Ollama stores models covers it. Note also that the keypair in that directory is the server’s identity for pushing to a registry, so a volume you throw away is a new identity next time.

Configuration and compose

Every setting is an environment variable, which is what makes this image pleasant in a compose file. The ones you are most likely to want are the keep-alive duration, the number of models resident at once, and the default context length:

services:
  ollama:
    image: ollama/ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    environment:
      OLLAMA_KEEP_ALIVE: "30m"
      OLLAMA_MAX_LOADED_MODELS: "2"
      OLLAMA_CONTEXT_LENGTH: "8192"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
volumes:
  ollama:

The deploy.resources.reservations.devices block is compose’s equivalent of --gpus=all, and it depends on exactly the same host-side toolkit. If the plain docker run above worked, this will; if it did not, this will not either, and the compose syntax is not the thing to debug.

Two container-specific gotchas are worth carrying away. First, model loading is slow enough that a naive healthcheck against a generation endpoint will mark the container unhealthy during a cold start; check /api/tags, which answers immediately, instead. Second, an image with no model in the volume is a container that answers the API and 404s every generation, so a first run needs a pull step — either the docker exec above or an init container that performs it. General container packaging concerns beyond this are covered in containerising an AI service.