A Docker Build for an AI Project That Takes 40 Minutes

A forty-minute Python ML image is almost always one of four things: the dependency layer is being invalidated on every source change, no package cache survives between builds, packages are compiling from source because no wheel matches your base image, or you are installing multi-gigabyte CUDA wheels you do not need. Three commands tell you which.

Find your own slow layer first

Do not optimise from a list. Measure, and the list becomes one item.

# 1. Per-step timings. BuildKit prints elapsed time for every step.
DOCKER_BUILDKIT=1 docker build --progress=plain --no-cache -t app . 2>&1 \
  | grep -E '^#[0-9]+ DONE' | sort -t' ' -k3 -rn | head

# 2. How big is the build context? Printed as "transferring context".
du -sh . ; cat .dockerignore 2>/dev/null || echo "NO .dockerignore"

# 3. Which layers are large, in the finished image
docker history --human --format '{{.Size}}\t{{.CreatedBy}}' app | head -20

Those three outputs partition the problem completely. A single step dominating the wall clock is a dependency or compilation problem. A large context transfer before any step runs is a missing .dockerignore. A large final image with fast steps is a packaging problem rather than a speed one. Record the numbers before you change anything, because that is what makes the improvement arguable rather than felt.

Layer order, which is most of it

Docker caches layers and invalidates every layer after the first one whose inputs changed. COPY . . before pip install means a one-character change to a source file discards the entire dependency install — the whole forty minutes, on every commit.

# Wrong: every source edit reinstalls every dependency
COPY . /app
RUN pip install -r requirements.txt

# Right: dependencies are cached until the requirements file changes
COPY requirements.txt /app/
RUN pip install -r requirements.txt
COPY . /app

This one change is frequently the whole fix, and it is the reason a build can be forty minutes on CI and twenty seconds locally: the local cache is warm and the CI cache is not. Two corollaries. Order your layers from least to most frequently changed — system packages, then Python dependencies, then application code. And copy only the dependency manifest in the first step; copying a directory that contains it invalidates on every file in that directory.

Wheels, and the CUDA question

A pip install that takes twenty minutes is usually compiling. Watch the output: Building wheel for X (pyproject.toml) means no pre-built wheel matched your platform and setuptools is invoking a compiler. The two usual causes:

• An Alpine base image. Alpine uses musl libc, and the manylinux wheels that the scientific Python ecosystem publishes are built for glibc. So Alpine gets no wheels for numpy, scipy, pandas, pyarrow or anything else with C extensions, and compiles all of them from source. A Debian-based python:3.12-slim is both faster to build and usually smaller in the end, which surprises people who chose Alpine for size.
• An architecture with fewer published wheels. Building an amd64 image on an arm64 machine under emulation, or targeting a platform the package does not publish for, has the same effect and is much slower besides. Check --platform against what you actually need.

The CUDA question is separate and is usually the largest single number in the image. The default PyTorch wheels bundle CUDA runtime libraries and are enormous; the CPU-only build is a fraction of the size. If the container serves an API that calls a hosted model, or runs a tokeniser, or does data preparation, it does not need CUDA at all.

# CPU-only, when the container will never touch a GPU
RUN pip install --no-cache-dir torch --index-url \
      https://download.pytorch.org/whl/cpu

# Verify what you actually pulled, and how big it is
python -c "import torch; print(torch.__version__, torch.cuda.is_available())"
du -sh /usr/local/lib/python3.12/site-packages/* | sort -h | tail -15

That last command is the honest way to answer “where did four gigabytes go”. Run it inside your own image and the answer is usually two or three directories, in order, with no ambiguity.

Cache mounts and the build context

A BuildKit cache mount keeps pip’s download cache between builds without putting it in a layer. Dependencies that do change then reinstall from local files rather than re-downloading.

# syntax=docker/dockerfile:1
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install -r requirements.txt

The syntax comment on the first line is required for the mount option to be understood. Note that the cache mount and --no-cache-dir are mutually exclusive strategies: use the mount when builds repeat on the same machine, and --no-cache-dir in a final stage where you want no cache in the layer at all.

Then the build context. Docker uploads the context to the daemon before the first instruction runs, so a repository containing a models/, data/ or .venv/ directory pays that transfer on every build, including builds that would otherwise be fully cached.

# .dockerignore
.git
.venv
__pycache__/
*.pyc
data/
models/
*.safetensors
*.gguf
*.ckpt
notebooks/
.pytest_cache/

Where the gigabytes are

Image size is arithmetic over layers, and it is worth doing rather than guessing. The recurring contributors:

• Bundled GPU runtime libraries inside deep-learning wheels. Usually the largest item by a wide margin when present, and usually removable entirely by choosing a CPU wheel index.
• Build toolchains left in the final image. A compiler installed to build one package and never removed. Fix with a multi-stage build: compile in a builder stage, copy only the installed packages into a slim runtime stage.
• Package manager caches. apt-get lists and pip’s cache directory, each a few hundred megabytes, both removable in the same RUN that created them — a separate RUN that deletes them saves nothing, because the earlier layer still contains them.
• Model weights baked into the image. Gigabytes that make every pull slow and every rebuild slower. Download at start-up into a mounted volume, or bake them into a separate rarely-rebuilt base image.

CI is a different problem from local

If the build is fast on your machine and slow in CI, layer ordering is not your problem — cache persistence is. A fresh CI runner has no layer cache at all, so every build is the cold build, and the cache mount from the previous section lives on a machine that no longer exists.

# Persist the layer cache in a registry between runs
docker buildx build \
  --cache-from type=registry,ref=registry.example.com/app:buildcache \
  --cache-to   type=registry,ref=registry.example.com/app:buildcache,mode=max \
  --push -t registry.example.com/app:$GIT_SHA .

mode=max exports intermediate layers as well as the final ones, which is what makes a partial cache hit useful; the default exports only the final image and helps much less. Several CI providers offer a local cache backend instead, which avoids registry round trips at the cost of a size limit.

• A base image pinned by tag, not by digest. If the tag moves upstream, every layer after the FROM invalidates, and it happens on a schedule you do not control. Pin by digest for reproducible cache behaviour as well as reproducible builds.
• Build arguments that change every run. A commit SHA or a timestamp passed as a build argument, used early in the file, invalidates everything after it. Put such arguments as late as possible.
• Emulated cross-architecture builds. Building arm64 on amd64 runners under QEMU is dramatically slower than native, and it is easy to enable accidentally through a multi-platform target. If you do not ship both architectures, build one.

A Dockerfile that does all of it

# syntax=docker/dockerfile:1
FROM python:3.12-slim AS builder

RUN apt-get update && apt-get install -y --no-install-recommends \
      build-essential ca-certificates \
 && rm -rf /var/lib/apt/lists/*

WORKDIR /app
COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install --prefix=/install -r requirements.txt

# ---- runtime: no compiler, no caches, no build tools ----
FROM python:3.12-slim
RUN apt-get update && apt-get install -y --no-install-recommends \
      ca-certificates \
 && rm -rf /var/lib/apt/lists/*

COPY --from=builder /install /usr/local
WORKDIR /app
COPY . .

ENV PYTHONUNBUFFERED=1 PYTHONDONTWRITEBYTECODE=1
RUN useradd -m app && chown -R app /app
USER app
CMD ["python", "-m", "myapp"]

Four things are doing the work there: the dependency layer is copied and installed before the source, the pip cache is a mount rather than a layer, the compiler exists only in the builder stage, and the apt lists are removed in the same instruction that created them. Rerun the three measurement commands from the first section afterwards — that comparison is your number, for your project, and it is worth more than anyone else’s.