Self-Hosted GPU Runners for GitLab CI

Most GitLab Runner instructions you will find register with a registration token and set the executor in one command. That flow is deprecated, and the GPU part was never in the command anyway — it is a setting in config.toml that you add afterwards.

Registration tokens are not how this works now

GitLab replaced registration tokens with runner authentication tokens. You create the runner first in the GitLab UI or API, which produces a token prefixed glrt-, and then pass that to gitlab-runner register as --token. GitLab documents the registration token flow and several accompanying registration arguments as deprecated and scheduled for removal in GitLab 20.0.

The practical differences matter more than the ceremony. Under the new flow the runner’s tags, its untagged-job setting and its access scope are set when you create the runner, not when you register it — so --tag-list and --run-untagged on the register command no longer control what you think they control. Set them in the creation step, and treat registration as attaching a machine to a runner that already has a configured identity.

Registering the runner

Prepare the host first, and in this specific order — the toolkit is useless without the driver, and the Docker executor is useless without the toolkit. GitLab’s GPU documentation lists installing the NVIDIA driver and the NVIDIA Container Toolkit as prerequisites for the Docker executor.

1. Install the NVIDIA driver and verify with nvidia-smi on the host.
2. Install the NVIDIA Container Toolkit and verify that a container can see the GPU, independently of GitLab: docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi. If this fails, nothing downstream will work and the error will be far clearer here than in a job log.
3. Create the runner in GitLab with its tags and its untagged setting, and copy the glrt- token.
4. Register the machine against it.

gitlab-runner register \
  --non-interactive \
  --url "https://gitlab.com/" \
  --token "$GLRT_TOKEN" \
  --executor "docker" \
  --docker-image "nvidia/cuda:12.4.0-runtime-ubuntu22.04" \
  --description "gpu-a10-01"

The config.toml setting that exposes the GPU

Registration alone gives you a Docker executor with no access to the GPU. GitLab’s documentation is specific: use the gpus or service_gpus options in the [runners.docker] section.

[[runners]]
  name = "gpu-a10-01"
  url = "https://gitlab.com/"
  token = "glrt-..."
  executor = "docker"

  [runners.docker]
    image = "nvidia/cuda:12.4.0-runtime-ubuntu22.04"
    gpus = "all"
    # service_gpus = "all"   # only if your services containers need a GPU
    privileged = false
    volumes = ["/cache", "/mnt/model-cache:/mnt/model-cache"]

gpus = "all" maps to Docker’s own --gpus flag, so the same syntax applies: "all", or a device selection such as "device=0" to pin the runner to one GPU on a multi-GPU host. Pinning is how you run two runners on one machine without them fighting — give each a different device rather than letting both see everything.

service_gpus is separate and usually should stay off. It grants GPU access to service containers — the databases and helpers GitLab starts alongside your job — and there is rarely a reason for a Postgres service to hold a GPU while your job waits for one.

Edit config.toml and the runner picks it up; the file is watched, so a restart is not strictly required, but restarting after a GPU change is worth the certainty. If the shell executor is used instead of Docker, GitLab documents that no runner configuration is needed at all — the job inherits the host’s GPU access directly, along with every hygiene problem that implies.

Two jobs fighting over one GPU

Nothing in the configuration so far stops GitLab running two jobs on this machine at once, and gpus = "all" means both of them get the whole device. There is no memory isolation between them. The second job’s allocation fails with a CUDA out-of-memory error that looks exactly like a bug in the model code, on a machine that has plenty of free GPU memory whenever you check it afterwards.

Three settings control this and they operate at different levels. GitLab documents concurrent as a global cap on how many jobs run at once across all registered runners, with each [[runners]] section able to set its own limit beneath it — where limit is how many jobs that one registered runner handles concurrently, and 0, the default, means no limit. A third, request_concurrency, caps concurrent requests for new jobs from GitLab and defaults to 1.

For a single-GPU host the answer is simply limit = 1. Do not rely on the global concurrent value to enforce it: that is a cap across every runner on the box, so adding an unrelated CPU runner later silently changes the behaviour of your GPU one.

concurrent = 3

[[runners]]
  name = "gpu-a10-01-dev0"
  executor = "docker"
  limit = 1

  [runners.docker]
    gpus = "device=0"
    shm_size = 2147483648
    pull_policy = "if-not-present"

[[runners]]
  name = "gpu-a10-01-dev1"
  executor = "docker"
  limit = 1

  [runners.docker]
    gpus = "device=1"
    shm_size = 2147483648
    pull_policy = "if-not-present"

That is the multi-GPU pattern: one registered runner per physical device, each pinned with gpus = "device=N" and each limited to one job. Two jobs then run genuinely in parallel on separate hardware instead of colliding on the same one.

Two of the other options above are not decoration. pull_policy takes never, if-not-present or always, and defaults to always — which on a CUDA image means re-pulling several gigabytes at the start of every job. Setting if-not-present is the single biggest speed-up available here, with a real trade: a cached image is reused without re-authenticating, so on a shared runner a job can end up using a private image it could not have pulled itself. Weigh that against the minutes.

shm_size sets the container’s shared memory, and Docker’s 64 MB default is far too small for PyTorch data loaders with multiple workers. The failure is a worker process dying with a bus error partway through a run, which reads as a flaky test rather than a configuration problem. Raise it once and the class of bug disappears.

Finally, concurrency across pipelines rather than within a runner. If two jobs write to the same weight cache directory, limiting each runner to one job does not help — they are different runners. Use resource_group in the job definition, which makes GitLab run jobs in that group one at a time across the whole project. It is the right tool for anything with a shared mutable directory behind it.

The Kubernetes executor is different

On the Kubernetes executor there is no gpus option, because GPU access is a pod resource request rather than a Docker flag. GitLab documents configuring it through a pod spec patch, which requires the FF_USE_ADVANCED_POD_SPEC_CONFIGURATION feature flag to be enabled.

[[runners.kubernetes.pod_spec]]
  name = "gpu"
  patch = '''
    containers:
    - name: build
      resources:
        requests:
          nvidia.com/gpu: 1
        limits:
          nvidia.com/gpu: 1
  '''
  patch_type = "strategic"

The container name must be build — that is the job container in a GitLab Runner pod, and patching a name that does not exist silently does nothing. You also need the node selector to place the pod on a GPU-capable node, and the cluster needs the NVIDIA device plugin running for nvidia.com/gpu to be a schedulable resource at all. If the node pool itself is not yet in place, provisioning a GKE GPU node pool covers the driver installation side.

Tags, and the untagged-job trap

GitLab’s model is the inverse of the GitHub one and the difference is the thing to internalise. A GitLab job with no tags key can run on any runner that is configured to accept untagged jobs. So a GPU runner with the untagged setting enabled will pick up every job in the project — including the ones that just run a linter.

Turn it off. A GPU runner should accept tagged jobs only, and every job that needs it should say so:

model-eval:
  stage: test
  tags:
    - gpu
    - cuda-12
  image: nvidia/cuda:12.4.0-runtime-ubuntu22.04
  timeout: 45m
  rules:
    - changes:
        - models/**/*
        - eval/**/*
  before_script:
    - nvidia-smi
  script:
    - pip install --no-cache-dir -r eval/requirements.txt
    - python -m eval.run --device cuda --report eval-report.json
  artifacts:
    when: always
    paths:
      - eval-report.json

A job whose tags no runner satisfies stays stuck rather than failing, with a message that no runners match. That is the correct failure — much better than the alternative, where a GPU job silently runs on a CPU runner and fails deep inside CUDA initialisation, or worse, succeeds slowly on a fallback path.

The rules: changes block and the explicit timeout earn their place for the same reason they do on GitHub Actions: GPU minutes are the expensive minutes, and a hung job holds the machine. What a GPU CI runner costs per minute is the number that makes both of those feel worth the effort.