Adding a GPU Runner to a GitHub Actions Workflow

Attaching a GPU runner is fifteen minutes of work. The two things that go wrong afterwards — every unrelated job scheduling onto your expensive machine, and a runner that slowly stops being reproducible — are both decided by choices you make in those fifteen minutes.

Decide this before you register anything

GitHub’s documentation is direct about the risk: it recommends using self-hosted runners only with private repositories, because forks of a public repository can potentially run dangerous code on the runner machine by creating a pull request that executes code in a workflow.

That is not a caveat to note and move past. A self-hosted runner is a persistent machine with your network position and, in the GPU case, probably a cache of model weights and possibly a provider API key in its environment. On a public repository, a pull request is an arbitrary code execution primitive against it. If the repository is public, either do not do this, or run every job in a single-use machine that is destroyed afterwards — which is what an ephemeral spot GPU runner is for.

The alternative worth pricing first: GitHub announced GPU-enabled hosted runners as generally available in a July 2024 changelog entry, available on Team and Enterprise plans as part of its larger runners. If a hosted GPU runner covers your job, it removes this entire page. GitHub’s changelog entry is the announcement; current specifications and pricing are on the larger-runners documentation.

Registering the runner

Prepare the machine first: NVIDIA driver, container toolkit if jobs run in containers, and a non-root user that owns the runner. Confirm nvidia-smi works as that user before involving GitHub at all — debugging a driver problem through a workflow log is miserable.

# As the runner user, in its own directory.
mkdir -p ~/actions-runner && cd ~/actions-runner
curl -o runner.tar.gz -L https://github.com/actions/runner/releases/download/vX.Y.Z/actions-runner-linux-x64-X.Y.Z.tar.gz
tar xzf runner.tar.gz

./config.sh \
  --url https://github.com/acme/inference \
  --token "$RUNNER_TOKEN" \
  --name gpu-a10-01 \
  --labels gpu,cuda-12,a10 \
  --runnergroup gpu-runners \
  --unattended \
  --replace

sudo ./svc.sh install "$(whoami)"
sudo ./svc.sh start

The registration token is short-lived — GitHub documents it as expiring after one hour — so generate it at the moment you register rather than storing it. --unattended suppresses the interactive prompts and makes --url and --token mandatory, which is what you want in a provisioning script. --replace lets a rebuilt machine take over an existing runner name instead of accumulating dead entries.

--runnergroup is worth using from the start. Runner groups control which repositories and workflows may reach a runner, and moving an existing runner into a group later is a change nobody remembers to make. On an organisation account, a GPU runner should be in a group with an explicit repository allowlist.

If your jobs should never leave residue between runs, add --ephemeral: the runner accepts exactly one job and then deregisters. It is the single most effective hardening step available here, and it requires something to bring a replacement up.

Labels are an AND match

This is the mechanism that catches people. When runs-on is given an array of labels, GitHub queues the job on runners that have all of the labels specified — it is a conjunction, not a disjunction. And the runner already carries default labels: GitHub documents the defaults on a Linux x64 runner as self-hosted, Linux and X64, with --labels adding to them rather than replacing them.

The consequence is the mistake. A team registers one GPU runner, writes runs-on: self-hosted in a workflow, and every job in the repository — lint, unit tests, docs builds — now queues behind the GPU machine, because it is the only self-hosted runner and it matches that single label. The GPU sits idle running eslint while the job that needs it waits.

# Matches only runners with ALL FOUR labels.
runs-on: [self-hosted, linux, x64, gpu]

# Runner group plus label, for organisation-level runners.
runs-on:
  group: gpu-runners
  labels: [gpu, cuda-12]

Label the capability, not the machine. gpu is a weak label because it says nothing about which GPU; cuda-12 and a10 let a job that needs 24 GB of VRAM avoid a runner that has 16. Then add the specific label to the job that needs it and leave everything else on hosted runners.

The workflow step

name: model-eval

on:
  pull_request:
    paths:
      - "models/**"
      - "eval/**"

jobs:
  eval:
    runs-on: [self-hosted, linux, x64, gpu, cuda-12]
    timeout-minutes: 45

    steps:
      - uses: actions/checkout@v4

      - name: Confirm the GPU is visible
        run: nvidia-smi

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install
        run: pip install --no-cache-dir -r eval/requirements.txt

      - name: Run evaluation
        env:
          HF_HOME: /mnt/cache/huggingface
        run: python -m eval.run --device cuda --report eval-report.json

      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: eval-report
          path: eval-report.json

Three deliberate choices. The paths filter means the GPU job does not run on documentation changes, which is the cheapest optimisation available. timeout-minutes is set explicitly because a hung CUDA process on a self-hosted runner blocks every subsequent job on that machine, not just its own. And nvidia-smi runs as an early step so that a driver problem fails in four seconds with an obvious message rather than forty minutes later inside a training loop.

Pointing HF_HOME at a persistent path is the one place statefulness is an advantage — see caching model weights in CI for how to do that without the cache growing without bound.

State, and why the runner rots

A hosted runner is a fresh machine every time. A self-hosted one is not, and everything that follows from that is the ongoing cost of this setup.

• The workspace persists. Files left by a previous job are visible to the next one. A build that passes only because an artefact from the last run is still on disk is the classic self-hosted failure, and it appears as an unreproducible failure the day the disk is cleaned.
• The disk fills. Docker layers, pip caches and model weights are all large. A GPU runner that stops working for no apparent reason is usually out of disk. Prune on a schedule, not when it breaks.
• The driver and CUDA drift. An unattended upgrade that bumps the NVIDIA driver can break every job. Pin the driver package or accept that upgrades are a change requiring a test run.
• Nothing is monitored by default. A runner that stopped is a queue that never drains, and GitHub will show jobs queued rather than failed. Alert on queue time, which is the only signal that distinguishes a dead runner from a busy one.

Each of these is an argument for ephemeral runners, and each is a reason people do not use them: a fresh machine re-downloads weights and re-pulls a multi-gigabyte CUDA image every job. That trade — spend minutes per job or accept drift — is the real decision, and it is the same one GitLab CI GPU runners present in a slightly different form.