Running Model Eval Tests on a GPU in CI

A GPU eval job in CI is two problems wearing one hat. Getting a GPU attached to a runner is the easy one and the one every guide covers. Deciding what number fails the build, on a suite whose score moves a little every run, is the one that decides whether the job survives its first month.

Getting a GPU into the job

There are two supply routes and they price completely differently. GitHub sells hosted larger runners with a GPU attached; its runner pricing reference lists a linux_4_core_gpu SKU, and larger runners of any kind are available only to organisations and enterprises on the Team or Enterprise Cloud plans. The alternative is a self-hosted runner on hardware you already have, or an ephemeral one you launch per job — which is the subject of the spot-instance runner page.

Either way the job selects it by label, and the label is the only thing your workflow file knows about the hardware. That is worth keeping in mind: a workflow that says runs-on: gpu will happily schedule onto a completely different GPU next quarter and report a different score. Pin the accelerator explicitly in the job itself, not just in the runner pool, and make the job assert what it got.

jobs:
  eval:
    runs-on: [self-hosted, linux, gpu]
    timeout-minutes: 30
    concurrency:
      group: gpu-eval-${{ github.ref }}
      cancel-in-progress: true
    container:
      image: nvcr.io/nvidia/pytorch:24.10-py3
      options: --gpus all --shm-size=8g
    steps:
      - uses: actions/checkout@v4

      - name: Assert the accelerator
        run: |
          nvidia-smi --query-gpu=name,memory.total,driver_version \
                     --format=csv,noheader
          python -c "import torch; assert torch.cuda.is_available()"

The --gpus all flag only works if the NVIDIA Container Toolkit is installed on the runner host and configured as a Docker runtime. On a self-hosted box that is a host-provisioning step, not something the workflow can fix; if the container starts but nvidia-smi is missing inside it, that is the failure you are looking at. --shm-size matters too: the default 64 MB of shared memory in a container is smaller than a PyTorch dataloader with several workers expects, and the symptom is a bus error rather than anything that mentions shared memory.

An eval harness that returns a number

A CI job can only act on an exit code, so the harness has to end in one. The structure that works is to keep scoring and gating in separate steps: the eval writes a machine-readable artifact, and a second, cheap step compares that artifact to a committed baseline. Fusing them is tempting and costs you the ability to re-score a historical run against a new threshold.

# eval/run.py — writes results.json, always exits 0
import json, pathlib

results = {
    "suite": "retrieval-qa-v3",
    "n": len(cases),
    "exact_match": correct / len(cases),
    "mean_latency_ms": total_ms / len(cases),
    "commit": os.environ["GITHUB_SHA"],
}
pathlib.Path("results.json").write_text(json.dumps(results, indent=2))

Emitting the case count alongside the score is not decoration. Half the confusing regressions in an eval suite are a loader that silently dropped rows, and a score computed over 180 cases instead of 200 looks exactly like a model regression until somebody notices n changed.

Failing the build on a threshold

1. Commit a baseline file — eval/baseline.json — with the score you are defending and the commit it came from. It is a reviewed artifact, so raising or lowering the bar is a pull request somebody looks at rather than an environment variable somebody edits.
2. Add a gate step that loads both files, computes the delta, and exits non-zero only when the delta is worse than the tolerance band.
3. Upload results.json with actions/upload-artifact on every run, including failures. A failed gate with no artifact tells the next person nothing.
4. Write the delta to $GITHUB_STEP_SUMMARY so the number is on the run page rather than buried in the log.

# eval/gate.py
import json, os, sys

got = json.load(open("results.json"))
want = json.load(open("eval/baseline.json"))
delta = got["exact_match"] - want["exact_match"]
band = want.get("tolerance", 0.01)

with open(os.environ["GITHUB_STEP_SUMMARY"], "a") as fh:
    fh.write(f"exact_match {got['exact_match']:.4f} "
             f"(baseline {want['exact_match']:.4f}, delta {delta:+.4f})\n")

if delta < -band:
    sys.exit(f"regression: {delta:+.4f} exceeds tolerance {band}")

Why a raw threshold flakes

The first version of this job almost always compares the score to a fixed number with no band, and it goes red on a commit that changed a README. Two independent sources of movement cause that.

The first is sampling. If the eval calls a model with any temperature above zero, the score is a draw from a distribution and a single run is a sample of one. Setting temperature to zero removes most of it, and where a provider supports a seed parameter, setting that removes more — but zero temperature is a greedy decode, not a promise of byte-identical output across a provider’s own fleet.

The second is the GPU. Floating-point reduction on a GPU is not associative, and kernels that use atomics or pick an algorithm based on available workspace can produce different last-bit results between runs on the same hardware. For a local model this is addressable: PyTorch exposes torch.use_deterministic_algorithms(True), and cuBLAS requires CUBLAS_WORKSPACE_CONFIG=:4096:8 in the environment for its own reductions to be reproducible. Determinism costs throughput, which is a fine trade in an eval job and a bad one in training.

Even with both handled, keep the band. A tolerance of one point on a 200-case suite is one case flipping, and one case flipping is noise you do not want paging a reviewer. What you actually want to catch is the five-point drop, and a band does not blunt that at all.

• Fix the suite, then fix the threshold. A band on a suite that is still growing is meaningless, because every new case moves the baseline.
• Store the baseline commit. When the gate fires, the first question is what changed between that commit and this one.
• Gate on the metric you would actually ship on. If latency is a release criterion, gate on the p95, not the mean; one slow case drags a mean past every request anyone experienced.

Keeping the job off the critical path

A GPU minute is one to two orders of magnitude dearer than a standard CI minute, and the per-minute arithmetic is worth working before you attach this to every push. Three structural choices keep it affordable without weakening the gate.

Run it on a path filter, so a docs-only change never touches a GPU. Use a concurrency group keyed on the ref with cancel-in-progress, so five pushes in ten minutes produce one eval rather than five. And cache the weights, because on a cold runner the download can easily be longer than the eval — caching model weights between CI runs covers the key design, which is subtler than it looks.

Finally, set timeout-minutes on the job. The default is six hours. A hung eval on a metered GPU runner is the single most expensive failure mode this job has, and it is one line to prevent.