GPU Memory Not Freed After a Local Model Crashes

Your model server died. You restart it and it fails immediately with CUDA error: out of memory. You run nvidia-smi and the memory column says something like 21504MiB / 24576MiB, while the processes table underneath says No running processes found. Nothing is running, and the memory is gone.

The symptom

The two halves of that output are produced by different queries and they are not inconsistent. The memory figure comes from the driver’s own accounting of allocations on the device. The process list comes from enumerating the compute processes the driver can attribute and that nvidia-smi is permitted to see. There are several ordinary situations in which the first is non-zero and the second is empty, and none of them means the memory has leaked irrecoverably.

Before anything else, ask the driver directly rather than reading the summary table:

nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv
nvidia-smi -q -d PIDS

Why the driver still holds it

Device memory is not owned by a process in the sense the process table means. It is owned by a CUDA context, and the context is anchored to open file descriptors on the driver’s character devices — /dev/nvidia0, /dev/nvidiactl, /dev/nvidia-uvm. The driver releases the allocations when the last descriptor referencing that context is closed, which the kernel does on process teardown. So the rule is simple and it explains every case below: the memory is still allocated because something still has the device open.

• A process stuck in uninterruptible sleep. If the crashing thread was inside a driver ioctl when it was killed, it sits in state D. SIGKILL is queued and not delivered until the call returns, the descriptors stay open, and the memory stays allocated. This is the most common case and it is why “I already killed it” and “the memory is still held” are both true.
• A different PID namespace. If the model ran in a container, the host sees a PID your shell cannot resolve, and nvidia-smi run inside a container generally cannot enumerate compute processes at all. The table looks empty from wherever you happen to be standing.
• A surviving child or a forked worker. A supervisor that died leaving a worker holding the context, or a Python dataloader subprocess, keeps the allocation alive under a name you are not looking for.
• The MPS control daemon. With CUDA MPS in use, the server process holds contexts on behalf of clients and outlives them.
• A fault below the process level. If dmesg shows an Xid error, the GPU hit a hardware or driver fault. That is not a stuck process, and no amount of killing will help.

Notice what is not on that list: a zombie process. A zombie has already been torn down and has no open descriptors; only its exit status remains in the table. If the offender were a zombie the memory would already be free.

Find the process nvidia-smi cannot see

Ask the kernel who has the device files open, which sidesteps the driver’s attribution entirely:

sudo fuser -v /dev/nvidia*
sudo lsof /dev/nvidia* 2>/dev/null

# for anything that turns up, check its state and where it is blocked
ps -o pid,ppid,stat,wchan:24,etime,cmd -p <PID>

Read the STAT column carefully, because it decides which remedy applies. S or R is an ordinary live process: terminate it and the memory returns. D is uninterruptible sleep: it cannot be killed until the driver call it is inside completes, and sending signals is wasted effort. Z means you are looking at the wrong process, per the paragraph above. If fuser lists a PID inside a container, act on it from the host or stop the container rather than the process.

Reclaim, in increasing order of violence

1. Ask politely first. A model server given SIGTERM unloads the model, frees the context and exits cleanly. Reaching for -9 immediately is what creates the stuck case in the first place, because SIGKILL cannot interrupt an in-flight driver call and removes any chance of an orderly teardown.

   kill -TERM <PID>; sleep 10; kill -0 <PID> 2>/dev/null && kill -KILL <PID>

2. Wait, if the state is D. Many driver calls do return, and the queued SIGKILL is delivered the moment they do. Give it a minute before escalating; if the state never changes, the fault is at the driver level and steps three and four apply.

3. Reset the device. This reinitialises the GPU’s state. It requires root, and it fails if anything still has the device open — including a display server, so on a machine with a monitor attached to the same card you will need to stop the graphical target first.

   sudo nvidia-smi --gpu-reset -i 0
   # equivalently: sudo nvidia-smi -r -i 0

4. Reload the kernel modules. Unloading and reloading the unified-memory module tears down every context on every GPU in the machine, so it is not a targeted fix; it is a reboot that skips the firmware. It only succeeds if nothing has the device open, which is the same precondition that made step three fail.

   sudo systemctl stop llama-server
   sudo rmmod nvidia_uvm && sudo modprobe nvidia_uvm
   nvidia-smi --query-gpu=memory.used --format=csv

5. Check dmesg before rebooting. If there is an Xid in the log, record it — the number identifies the class of fault, and a recurring one points at overheating, an unstable overclock, a marginal power supply or failing memory rather than at your software.

   sudo dmesg -T | grep -i -E 'xid|nvrm' | tail -20

Stopping it happening again

Most repeat occurrences come from how the server is stopped, not from how it crashed. Run it under a supervisor that sends SIGTERM and waits — systemd’s default KillSignal is exactly that, and TimeoutStopSec controls how long it waits before escalating. Raise that timeout above the time your model takes to unload, or systemd will supply the SIGKILL you were trying to avoid. The systemd unit page has a unit that gets this right.

Second, stop the crash. The overwhelmingly common cause of a local model dying mid-generation is running out of device memory in the first place, at the point where the KV cache grows past what was left after the weights loaded — which is a sizing problem, worked in the context and quantisation tradeoff. Leaving genuine headroom converts a crash into a slower answer.

Third, if you run in containers, make the container the unit of cleanup. Stopping the container reliably tears down its PID namespace and closes the descriptors; hunting the process from the host does the same job with more steps and more chances to kill the wrong thing.