GPU Power Draw While a Local Model Sits Idle but Loaded

A model sitting loaded and answering nothing is not free. It is also not drawing the card’s rated power. The number in between is one nobody publishes, so this page explains why the floor is raised, gives the arithmetic that turns whatever you measure into an annual cost, and gives you the command that produces the figure for your own card.

What vendors publish, and what they do not

NVIDIA publishes total graphics power for each board. The GeForce RTX 4090 product page lists 450 W alongside its 24 GB of GDDR6X memory, and that figure is a design limit for sustained full load — not a typical draw and definitely not an idle draw. NVIDIA publishes the board specification for each card in the series.

What is not published anywhere, by any vendor, is idle power — and certainly not idle power with a CUDA context alive and several gigabytes of VRAM allocated. That figure depends on the board partner, the VBIOS, the driver version, how many displays are attached and at what refresh rate, and whether the operating system has anything else touching the GPU. Any single number quoted for it is somebody’s measurement of their own machine, which is why this page gives you the command instead.

Why resident weights raise the floor

A GPU at rest wants to drop into a low power state: core clocks down, memory clocks down, some blocks gated off entirely. Two things about a loaded model interfere with that.

The first is that VRAM is DRAM, and DRAM contents survive only because they are refreshed. Allocated memory has to be maintained whether or not anything is reading it, and the refresh cost scales with how much is allocated rather than with how busy it is. A card holding 20 GiB of weights is doing strictly more work at rest than the same card holding nothing.

The second is the memory clock. Drivers are conservative about dropping the memory clock while a context has live allocations, because ramping it back up has a latency cost and can glitch anything reading from memory. The practical result is that a card with a model resident often sits in a higher memory power state than the same card at the desktop, and the difference is visible in nvidia-smi’s reported clocks.mem before you look at watts at all.

A third effect is not the model’s fault but gets blamed on it: a multi-monitor setup, or any single display above 120 Hz, already pushes many cards into a raised memory state at the desktop. If your idle number looks high, unplug the second monitor and measure again before concluding the model is responsible.

Measuring your own idle figure

Two measurements, subtracted. Take a baseline with nothing loaded, then load the model and leave it entirely alone. Sample for long enough to see the card settle — power readings are noisy and a single instantaneous sample is close to useless.

# Baseline: no model loaded, nothing else using the GPU.
nvidia-smi --query-gpu=power.draw,clocks.sm,clocks.mem,memory.used \
  --format=csv,noheader -l 1 | head -n 120 > baseline.csv

# Now start the server and let it finish its warmup pass, then idle it.
llama-server -m model.gguf -c 8192 -ngl all &
sleep 60
nvidia-smi --query-gpu=power.draw,clocks.sm,clocks.mem,memory.used \
  --format=csv,noheader -l 1 | head -n 120 > loaded.csv

# Compare the medians, not the means — a single sample spike drags a mean.

On AMD, rocm-smi --showpower --showmemuse is the equivalent, and on Apple Silicon there is no discrete GPU power rail to read — sudo powermetrics --samplers gpu_power gets you the closest thing. The subtraction is the same in all three cases: what the machine draws at rest, and what it draws at rest with weights resident.

Turning watts into an annual number

Once you have a figure in watts, the rest is arithmetic. Let P be the idle draw in watts and r the electricity rate in currency per kWh:

cost_per_day  = P / 1000 x 24 x r
cost_per_year = P / 1000 x 8760 x r

Worked, with P = 40 W and r = 0.18 (see the assumptions below):
  per day  = 0.040 kW x 24 h x 0.18 = 0.173
  per year = 0.040 kW x 8760 h x 0.18 = 63.07

And at P = 15 W, the same arithmetic gives 23.65 per year.

The 40 W and 15 W here are illustrative placeholders chosen to bracket a plausible range for a desktop card, not measurements — substitute the median from your own loaded.csv. The rate of 0.18 is a stated assumption; the US Energy Information Administration’s Short-Term Energy Outlook put the 2026 US residential average around 18 cents per kWh, and rates in Europe and the UK are materially higher. The full cost derivation, including generation works through the loaded case.

The conclusion that survives whatever number you plug in is that idle draw is a small annual figure and an ongoing one. It is not a reason to unload the model on a machine you use daily. It is a real reason to unload it on a machine that serves three requests a week, because there you are paying the residency cost continuously for a duty cycle close to zero.

The two levers that change it

The first is residency itself. Ollama unloads a model after a period of inactivity, controlled by OLLAMA_KEEP_ALIVE and by the keep_alive field on a request; setting it to 0 unloads immediately after each response and setting it to -1 keeps the model resident indefinitely. llama.cpp has no equivalent — the server holds the model until the process exits — so the unit of control there is the process, which is an argument for a socket- or timer-activated service rather than an always-running one.

The second is the power limit. nvidia-smi -pl sets a board power cap, and while it barely touches idle draw, it does cap the peak, which matters more for a thermally constrained machine than for the electricity bill. Persistence mode (nvidia-smi -pm 1) moves in the opposite direction: it keeps the driver loaded, which raises idle slightly and removes a chunk of cold-start cost. That is a latency-for-watts trade and worth making deliberately.