Why an Idle GPU Node Pool Costs More Than It Looks Like

A GPU node pool with a minimum size of one, holding no workload, is running at least four meters. Only one of them is the number people quote, and the sum is consistently larger than the estimate that justified leaving the pool warm.

What the bill is made of

Take each meter in turn, because they belong to different services and appear on different lines.

• The instance. On both major providers a GPU node is billed as a virtual machine for as long as it is running, whether or not any pod is scheduled on it. Utilisation is not an input to the price.
• The control plane. Google documents a flat cluster management fee of $0.10 per cluster per hour for GKE, charged regardless of mode, cluster size or topology, with a monthly credit applied to that fee. Google’s GKE pricing page is the reference. Amazon charges a comparable per-cluster hourly fee for the EKS control plane.
• The boot disk. A GPU node image is large, and the disk is billed by provisioned size for the node’s whole lifetime. This is small per node and stops being small at fleet scale.
• Everything the node pulls. Logging and metrics agents on an idle node still ship data, and ingested log volume is billed by volume. An idle node is not a silent node.

The arithmetic on a one-node pool

Worked with AWS list prices so the numbers are checkable. AWS listed g5.xlarge — one NVIDIA A10G — at $1.006 per hour on-demand in us-east-1 on its EC2 on-demand pricing page at the time of writing, August 2026. AWS publishes the table here. Assumptions: 730 hours in an average month, one node, on-demand with no savings plan or reserved instance, a 200 GB gp3 root volume, and one dedicated cluster.

Instance
  730 h x $1.006/h                        = $734.38

EKS control plane, at the published $0.10/h
  730 h x $0.10/h                         =  $73.00

Root volume, gp3 at $0.08/GB-month, 200 GB
  200 x $0.08                             =  $16.00
                                          ---------
Idle monthly total, one node              = $823.38

Share that is the GPU instance line       =  89.2%
Share that is not                         =  10.8%

So the “the GPU is about seven hundred a month” estimate is roughly 11% low before anyone has run a single inference, and the missing 11% is entirely charges that do not scale down when the node idles. A three-node minimum triples the instance line while the control plane stays flat, which is the one piece of good news in the model: the per-node overhead falls as the pool grows, so small always-on pools are proportionally the worst deal.

Why it does not go to zero on its own

The obvious fix is to let the pool scale to zero, and both providers support that for GPU pools specifically. On GKE you set --min-nodes=0 with autoscaling enabled, and GKE applies a taint to GPU nodes so only pods requesting a GPU land there — which is what lets the pool empty cleanly.

gcloud container node-pools create gpu-pool \
  --cluster=my-cluster \
  --accelerator type=nvidia-tesla-t4,count=1 \
  --machine-type=n1-standard-8 \
  --num-nodes=0 --enable-autoscaling --min-nodes=0 --max-nodes=10

There is a distinction here that catches people. The GKE cluster autoscaler does not scale the whole Standard cluster to zero nodes, because system pods have to run somewhere — but an individual GPU node pool can go to zero while a small general-purpose pool carries the system workload. If you put system components on the GPU pool because it was the only pool, the GPU pool can never empty, and that is a cluster topology problem rather than an autoscaler setting.

The other half of scale-to-zero is scale-from-zero, and it is not free either. A cold GPU node has to be provisioned, boot a large image, have the driver installed or verified, and pull a container image that is frequently several gigabytes. Minutes, not seconds. Whether that latency is acceptable is the actual question behind keeping a minimum of one, and it should be answered with the $823 figure in hand.

What pins the node up

When a pool that should empty does not, the cause is nearly always a pod the autoscaler will not evict. The autoscaler is deliberately conservative, and it refuses to remove a node when any of the following hold.

• A pod not managed by a controller. A bare pod — no Deployment, no Job, no ReplicaSet — has nothing to recreate it, so the autoscaler will not delete the node under it. A debugging pod somebody left running last Thursday costs $1.006 an hour.
• Local storage. A pod with an emptyDir or hostPath volume is treated as unsafe to evict by default, because the data is not reproducible elsewhere.
• A restrictive PodDisruptionBudget. A budget that cannot be satisfied by moving the pod blocks the eviction, and therefore the scale-down, indefinitely.
• A DaemonSet with no exclusion. DaemonSets are normally ignored for scale-down purposes, but a pod that looks like a DaemonSet workload and is not one behaves like a bare pod.

The cluster autoscaler emits events explaining exactly which pod blocked a scale-down. Reading those events is faster than any amount of reasoning about the config, and it is the first thing to do when a pool is stuck at its floor.

What to check for your own cluster

1. Read the current on-demand rate for your exact instance type and region from the provider’s pricing page. GPU instance pricing varies more by region than general-purpose pricing does.
2. Add the control plane fee for the cluster, then divide it by the number of nodes to get the honest per-node overhead.
3. Check GPU utilisation over a fortnight rather than a day. A pool that is busy for two hours a weekday is idle about 88% of the time, and that ratio — not the peak — is what decides whether the minimum should be one or zero.
4. Measure your own cold-start time from an empty pool. If it is under the latency your workload tolerates, the case for a warm minimum evaporates.