Multi-GPU Node Bin Packing for Inference Pods

Eight single-GPU pods across four eight-GPU nodes leaves every node partly occupied and none of them removable. The scheduler is not malfunctioning; it is doing exactly what its default scoring strategy says. Changing that is a scheduler configuration, and there is one step past the obvious one.

Why the default spreads

Scheduling has two phases. Filtering eliminates nodes that cannot run the pod at all — insufficient resources, unmatched selectors, untolerated taints. Scoring ranks the survivors, and the pod goes to the highest scorer. Spreading is a scoring outcome, not a filtering one.

The dominant plugin here is NodeResourcesFit, and Kubernetes documents its default scoring strategy as LeastAllocated: it favours nodes with more resources still available. That is a sound default for general workloads. It maximises headroom for the next pod, it reduces the blast radius of losing a node, and it keeps noisy neighbours apart.

For GPU inference the same default is expensive. Nodes are the billing unit, not pods; a node with one GPU busy costs the same as a node with all eight busy. Spreading therefore guarantees you pay for the maximum number of nodes your pods can be spread across, and it guarantees the cluster autoscaler can never remove a node, because every node has something on it. A scale-down that never happens is the real cost of the default here.

What the scheduler actually scores

Scoring is a weighted sum across several plugins, each returning 0 to 100. NodeResourcesFit is one. NodeResourcesBalancedAllocation is another, and it rewards nodes where CPU and memory utilisation are close to each other. ImageLocality favours nodes that already have the image — which matters a great deal for multi-gigabyte inference images. Node affinity preferences, pod topology spread, and inter-pod affinity all contribute too.

This is why a single preference rarely decides placement on its own, and why changing the fit strategy is a change of tendency rather than a guarantee. If you need a hard guarantee that two pods land together or apart, that belongs in a filter — inter-pod affinity or a topology spread constraint with whenUnsatisfiable: DoNotSchedule — not in a score.

It is also worth being clear that scoring is per pod, in arrival order, with no lookahead. The scheduler places one pod at a time against the cluster as it exists at that instant; it never reconsiders an earlier placement to make a later pod fit. So even a perfectly configured bin-packing profile produces poor packing if pods arrive in an awkward order — four single-GPU pods created while four nodes were each already holding one pod will stay where they landed, and no amount of scoring moves them afterwards. Packing is a policy for new placements, not a defragmenter. Rebalancing an already-fragmented cluster needs something that evicts and reschedules, which is a different tool.

Configuring MostAllocated

NodeResourcesFit supports three strategies: LeastAllocated (the default), MostAllocated, which favours nodes with higher utilisation, and RequestedToCapacityRatio, which scores against a shape you define. Bin packing is MostAllocated, expressed in a KubeSchedulerConfiguration.

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
  - schedulerName: gpu-binpack-scheduler
    pluginConfig:
      - name: NodeResourcesFit
        args:
          scoringStrategy:
            type: MostAllocated
            resources:
              - name: nvidia.com/gpu
                weight: 10
              - name: cpu
                weight: 1
              - name: memory
                weight: 1
    plugins:
      score:
        enabled:
          - name: NodeResourcesFit
            weight: 5
        disabled:
          - name: NodeResourcesBalancedAllocation

Disabling NodeResourcesBalancedAllocation is deliberate. It rewards even utilisation across resource dimensions, which pulls against packing; leaving it enabled at its default weight while trying to bin-pack produces a scheduler that half does each.

The part that is easy to miss

Setting type: MostAllocated and stopping there packs your nodes by CPU and memory and does nothing at all about GPUs.

The Kubernetes scheduler configuration reference documents the default resource set for NodeResourcesFit scoring as cpu and memory, each with weight 1. Extended resources are not scored unless you list them. So a GPU pod requesting one device and modest CPU will be placed according to CPU and memory allocation — which on a fleet of identical GPU nodes is very nearly noise, and produces placements that look random because they effectively are.

Listing nvidia.com/gpu in resources is what makes the strategy see the dimension you care about, and giving it a disproportionate weight is what makes GPU occupancy dominate the other two. Once you specify resources at all you replace the defaults entirely, so CPU and memory must be re-listed if you still want them counted.

Bin packing also has a cost, and it is worth stating plainly. Packed nodes mean a single node failure takes more replicas with it, and preemption dynamics change because victims concentrate — which is sometimes helpful, as in preempting low-priority inference pods, and sometimes not. Pair packing with a topology spread constraint on the workloads that must survive losing a node, and let everything else pack.

A second thing packing does not fix is worth naming, because people reach for this setting expecting it. Packing GPU pods tightly does not make an underused GPU shared — each pod still holds whole devices, and eight pods on one eight-GPU node use eight GPUs exactly as eight pods on eight nodes did. What changes is the number of nodes you rent, not the number of devices you occupy. If the actual waste is that each pod uses a fraction of the card it holds, the answer is time-slicing or MIG partitioning, which change what the device plugin advertises, and no scheduler configuration substitutes for them.

Managed clusters and the second scheduler

On GKE, EKS and AKS the control plane is managed and you cannot hand the default kube-scheduler a configuration file. The supported route is to run a second scheduler as a Deployment in the cluster with your own configuration, and to name it on the pods that should use it:

spec:
  template:
    spec:
      schedulerName: gpu-binpack-scheduler
      containers:
        - name: server
          resources:
            limits:
              nvidia.com/gpu: "1"

Pods without schedulerName keep using the default scheduler, so this is an opt-in change rather than a cluster-wide one — which is the right blast radius for a scheduling policy change, and lets you compare the two on the same cluster. The cost is a second component with its own RBAC, its own leader election and its own upgrade cadence; run the same minor version as your control plane, because scheduler plugin arguments do change between releases.

If that is more machinery than the saving justifies, the alternative worth considering first is node-level consolidation. Karpenter’s consolidation actively moves pods off underutilised nodes and deletes them, which attacks the same waste from the other end and needs no scheduler configuration — scaling GPU nodes to zero with Karpenter covers that path.