Skip to content
Multigrid

Serving models on Kubernetes

GPU scheduling, autoscaling and container build decisions for model serving, at the level of the manifest that has to be correct.

Kubernetes was designed around two resources it can divide: CPU, which is a rate and can be handed out in thousandths, and memory, which is a quantity and can be handed out in bytes. A GPU is neither. It arrives through a device plugin as a whole, countable, non-overcommittable thing, and almost every surprising behaviour in this cluster — why a pod sits Pending next to an idle node, why a node with eight GPUs will not run a ninth pod, why scaling to zero is a different problem from scaling to one — follows from that difference.

These pages work at the level of the object you have to get right: the pod spec, the DaemonSet, the ConfigMap, the NodePool, the probe. Each one ends where you can confirm the thing worked, and then names what you will see when it did not.

Scheduling GPU Pods on Kubernetes

Request a GPU on a pod spec, confirm the scheduler placed it on a GPU node, and read the allocatable count that decides whether it could.

9 min read

Installing the NVIDIA Device Plugin for Kubernetes

Install the device plugin DaemonSet and confirm nvidia.com/gpu appears as an allocatable resource, including the runtime prerequisite that silently produces zero devices.

9 min read

Taints and Tolerations for a GPU Node Pool

Taint a GPU node pool so only GPU workloads land on it, and understand why a toleration alone does not attract a pod to those nodes.

9 min read

GPU Time-Slicing on Kubernetes

Configure the device plugin to advertise replicas of each GPU so more pods than physical GPUs can schedule, and understand what that does not give you.

9 min read

MIG Partitioning for Kubernetes GPU Nodes

Enable Multi-Instance GPU on a supported node and expose the partitions as separately schedulable resources, with the two strategies that decide what the resource is called.

10 min read

A Horizontal Pod Autoscaler on a Custom Inference Metric

Wire a custom metrics adapter so the HPA scales on queue depth instead of CPU, and read the algorithm that turns that number into a replica count.

10 min read

A KEDA ScaledObject for Queue-Driven Inference Pods

Scale inference pods from zero on a queue's message count, and understand the activation threshold that decides when zero becomes one.

10 min read

Scaling GPU Nodes to Zero With Karpenter

Configure a Karpenter NodePool for GPU instances that provisions on demand and consolidates back to nothing, with the disruption settings that decide how fast.

10 min read

Cluster Autoscaler and GPU Node Pools

Configure the cluster autoscaler against a GPU node group, including the tags that let it scale from zero and the GPU-specific utilisation threshold.

10 min read

Fixing "0/N Nodes Are Available" for a GPU Pod

Read the scheduler's FailedScheduling message clause by clause and work through the six causes that produce Insufficient nvidia.com/gpu.

10 min read

Readiness Probes for a Model-Serving Pod That Loads Slowly

Set a startup probe long enough for model weights to load, so readiness and liveness checks only begin once the server can answer them.

9 min read

Resource Requests and Limits for a GPU Inference Pod

Why a GPU has no fractional request the way CPU and memory do, and what follows from that for QoS class, quota and the CPU and memory you set alongside it.

9 min read

Running KServe for Model Serving on Kubernetes

Install the KServe controller, deploy an InferenceService, and make it add replicas as request volume rises.

10 min read

Deploying vLLM on Kubernetes

A vLLM Deployment and Service that survives a real cluster, ending with a verified OpenAI-compatible request.

10 min read

Fixing PodUnschedulable Because No Node Matches the GPU Selector

Read the FailedScheduling message clause by clause, compare your selector against the labels that exist, and fix the mismatch.

9 min read

Node Affinity for Pinning Inference Pods to a GPU Type

Write required node affinity that lands a pod on a named GPU SKU, with a preferred rule for the case where you have a second choice.

9 min read

Spot Node Pools for Kubernetes GPU Workloads

Create a tainted spot GPU pool on GKE, EKS or AKS so only interruption-tolerant work lands there, and handle the reclaim.

10 min read

A PriorityClass for Preempting Low-Priority Inference Pods

Define priority classes so a latency-sensitive request can evict a batch job from a full GPU node, and understand what preemption cannot do.

9 min read

Namespace Resource Quotas for a Shared GPU Cluster

Cap how many GPUs one namespace can request, and understand exactly which requests the quota rejects and when.

9 min read

A Helm Chart for Deploying a Model-Serving Service

Package a Deployment, Service and HPA into one chart with per-environment values files, and upgrade it without surprises.

10 min read

Draining a GPU Node Without Killing In-Flight Requests

Set a preStop hook, a grace period and a disruption budget so a node drain waits for the generation that is already streaming.

10 min read

Ingress and TLS for a Kubernetes-Hosted Model Endpoint

Put an ingress controller and a real certificate in front of an internal inference Service, without breaking streaming.

10 min read

Multi-GPU Node Bin Packing for Inference Pods

Why the default scheduler spreads GPU pods across nodes, and the scoring configuration that packs them onto as few nodes as possible.

9 min read

Fixing CrashLoopBackOff on a GPU Inference Pod

Read the container's exit code and last-state reason first, because it separates a memory kill from a CUDA driver mismatch before you change anything.

9 min read

A NetworkPolicy Restricting Egress From an Inference Pod to a Model Provider

Write an egress NetworkPolicy that lets an inference pod reach one provider's published IP ranges and nothing else, including the DNS rule everyone forgets.

10 min read

Multi-Stage Docker Builds for a Python Inference Service

Split a build stage from a runtime stage so compilers, headers and build caches never reach the image you ship to every node.

9 min read

Reducing the Size of a Model-Serving Docker Image

Measure where the bytes actually are with docker history before touching the base image, because on an inference image one layer is usually most of the total.

10 min read

Baking Model Weights Into an Image or Mounting Them at Runtime

The choice is between paying for weights at build and pull time or at container start time, and which is cheaper depends entirely on how often each happens.

9 min read

Fixing a Push Timeout on a Large AI Docker Image

Read which error you got, then check layer sizes against the registry's published limits and the proxy in front of it before blaming the network.

10 min read

Using the NVIDIA Container Toolkit With Docker

Install the toolkit, point the Docker daemon at the NVIDIA runtime, and confirm a container sees the host GPU with a working nvidia-smi.

9 min read

Docker Compose for a Local Inference Stack With a Vector Database

One compose file that runs a GPU inference server, a vector database and your application, with the start-order and GPU stanzas that actually work.

10 min read

Health Checks in a Dockerfile for a Model-Serving Container

Write a HEALTHCHECK that only reports healthy once weights are loaded, using start-period so a slow load is not counted as a failure.

9 min read

Layer Caching to Speed Up Rebuilds of an AI Docker Image

Why a one-line code change reinstalls the whole dependency tree, and the ordering and cache-mount rules that stop it.

9 min read

Fixing a Docker Build Context That Is Too Large for a Model Repo

The build sends your entire repository to the builder before running a single instruction; here is how to see what it sent and the .dockerignore that stops it.

9 min read

A Non-Root User in a Model-Serving Dockerfile

Adds an unprivileged user to an inference image and fixes the permission-denied errors that appear the moment model weights live on a mounted volume.

9 min read

Buildx and Multi-Platform Images for an Inference Service

Builds one inference image for amd64 and arm64 and pushes a single manifest, including the parts of a Python AI image that emulation makes unbearable.

9 min read

Distroless Base Images for a Model-Serving Container

What a distroless image actually removes, what that does to an inference container, and how you debug one when the shell you would have used is gone.

9 min read

AWS Lambda SnapStart for a Model-Calling Function

Enables SnapStart on a supported runtime, and covers the two things about a model-calling function that make a restored snapshot behave differently from a fresh init.

10 min read

Cloud Run Cold Starts and Container Image Size

Google states that image size does not affect Cloud Run container startup time, which is not what most advice assumes — here is the mechanism, and what does move the number.

9 min read

Azure Functions Premium Plan to Keep an AI Function Warm

Moves a model-calling function to an Elastic Premium plan and sets always-ready instances, with the CLI properties, the SKU table and what the plan does not fix.

9 min read

Cold Start Behaviour of Cloudflare Workers for AI Calls

Why the isolate model gives Workers a cold-start profile unlike any container platform, what Cloudflare changed in 2025, and which limit actually constrains an AI proxy.

9 min read

How Lambda Packaging Format Changes Cold Start Time

The documented limits, loading mechanism and billing rules that separate a zip-packaged Lambda from a container-image one, and the query that measures the difference on your own functions.

10 min read

Fargate Task Startup Time for On-Demand Inference

The lifecycle stages between RunTask and a serving container, which of them you can influence, and the API fields that tell you where the time actually went.

9 min read

What Changed in Cold Start Behaviour When Cloud Functions Moved to Gen2

Gen2 functions are Cloud Run services, and the change that matters most for cold starts is concurrency rather than anything about how fast an instance starts.

9 min read

Keeping a Vertex AI Endpoint Warm to Avoid Scale-to-Zero Latency

Sets a minimum replica count on a Vertex AI endpoint so the first request after idle does not pay a model load, and covers the scale-to-zero configuration if you want the opposite.

9 min read

Other topics