Running KServe for Model Serving on Kubernetes

KServe is a controller that turns one custom resource into a Deployment, a Service, an autoscaler and a route. The part worth understanding before you install it is that it can do this two different ways, and the way you pick decides whether scale-to-zero is available to you at all.

What KServe adds over a Deployment

You can serve a model with a Deployment, a Service and an ingress rule. People do. What you get for the extra controller is a single object that owns the whole set, a storage initialiser that pulls weights out of object storage into the pod before the server starts, a set of prebuilt runtimes so the container image is not your problem, and a canary field that shifts a percentage of traffic to a second revision without you writing two Deployments and an ingress weight.

The storage initialiser is the piece that most repays the dependency. Model weights are large, and the naive approach — baking them into the image — makes every image push and every node pull proportional to the model rather than to your code. KServe runs an init container that fetches storageUri into a shared volume, so the serving image stays small and the weights are a runtime concern. That is the same separation described in baking versus mounting model weights, implemented for you.

Pick a deployment mode before you install

KServe has two deployment modes and they are not interchangeable. Serverless mode builds on Knative Serving, which brings its own activator and autoscaler; RawDeployment mode emits plain Kubernetes objects — a Deployment, a Service, an Ingress or Gateway API route, and a HorizontalPodAutoscaler — and nothing else. The KServe documentation describes RawDeployment as the mode with minimal dependencies on Kubernetes resources, and that is precisely the trade: fewer moving parts, and no request-driven activation.

The consequence you feel immediately is scale-to-zero. Knative can hold a request at the activator while a pod comes up from zero replicas, which is what makes minReplicas: 0 usable. RawDeployment has nowhere to hold that request, so scaling to zero means the first caller gets a connection failure rather than a slow response. On a GPU cluster this matters more than it does elsewhere, because a GPU sitting idle at one replica is the single most expensive thing in the namespace.

Set the default at install time with the controller’s deploymentMode value, or override it per object with the serving.kserve.io/deploymentMode annotation on an individual InferenceService.

Installing the controller

KServe publishes OCI Helm charts, and the CRDs are a separate chart from the controller because CRDs must exist before anything that references them. Install them in that order.

# 1. CRDs first, in their own release.
helm install kserve-crd oci://ghcr.io/kserve/charts/kserve-crd \
  --namespace kserve --create-namespace \
  --version <pin the current chart version>

# 2. The controller. RawDeployment keeps Knative out of the picture.
helm install kserve oci://ghcr.io/kserve/charts/kserve \
  --namespace kserve \
  --version <pin the current chart version> \
  --set kserve.controller.deploymentMode=RawDeployment

kubectl get pods -n kserve -w

Deploying an InferenceService

The custom resource is InferenceService in the API group serving.kserve.io/v1beta1. A predictor names a model format, a storage location, and the resources one replica needs. Everything else has a default.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: summariser
  namespace: inference
spec:
  predictor:
    minReplicas: 1
    maxReplicas: 8
    model:
      modelFormat:
        name: pytorch
      storageUri: s3://models/summariser/v3
      resources:
        requests:
          cpu: "4"
          memory: 24Gi
          nvidia.com/gpu: "1"
        limits:
          cpu: "8"
          memory: 32Gi
          nvidia.com/gpu: "1"

Note that nvidia.com/gpu is identical in requests and limits. That is not style. Extended resources cannot be overcommitted, so Kubernetes requires the two to be equal; setting only a limit is accepted and silently copies it into requests. The full reasoning is in GPU requests and limits.

1. Apply the manifest and watch the object rather than the pod: kubectl get isvc summariser -n inference -w. The READY column goes true when the predictor’s pods pass their readiness probe.
2. Read the URL out of status.url. In RawDeployment mode that is the cluster-internal Service address unless you have wired an ingress class; in Serverless mode it is the Knative route.
3. Send a request in the runtime’s inference protocol — for a v2 predictor that is POST /v2/models/summariser/infer. A 404 here almost always means the model name in the path does not match the object name.

Making it scale on request volume

In Serverless mode the autoscaler is Knative’s, and the knob is the autoscaling.knative.dev/target annotation: a target concurrency per replica. KServe’s documentation is explicit that this is a soft limit and can be exceeded during a burst, because the autoscaler reacts to observed concurrency rather than admitting requests against a hard cap.

In RawDeployment mode the autoscaler is a HorizontalPodAutoscaler, and the fields are scaleTarget and scaleMetric on the predictor. The default metric is CPU, which is the wrong signal for GPU inference — a pod saturating its GPU can sit at modest CPU while queueing requests. If you stay in RawDeployment, plan on driving the HPA from a served metric such as queue depth or in-flight requests, which is the subject of scaling inference on a custom metric.

Whichever mode you are in, maxReplicas is a promise about hardware you may not have. Eight replicas each requesting one GPU need eight GPUs to exist or to be creatable by the cluster autoscaler; if neither is true the extra pods sit Pending and the autoscaler looks broken when it is not.

Where it breaks first

A note on debugging shape before the list, because it saves a great deal of time. An InferenceService is several objects deep: the InferenceService owns a Deployment or a Knative Service, which owns a ReplicaSet, which owns pods, and errors surface at whichever level noticed them. kubectl describe isvc shows the top-level conditions and is often unhelpfully summarised; the useful command is to walk down one level at a time until you find an object with a real event on it. Most of the time the answer is two levels below where you started looking.

• The pod is Running but never Ready. Weight loading takes longer than the readiness probe allows, so the kubelet keeps restarting a container that was going to work. Use a startup probe rather than a generous initialDelaySeconds; see readiness probes for slow model loads.
• The storage initialiser fails. Its logs are in a separate container: kubectl logs POD -c storage-initializer. The usual cause is credentials — the service account annotation that binds a cloud identity is on the namespace default service account and not on the one the InferenceService uses.
• The pod is Pending with no GPU node. KServe does not create nodes. Either a GPU node pool exists, or something like Karpenter or the cluster autoscaler creates one, or the pod waits forever.
• Two revisions, one GPU. A canary rollout wants both the old and new predictor resident simultaneously. On a single-GPU node the new revision is unschedulable and the rollout stalls without a clear error, because from the scheduler’s point of view nothing is wrong — there is simply no GPU.