A Helm Chart for Deploying a Model-Serving Service

Three environments, three copies of the same YAML, and one of them has the wrong GPU count. A chart replaces the copies with one template and three small values files — and for model serving there are two template details that a generic web-service chart does not need.

The chart layout

charts/model-serving/
  Chart.yaml
  values.yaml
  values-dev.yaml
  values-staging.yaml
  values-prod.yaml
  templates/
    _helpers.tpl
    deployment.yaml
    service.yaml
    hpa.yaml
    configmap.yaml
    NOTES.txt

# Chart.yaml
apiVersion: v2
name: model-serving
description: An OpenAI-compatible inference server on GPU nodes
type: application
version: 0.4.2
appVersion: "1.9.0"

Two versions, two meanings, and conflating them causes real confusion six months in. version is the chart’s own version and must be bumped whenever a template changes; appVersion is the default version of the thing being deployed and is what you would use as the default image tag. A chart change with no image change bumps only the first.

_helpers.tpl is where the name and label templates live, and separating two of them is not cosmetic. spec.selector.matchLabels on a Deployment is immutable after creation, so if the selector is computed from the same helper as the full label set, adding a version label later makes the next helm upgrade fail with an error about an immutable field. The only recovery is to delete and recreate the Deployment, which on a GPU service means every replica goes away at once and then queues for devices on the way back. Emit a minimal, permanent selectorLabels helper and a larger labels helper that includes it, and the version label can move freely for the life of the chart.

Values, and what belongs in them

# values.yaml — defaults, overridden per environment
replicaCount: 1

image:
  repository: registry.example.com/inference
  tag: ""            # falls back to .Chart.AppVersion
  pullPolicy: IfNotPresent

model:
  id: mistralai/Mistral-7B-Instruct-v0.3
  maxModelLen: 8192
  gpuMemoryUtilization: 0.90

resources:
  limits:
    cpu: "8"
    memory: 32Gi
    nvidia.com/gpu: 1

nodeSelector: {}
tolerations: []
affinity: {}

autoscaling:
  enabled: false
  minReplicas: 1
  maxReplicas: 4
  targetCPUUtilizationPercentage: 70

service:
  type: ClusterIP
  port: 80

terminationGracePeriodSeconds: 120

The rule for what goes in values.yaml is: anything that differs between environments, and nothing else. A field that is identical everywhere belongs in the template, where it cannot be set wrong. Every value you expose is a value somebody can misconfigure at three in the morning, and a chart with ninety knobs is not more flexible than one with twelve — it is less reviewable.

nvidia.com/gpu deserves particular care. Expose it as a plain integer under resources.limits and never as a boolean like gpu.enabled, because the count interacts with the server’s tensor-parallel setting and a chart that can produce a two-GPU node allocation with a one-GPU server flag will start and then waste half the hardware silently.

What must not go in a values file is a credential. Values files are committed, rendered into release history, and readable by anyone with helm get values on the release. A Hugging Face token or a provider API key belongs in a Secret managed outside the chart, referenced by name — the chart takes existingSecret: hf-token-secret and templates a secretKeyRef, and never sees the value. That also keeps rotation independent of deployment: replacing the Secret and restarting is a smaller operation than cutting a chart release.

The three templates

{{/* templates/deployment.yaml */}}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "model-serving.fullname" . }}
  labels: {{- include "model-serving.labels" . | nindent 4 }}
spec:
  {{- if not .Values.autoscaling.enabled }}
  replicas: {{ .Values.replicaCount }}
  {{- end }}
  selector:
    matchLabels: {{- include "model-serving.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
      labels: {{- include "model-serving.selectorLabels" . | nindent 8 }}
    spec:
      terminationGracePeriodSeconds: {{ .Values.terminationGracePeriodSeconds }}
      {{- with .Values.tolerations }}
      tolerations: {{- toYaml . | nindent 8 }}
      {{- end }}
      {{- with .Values.affinity }}
      affinity: {{- toYaml . | nindent 8 }}
      {{- end }}
      containers:
        - name: server
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          args:
            - "--model={{ .Values.model.id }}"
            - "--max-model-len={{ .Values.model.maxModelLen }}"
            - "--gpu-memory-utilization={{ .Values.model.gpuMemoryUtilization }}"
          ports:
            - name: http
              containerPort: 8000
          resources: {{- toYaml .Values.resources | nindent 12 }}

The two lines that matter most here are easy to skim past. The replicas field is emitted only when autoscaling is off: if the chart always sets it, every helm upgrade resets the replica count to the chart default and undoes whatever the HPA had decided, producing a scale-down under load that looks like an autoscaler bug. And the checksum/config annotation hashes the ConfigMap into the pod template, so changing configuration actually triggers a rollout — without it, Helm updates the ConfigMap and the running pods keep the old contents until something unrelated restarts them.

{{/* templates/hpa.yaml */}}
{{- if .Values.autoscaling.enabled }}
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: {{ include "model-serving.fullname" . }}
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: {{ include "model-serving.fullname" . }}
  minReplicas: {{ .Values.autoscaling.minReplicas }}
  maxReplicas: {{ .Values.autoscaling.maxReplicas }}
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }}
{{- end }}

autoscaling/v2 is the current stable API version and is what supports the metrics list shape above. CPU utilisation is a poor proxy for GPU inference load — see scaling inference on a custom metric — but it is the right default for a chart, because it works without a metrics adapter installed.

The Service template is the dull one and it still has a trap worth naming. Because the HPA targets the Deployment by name and the Service selects pods by label, the two must agree about what the release is called; both should come from the same helper. A chart where the Service selector was hand-written and the Deployment labels came from a helper produces a Service with no endpoints — the pods are healthy, the Deployment is Available, and every request gets a connection refused. kubectl get endpointslices showing an empty address list is the symptom, and it is the first thing to check whenever a freshly installed chart serves nothing.

Per-environment values files

# values-prod.yaml
replicaCount: 4
image:
  tag: "1.9.0"
resources:
  limits:
    cpu: "16"
    memory: 64Gi
    nvidia.com/gpu: 2
model:
  maxModelLen: 32768
autoscaling:
  enabled: true
  minReplicas: 4
  maxReplicas: 12
tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule

1. Render before you install, always: helm template model-serving ./charts/model-serving -f values-prod.yaml. This is the only way to see what will actually be applied, and it catches indentation errors from nindent that a linter will not.
2. Run helm lint ./charts/model-serving -f values-prod.yaml in CI for every values file. A chart that renders correctly for dev and not for prod is the standard way this goes wrong.
3. Install into a namespace explicitly: helm upgrade --install serving ./charts/model-serving -n inference --create-namespace -f values-prod.yaml.
4. Use --atomic --timeout 15m for GPU workloads. A model that takes eight minutes to load will exceed the default five-minute timeout, and without --atomic the failed release is left half-applied.

Upgrading and rolling back

helm history serving -n inference lists revisions with their chart and app versions, and helm rollback serving 7 reverts to one. That is worth relying on for template changes and worth distrusting for anything with an external side effect: rolling back a chart does not roll back a model artefact in object storage, a database migration, or a schema in a vector store.

One habit prevents most upgrade incidents on GPU charts. Because the new pod needs a free GPU before the old one releases it, the default rolling-update strategy deadlocks on a cluster with no spare device: the new pod is Pending waiting for a GPU, the old pod will not terminate until the new one is Ready, and helm upgrade hangs until its timeout. Either set maxSurge: 0 and maxUnavailable: 1 to accept a brief gap, or ensure the pool can grow by one node. It is a scheduling constraint expressing itself as a Helm timeout, which is why it is so often misdiagnosed.

The related failure is a rollback that appears to succeed and changes nothing. helm rollback restores the manifest, so if the previous release used a mutable image tag and the registry now serves different bytes under that tag, you have rolled back the YAML and not the software. This is the concrete argument for immutable tags or digests in values-prod.yaml: without them, the release history records what you asked for rather than what ran, and no amount of Helm machinery recovers the difference.

Finally, resist the urge to reach for helm upgrade --force when something is stuck. It deletes and recreates resources rather than patching them, which for a GPU Deployment means terminating every replica and re-queueing for devices that another workload may take first. When an upgrade will not apply, the answer is almost always an immutable field — read the error, which names it — and the fix is a deliberate replacement of that one object during a window you chose, not a flag that replaces all of them at once.