Ingress and TLS for a Kubernetes-Hosted Model Endpoint

The Service works from inside the cluster. Getting it to work from outside is an Ingress and a certificate — and then three proxy defaults that are correct for a JSON API and wrong for a token stream.

What sits in front of what

An Ingress object is not a proxy. It is a routing rule that an ingress controller reads and turns into proxy configuration, so nothing happens until a controller is installed and an ingressClassName points at it. On a managed cluster there may already be one; check with kubectl get ingressclass before installing a second, because two controllers watching the same Ingress objects will both claim them.

The chain for an inference endpoint is: client, cloud load balancer, ingress controller pod, Service, model pod. TLS normally terminates at the ingress controller. That means the certificate lives in a Kubernetes Secret and the hop from controller to model pod is plaintext inside the cluster — acceptable in most threat models, and worth replacing with a service mesh if yours says otherwise.

The Ingress object

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: inference
  namespace: inference
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-buffering: "off"
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - inference.example.com
      secretName: inference-tls
  rules:
    - host: inference.example.com
      http:
        paths:
          - path: /v1
            pathType: Prefix
            backend:
              service:
                name: mistral-7b
                port:
                  number: 80

networking.k8s.io/v1 is the stable API version; anything you find using extensions/v1beta1 predates Kubernetes 1.22 and will not apply. Note that pathType: Prefix matches on path segments, not on string prefixes, so /v1 matches /v1/chat/completions but not /v1beta — which is usually the behaviour you want and occasionally a surprise.

Two structural constraints bite early. The TLS Secret must live in the same namespace as the Ingress, so a single wildcard certificate cannot simply be shared across namespaces without something replicating it — cert-manager can issue per-namespace, which is usually simpler than copying Secrets around. And the Ingress backend Service must also be in that namespace; there is no cross-namespace backend reference in networking.k8s.io/v1. If your model Service lives elsewhere, either move the Ingress to it or put an ExternalName Service in between, and prefer moving the Ingress.

A certificate that renews itself

The Secret named under tls.secretName must contain a certificate and key. You can create it by hand and then remember to replace it before it expires, which nobody does. The maintainable option is cert-manager: a controller that watches Ingress objects for the cert-manager.io/cluster-issuer annotation, obtains a certificate via ACME, writes it into the Secret, and renews it before expiry.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
      - http01:
          ingress:
            ingressClassName: nginx

1. Point DNS at the load balancer address first: kubectl get ingress inference -n inference shows it under ADDRESS. HTTP-01 validation fails if the name does not already resolve to the ingress.
2. Apply the Ingress and watch the Certificate object cert-manager creates: kubectl describe certificate inference-tls -n inference. Its events narrate the ACME order.
3. Use the staging ACME endpoint while you are getting this working. The production endpoint enforces rate limits per registered domain, and a loop of failed orders can lock you out for a week. Take the current limits from Let’s Encrypt’s rate limits page.
4. Verify end to end with a request that streams, not just a health check: a curl -N against the completions endpoint with "stream": true in the body.

The defaults that break streaming

A generic ingress works fine for a request that returns one JSON body in 200 milliseconds. Token streaming is a long-lived response delivered in small chunks, and three proxy defaults are hostile to it.

• Response buffering. With buffering on, the proxy accumulates the upstream response and forwards it when it has enough — which turns a stream into one delayed blob. The ingress-nginx annotation is nginx.ingress.kubernetes.io/proxy-buffering: "off". The symptom is unmistakable once you know it: streaming works against the Service and stops working through the ingress, with the whole answer arriving at once at the end.
• Read timeout. The proxy’s read timeout is the gap it will tolerate between chunks, and for a reasoning model that thinks before emitting, that gap can be tens of seconds. Raise proxy-read-timeout and proxy-send-timeout together; a generation cut off at exactly the same elapsed time on every request is a timeout, not a model problem.
• Body size. Long prompts with retrieved context exceed the default request body limit, and the failure is a 413 from the proxy that never reaches your server, so your logs show nothing. proxy-body-size: "0" disables the cap; a large explicit value is the safer choice on a public endpoint.

Do not put it on the internet unauthenticated

An inference endpoint with no authentication is a GPU somebody else can use. Ingress gives you a hostname on the public internet within minutes, and the model server behind it almost certainly has no notion of a caller identity.

The cheapest correct answer is not to expose it at all: keep the Service internal, and reach it from your own applications over cluster DNS. Where external access is genuinely needed, put authentication in front — an ingress auth annotation delegating to an authentication service, an API gateway, or mTLS — and pair it with rate limiting, because a single caller can saturate a GPU far more cheaply than they can saturate a web server. Egress deserves the same attention in the other direction; see network policy for egress to a model provider.

Rate limiting deserves a sentence of its own because the usual defaults are calibrated for the wrong workload. A request-per-second limit is nearly meaningless for inference: one request asking for four thousand output tokens occupies a GPU for a minute, while a hundred requests asking for ten tokens each finish in a second. Limiting concurrent in-flight requests per caller is the control that maps to the scarce resource, and capping max_tokens server-side is what stops a single request monopolising a device regardless of how many are allowed. Neither is expressible in a standard ingress annotation, which is the honest reason this belongs in a layer that understands the protocol rather than in the proxy.