Text Generation Inference in a Container

TGI is a production inference server distributed primarily as a container image, which makes it the path of least resistance when your deployment story is already Kubernetes or a container platform. The two things worth getting right are the token-budget arguments, which decide whether it starts at all, and the health endpoints, which decide whether your platform routes traffic to a server that is not ready.

What TGI is for

It occupies the same category as vLLM — continuous batching, efficient attention kernels, an HTTP API — and the choice between them is usually about operational fit rather than about a benchmark. TGI leans harder into being an operable service: a published container image, arguments configurable as environment variables, Prometheus metrics and distinct health endpoints out of the box.

If your team ships containers with a Helm chart and expects a readiness probe and a metrics endpoint from anything it runs, that fit is worth more than a throughput difference you would have to measure to find. For the throughput comparison itself, see serving engines compared.

Running the container

docker run --gpus all --shm-size 1g -p 8080:80 \
  -v $PWD/models:/data \
  -e HF_TOKEN=$HF_TOKEN \
  ghcr.io/huggingface/text-generation-inference:<tag> \
  --model-id <model-id> \
  --max-input-tokens 4096 \
  --max-total-tokens 8192

# Then:
curl 127.0.0.1:8080/health
curl 127.0.0.1:8080/info
curl 127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"tgi","messages":[{"role":"user","content":"hello"}]}'

Four things in that command are load-bearing, and three of them are the usual reasons a first run fails. The volume mount gives the container somewhere persistent to put the downloaded weights, without which every restart re-downloads tens of gigabytes. The token is needed for gated repositories. The shared-memory size matters for multi-GPU sharding, which uses shared memory for inter-process communication and fails obscurely at the container default. And every command-line argument has an environment-variable equivalent, which is what makes the image configurable from a chart without rewriting the entrypoint.

The arguments that decide startup

The relationship between these and memory is the same arithmetic derived on the vLLM page: weights take a fixed amount, what remains holds the key-value cache, and the maximum sequence length divides that remainder into concurrent slots. Setting a total token budget far above what your application sends buys nothing and costs concurrency.

Health checks that mean something

A model server has a property most web services do not: the process is alive and listening long before it can serve anything, because weights take minutes to download and load. Configuring one probe for both questions produces one of two bad outcomes — traffic routed to a server that cannot answer, or a container killed for being slow to start.

• Liveness: is the process wedged? Should fail only when a restart is the right remedy. Give it a generous failure threshold. A liveness probe that fires during model loading produces a container that restarts forever and never serves a request, which is a memorable way to spend an afternoon.
• Readiness: can it serve now? This is the one that controls traffic. It should not pass until the model is loaded, so it needs a long enough startup allowance — for a large model on a cold cache, several minutes is normal, and a separate startup probe is the cleaner way to express that.
• Check the informational endpoint at deploy time. The server reports which model and revision it actually loaded. Asserting that in a smoke test catches the deploy that silently served yesterday’s cached weights.
• Scrape the metrics endpoint. Request counts, queue and batch sizes, and latency histograms are exposed in Prometheus format. These are what the next section is about, and they are the difference between an autoscaler that works and one that oscillates.

What to scale on, derived

The instinct is to autoscale on GPU utilisation. Do not: a GPU serving one request looks close to fully utilised, so the signal saturates long before the server does and tells you nothing about whether requests are waiting. The signal you want is queue depth, and the reason is a two-line derivation.

Little's law:  L = λ × W
  L  requests in the system    λ  arrival rate (req/s)    W  time in system (s)

A replica can hold C requests concurrently (its batch capacity).
So its sustainable arrival rate is:

    λ_max = C / W

Worked example — every value is one you measure, not one taken from here:

  C = 32 concurrent sequences        (from the server's batch metrics)
  W = 6 s per request                (500 output tokens at ~85 tokens/s)

  λ_max = 32 / 6 ≈ 5.3 requests/s per replica

At λ = 8 req/s the queue grows at 8 − 5.3 = 2.7 requests per second.
After 30 seconds, 80 requests are waiting and p95 latency has doubled
even though GPU utilisation has not moved at all.

Replicas needed:  ceil(λ_peak / λ_max) = ceil(8 / 5.3) = 2

Three practical conclusions. Scale on the number of waiting requests, or on a latency percentile, because both move before users complain and neither saturates. Measure W at your own output lengths, since it is dominated by tokens generated rather than by anything else. And set the scale-up threshold well below λ_max, because a new replica must download and load the model before it serves — minutes, not seconds, which is the property that makes model serving different from scaling a web tier.

The mitigation for that last point is pre-pulled images and a persistent volume or cache for the weights, so a new replica starts in the time it takes to load from local disk rather than from the network. If scale-up takes five minutes, keep enough headroom to survive five minutes of growth, and treat the percentile you promise accordingly — see latency percentiles.

Container gotchas

The image is large and so are the weights. Two separate downloads, both slow, both worth caching. A node pool that pulls both on every scale event has an effective scale-up time measured in tens of minutes.

Graceful shutdown is not free. On a rolling deploy, in-flight generations should be allowed to finish. Set a termination grace period longer than your longest expected generation, or every deploy truncates somebody’s answer mid-sentence.

Licensing and gating are real. Gated repositories require an accepted licence attached to the token in the environment, and some model licences constrain commercial use. That check belongs before the deployment work, not after — the ground is covered in open model strategy.

One container, one model. Serving several models means several deployments and something in front routing between them. That router is a real component with its own failure modes, which is the argument for putting model choice behind a gateway rather than behind application logic.