Deploying a Custom Container on Vertex AI

Vertex AI will run any container that answers two HTTP routes. The interesting part is not the routes; it is that the platform starts probing one of them immediately, and most serving containers are not ready to answer for another minute.

The contract, in full

Google’s custom container requirements define a small set of environment variables injected into the running container, and reading them rather than hardcoding is the difference between a container that works on one deployment and one that works everywhere:

• AIP_HTTP_PORT — the port your HTTP server must listen on. Defaults to 8080 and is the value the platform will actually connect to. Hardcoding 8080 works until the day it does not.
• AIP_HEALTH_ROUTE — the path health checks are sent to, which your server must support. This is set from the --container-health-route you gave at upload.
• AIP_PREDICT_ROUTE — the path prediction requests are sent to, likewise from --container-predict-route.
• AIP_STORAGE_URI — the Cloud Storage path your model artifacts were staged to, from --artifact-uri. Treat it as read-only.

The health route must return 200 when the server is ready to accept requests, and anything else — a connection refused, a 503, a timeout — is read as not ready. The prediction route receives a JSON body and must return JSON. If you registered the model for :predict, the platform hands you an object with an instances array and expects an object with a predictions array back; if you intend to speak your own schema, callers must use :rawPredict, which passes bodies through unchanged in both directions.

A server that satisfies it

import os
from fastapi import FastAPI, Request

app = FastAPI()

# Read the routes before defining them; they are not compile-time constants.
HEALTH = os.environ.get("AIP_HEALTH_ROUTE", "/health")
PREDICT = os.environ.get("AIP_PREDICT_ROUTE", "/predict")
PORT = int(os.environ.get("AIP_HTTP_PORT", "8080"))

model = load_model_from(os.environ["AIP_STORAGE_URI"])  # runs at import time

@app.get(HEALTH, status_code=200)
def health():
    return {}

@app.post(PREDICT)
async def predict(request: Request):
    body = await request.json()
    outputs = [model.score(row) for row in body["instances"]]
    return {"predictions": outputs}

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY serve.py .
# 0.0.0.0, not 127.0.0.1: a loopback bind is unreachable from outside the container.
CMD exec uvicorn serve:app --host 0.0.0.0 --port ${AIP_HTTP_PORT:-8080}

Load first, bind second

This is the whole trick, and it is why the model load in the example above sits at module scope rather than inside a startup event handler that races the server.

The platform begins health-checking as soon as the container starts. A server that binds its port immediately and loads a large checkpoint in the background will answer the first probe — with a 200, because the framework is up — while being completely unable to serve a prediction. The deployment goes healthy, traffic arrives, and every early request fails against a half-initialised model. The opposite arrangement is the correct one: do the expensive work first, bind the port last, and accept that the platform sees a connection refused in the meantime. A refused connection is an honest “not yet”; a 200 from a server with no model loaded is a lie the platform believes.

The same ordering argument applies to warmup. If your first inference compiles kernels or allocates GPU memory, run one synthetic inference before you bind, so the first real request is not the one that pays for it.

Registering and deploying it

1. Push the image to Artifact Registry in a region compatible with your serving region: docker push us-central1-docker.pkg.dev/PROJECT/serving/sentiment:3.1.
2. Upload the model, declaring the routes and port so the platform and the container agree on them. The flags are --container-image-uri, --container-health-route, --container-predict-route, --container-ports and --artifact-uri.
3. Create an endpoint and deploy the model onto it with a machine type and replica range, as covered in deploying to a Vertex AI endpoint.
4. Call :predict with a real instance and confirm the shape of what comes back matches what your clients expect.

Google’s CLI reference documents --container-env-vars as a list of key-value pairs set as environment variables, which is where configuration that is not a secret belongs. Secrets do not belong in the image and do not belong in this flag; give the deployed model a service account via --service-account and let it read Secret Manager at startup.

What the platform does not do for you

The contract is deliberately thin, and everything it does not mention is yours. Four of those gaps cost people a week each.

• Request batching. Vertex AI routes requests to replicas; it does not merge several concurrent requests into one forward pass. On a GPU that is most of the throughput you are paying for, left on the table. If you want batching, it lives inside your container — a short collection window, a queue, and a batched inference call — or you adopt a serving runtime that already implements it.
• Concurrency limiting. Nothing stops your replica being handed more simultaneous requests than the model can hold. A single-GPU container that accepts twenty concurrent requests will interleave all twenty and return all twenty slowly, and from outside it looks like a latency problem rather than an admission-control one. Bound it yourself with a semaphore and return quickly when full.
• Graceful shutdown. During a scale-down or a redeploy the container is signalled and then terminated. Requests in flight when that happens are lost unless you handle the signal, stop accepting new work, and drain. A long generation is exactly the request most likely to be in flight.
• Structured logging. Anything written to stdout is captured, which means an unstructured traceback arrives in Cloud Logging as a pile of single-line entries with no severity. Emitting JSON lines with a severity field costs three lines of setup and turns the log into something you can filter during an incident.

There is also a payload ceiling on the prediction path, which is the constraint that most often forces an architecture change late. Online prediction is built for small request and response bodies; a container that wants to accept a large document or return a large artefact should exchange Cloud Storage URIs rather than bytes, with the container reading and writing through its own service account. Discovering this after building around inline payloads is a rewrite of the client as well as the server.

Debugging a container that never goes healthy

• Reproduce locally with the same variables. Run the image with AIP_HTTP_PORT, AIP_HEALTH_ROUTE, AIP_PREDICT_ROUTE and AIP_STORAGE_URI set to plausible values, then curl the health route. Most failures reproduce here in under a minute.
• Check the bind address. A server on 127.0.0.1 is invisible to the platform. It has to be 0.0.0.0.
• Check the architecture. An image built on an ARM laptop without --platform linux/amd64 fails to start on x86 serving hardware, with an error that reads like a corrupt binary rather than a mismatch.
• Read the container logs, not the operation error. The deploy failure message says the deployment did not become healthy; the reason is in Cloud Logging under the deployed model’s resource, and it is usually an ordinary Python traceback from the load step.
• Check the service account can read the artifacts. If AIP_STORAGE_URI points somewhere the deployed model’s identity cannot read, the load throws and the container exits in a loop.