Running Batch Prediction Jobs on Vertex AI

Batch prediction on Vertex AI is not an endpoint being called in a loop. It is a separate job resource that takes a model, provisions machines for the duration, writes results somewhere, and disappears. The differences are what make it worth using.

No endpoint is involved

An online prediction needs a Model deployed to an Endpoint. A BatchPredictionJob takes the Model resource name directly. There is no endpoint to create, no traffic split, no replicas left running after the work is done, and no accelerator sitting idle between batches. For a nightly scoring run this is the entire argument: the alternative is an endpoint that bills all night to be busy for eleven minutes.

The other structural difference is throughput. Online prediction is shaped by a per-request latency budget; a batch job is shaped by throughput and can use a much larger replica count for a short window without any of it being visible to a user. It is the same relationship that Cloud Run jobs have to Cloud Run services — a run-to-completion resource rather than a request-serving one.

Input and output configuration

The job carries an inputConfig and an outputConfig, and each declares a format alongside a location. On the input side, instancesFormat names the encoding — jsonl, csv, bigquery and file-list variants are the ones in general use — and then either gcsSource.uris (an array, wildcards allowed) or bigquerySource.inputUri in the form bq://project.dataset.table. On the output side, predictionsFormat is matched with gcsDestination.outputUriPrefix or bigqueryDestination.outputUri.

The BigQuery destination is the field people get wrong. You give it a dataset, not a table — bq://project.dataset — and Vertex AI creates tables underneath it for the run. If you point it at an existing table expecting an append, the job fails validation before any machine is provisioned, which at least is cheap.

Submitting the job

The REST body is small enough to write by hand, and writing it by hand is worth doing once because the same field names appear in the Python SDK and in Terraform.

{
  "displayName": "sentiment-nightly-2026-08-11",
  "model": "projects/PROJECT/locations/us-central1/models/MODEL_ID",
  "inputConfig": {
    "instancesFormat": "bigquery",
    "bigquerySource": { "inputUri": "bq://PROJECT.support.tickets_to_score" }
  },
  "outputConfig": {
    "predictionsFormat": "bigquery",
    "bigqueryDestination": { "outputUri": "bq://PROJECT.support_scored" }
  },
  "dedicatedResources": {
    "machineSpec": { "machineType": "n1-standard-8" },
    "startingReplicaCount": 4,
    "maxReplicaCount": 20
  }
}

POST that to the batchPredictionJobs collection on the regional host, or use gcloud ai custom-jobs’ sibling command for batch jobs with --config pointing at the same JSON. startingReplicaCount is the initial fleet and maxReplicaCount the ceiling Vertex AI may scale to; both draw on the same regional machine quota an online deployment would.

For Gemini and other foundation models the shape is the same but the input rows are request bodies rather than feature vectors: one JSON object per line containing a request field holding a full generateContent body. The output carries the request back alongside a response field and a status column, which is the only reason the next section is not a problem.

Reading the results back

This is the part that catches people, and it is not documented as loudly as it should be. Batch prediction is parallel across replicas. Output rows are not guaranteed to be in input order, and for the Cloud Storage destination they are spread across multiple shard files with no ordering between them. Any code that zips the input list against the output list by position is wrong, and it is wrong intermittently, which is the worst way to be wrong.

Carry a key. For a BigQuery source, non-feature columns are echoed into the output table, so include your primary key as a column and join on it. For a JSONL source, put the key inside the instance object and read it back out of the echoed request. For foundation-model batches, the returned request field contains your original body, so a key embedded in the prompt metadata survives the round trip.

Expect a per-row status too. A batch job does not fail because one row failed; it records the error against that row and carries on. Query the error column before you treat the run as complete, or you will silently treat a partial result as a full one.

How a job fails

A batch job has three distinct failure points and they surface at different times, which is why “the job failed” is never a complete description.

• Validation, before any machine starts. A malformed destination, a source the job’s identity cannot read, a mismatch between instancesFormat and what is actually at the source. These fail within seconds and cost nothing, which is the good case. Read the job resource’s error field rather than the state, because the state only tells you it is not running.
• Provisioning. The job asked for a machine shape the region has no quota or no capacity for. This looks like a job stuck in a pending state rather than a failure, and it is the same accelerator quota an online deployment competes for — one reason a large nightly batch and a live endpoint in the same region and the same project can starve each other.
• Per-row errors during the run. These do not fail the job. Each is recorded against its row and the run reports success.

That third case is the one that causes real damage, because a downstream pipeline reading the output table sees rows and proceeds. If twelve percent of rows carry an error and eighty-eight percent carry a prediction, a naive consumer silently drops twelve percent of the population — and if the failures correlate with something, which they usually do, the surviving sample is biased rather than merely smaller. Make the error count a checked precondition of the next step, with a threshold, rather than a column somebody may look at.

Cancellation is worth knowing about before you need it. A running job can be cancelled, and cancellation is not instant — replicas finish what they are doing. Anything already written stays written, so a cancelled job leaves a partial output table rather than nothing, which is another reason for the consumer to check completeness rather than existence.

What it costs and how long it takes

For a custom model, you pay for the machines the job provisions for as long as it runs, at the same node rates an online deployment uses. For Google’s foundation models, batch is priced per token like everything else, and Google’s Vertex AI generative AI pricing page lists batch and flex rates at half the standard rates — for example Gemini 2.5 Flash at $0.15 per million input tokens against $0.30 standard, as published at the time of writing. That trade is the reason batch exists commercially: you give up latency guarantees and get the throughput at half price. The rate card is structured so that this discount stacks with context caching.