Batch APIs: Half Price if You Can Wait

Batch inference endpoints typically offer a substantial discount — a 50% reduction is the common headline — in exchange for a completion window measured in hours rather than seconds. Check your provider’s current terms for the exact figure and window; what is stable is why the discount exists, and that tells you which work belongs there.

What is actually being sold

Not a different model, and usually not different hardware. What you are buying is a position on the scheduling curve. From throughput vs latency: an operator maximising tokens per second per GPU wants large batches and high utilisation, and those are exactly what ruin the tail latency of interactive traffic.

Work with no deadline solves that. It can be scheduled into the troughs between interactive peaks, batched as aggressively as the memory allows, and preempted whenever a paying real-time request arrives. It raises the operator’s average utilisation without touching their p99 — and the discount is them sharing that saving with you. Reading it this way makes the eligibility rule obvious: if the request would object to being preempted, it does not belong in the batch tier.

Four questions that decide it

• Is anybody waiting? Not “would it be nice if it were fast” — is a human or a synchronous caller blocked on the answer? If yes, stop here; batch is not for you.
• Are all the inputs known now? Batch submission is one shot: you send the whole file. Any workload where request n+1 depends on the output of request n — an agent loop, a chain of refinements, anything conversational — cannot be expressed as a batch at all. This disqualifies more workloads than the deadline does.
• Is partial completion acceptable? Batches can come back with per-item failures, and items may expire unfinished if the window elapses. Your pipeline needs to tolerate an 80% result and re-queue the rest.
• Is the volume large enough to matter? The discount is proportional; the engineering cost of building submission, polling, result parsing and partial-failure handling is fixed. Below some monthly spend the integration costs more than it saves.

Four yeses and it is very likely the right tool. One no on the first two and it is definitively the wrong one.

The shape of the API, and what changes

Implementations vary but the flow is consistent across providers: build a file of requests, one JSON object per line, each with your own identifier; upload it; create a batch job referencing it and a completion window; poll the job for status; download an output file whose lines carry your identifiers back.

# input.jsonl -- one request per line, custom_id is YOURS and must be unique
{"custom_id":"doc-4471","method":"POST","url":"/v1/chat/completions",
 "body":{"model":"...","messages":[...],"max_tokens":512}}
{"custom_id":"doc-4472", ...}

submit -> job id -> poll -> output.jsonl + errors.jsonl

Results are NOT in submission order. Join on custom_id, always.

Four differences from a synchronous call are worth planning for. Batch quotas are usually metered separately from your synchronous rate limits — which is the underrated benefit, because it means the nightly job stops competing with live traffic for the same bucket. Results are unordered. Errors arrive per item rather than per call, so your parser must handle a mixed output. And streaming does not exist, so anything whose value depends on partial output is out.

The arithmetic, including the parts people forget

ASSUMPTIONS: 2,000,000 items/month, 800 input + 200 output tokens each,
             a synchronous blended rate of $3 per million tokens,
             and a 50% batch discount. Substitute your own figures.

  tokens        = 2e6 * 1000            = 2.0e9  = 2,000 M tokens
  synchronous   = 2,000 * $3            = $6,000 / month
  batch at -50% = 2,000 * $1.50         = $3,000 / month
  saving                                = $3,000 / month

AGAINST WHICH:
  one-off engineering  ~ 2-5 days       (submission, polling, joining, retries)
  ongoing operational  ~ partial-failure handling, expiry re-queues, monitoring
  cash-flow effect     ~ results arrive up to a window later; anything
                         downstream inherits that latency

Two corrections people miss. First, the comparison is not batch-versus-synchronous-at-list — it is batch versus synchronous with prompt caching applied. If your items share a long system prompt, caching may already be delivering a comparable discount on the input half at no latency cost, and the two do not necessarily stack. Work out what caching is already saving you before assuming the batch discount is incremental.

Second, the window is a maximum and not a promise of speed, but it is also not a promise of slowness: jobs frequently complete far sooner. Design for the worst case, and treat early completion as luck rather than as a schedule.

The operational hygiene is worth stating explicitly, because batch mode removes the feedback loop you normally rely on and the failures are therefore quiet. Make custom_id a stable, meaningful identifier — a content hash or a primary key, never an array index — so that resubmitting only the failures is a set difference rather than a reconciliation exercise. Keep the submitted input file, because debugging an output you cannot reproduce the input for is unpleasant. Treat the output as at-least-once and make the downstream write idempotent, since the natural response to a partial batch is to resubmit, and the natural way that goes wrong is duplicate rows. And set an alarm on the job itself: a batch that silently expires unfinished produces no error anywhere in your system unless something was watching for the result that never came.

Patterns that fit, and one that does not

• Backfills. Classifying, summarising or embedding an existing corpus. The canonical fit: everything known up front, no deadline, high volume.
• Evaluation runs. Scoring a model against thousands of test cases. Nobody watches, and the cost saving lets you run the suite more often, which is worth more than the money.
• Synthetic data generation. Large, independent, deadline-free, and tolerant of partial failure by construction.
• Nightly enrichment. Yesterday’s records processed before this morning. The window fits inside the natural cadence.
• The anti-pattern: agents. Any loop where the next request is a function of the last answer cannot be batched, no matter how deadline-free it is. You can batch each level of a fan-out separately, at the cost of one window per level, which is usually enough to make it not worth doing.