Batching for Half Price

Providers with an asynchronous tier discount it substantially — often described as around half off — in exchange for a completion window measured in hours rather than seconds. It is the largest unconditional price reduction on the menu, and most teams leave it unclaimed because nobody ever wrote down which of their traffic has a human waiting for it.

What the discount is actually for

You are selling scheduling flexibility. A synchronous request has to be served now, on hardware that must be sized for your peak, with capacity sitting idle in the trough. A batch request can be run whenever there is spare capacity — overnight, between other customers’ peaks, in whatever batch size the scheduler prefers. Larger batches use the accelerator far more efficiently, and that efficiency is what is being shared with you.

The mechanics are the same everywhere: upload a file of requests, get a job id, poll it, download a file of responses, with a stated completion window. The discount usually applies to both input and output tokens, and prompt caching may or may not interact with it. As always, treat the exact discount and window as inputs to look up rather than as facts to remember — what does not change is the shape of the trade.

The one-question rule

Is there a human waiting for this specific response? If no, it is a batch candidate. That question is better than any list because it is about the request, not about the feature, and the same feature usually contains both kinds.

• Almost always batchable: nightly enrichment and classification of new records; embedding a corpus; generating summaries, titles or tags that are read later; evaluation suites and regression runs; backfilling a new field across historical data; synthetic data generation; scheduled reports; content moderation of an archive.
• Never batchable: anything in a request/response cycle a person is looking at, interactive agents, autocomplete, live chat.
• The interesting middle: work triggered by a user action whose result is delivered asynchronously anyway — “we will email you when the analysis is ready”, onboarding imports, document processing with a progress indicator. If the product already promises minutes-to-hours, the batch window may fit inside a promise you have already made.

A useful second pass: look at what fraction of your token volume, not your request count, sits in the batchable set. Offline jobs tend to have far more tokens per request than interactive ones, so the token share is usually much larger than the request share, and the token share is what the discount applies to.

There is also a category people miss entirely, which is work that is currently synchronous only because nobody questioned it. A great deal of inference happens on a write path — tagging a record when it is created, summarising a document at upload, generating a title when a conversation starts — where the result is not displayed until some later page load. That work is synchronous by inheritance from the code around it, not by requirement. Ask of every model call in a write path: how long could this take before anyone would notice? For a surprising share of them the honest answer is measured in hours, and the change is to enqueue rather than to call.

Is the migration worth the hours

Batching is not free to adopt: it needs a job submitter, a poller, a results ingester, error handling for partial failures, and somewhere to put results that arrive hours later. Payback:

monthly_saving = B * s * d

  B  monthly inference bill
  s  share of that bill which is batchable, by token spend
  d  the discount, as a fraction

payback_months = (h * rate) / monthly_saving

  h     engineer-hours to build and ship it
  rate  loaded hourly cost of an engineer

Worked, all inputs assumed: an $8,000/month bill, 30% of it batchable, a 50% discount, 24 hours of work at a loaded $120/hr.

monthly_saving = 8000 * 0.30 * 0.50 = $1,200
payback_months = (24 * 120) / 1200 = 2,880 / 1,200 = 2.4 months

Under three months, and the saving continues after that. Now try it on an $800 bill with the same work: $120/month saved, 24 months payback, and the answer flips. Batching is a lever whose worth scales with the bill and whose cost does not, which puts it low on a small bill’s priority list and high on a large one’s.

The soft-deadline hybrid

For the middle category — batchable, but with a deadline you would rather not miss — the pattern is to submit to the batch tier and keep a synchronous escape hatch. If the batch has not returned by your deadline D, run the outstanding items synchronously.

E[cost] = C_batch + f * C_sync

  f  fraction of items not returned by the deadline
  (you pay for the batch item either way once submitted)

With C_sync = $1.00 per 1k items, d = 0.5 so C_batch = $0.50,
and f = 0.10:

E = 0.50 + 0.10 * 1.00 = $0.60 per 1k   vs $1.00 all-sync
still a 40% saving while meeting the deadline 100% of the time.

The design only works if f stays small, so it needs the deadline to be comfortably inside the stated window rather than at its edge, and it needs the duplicate submission to be safe — which is a reason to make the downstream write idempotent on a request id before building it.

What batching costs you

• The window is a ceiling, not a promise of speed. Jobs often finish much sooner, and you must design for the ceiling anyway. Anything that assumes “usually quick” will fail on the day the queue is long.
• Partial failure is the normal case. A batch of 50,000 items can return with a few hundred errors of assorted kinds. The ingester must handle a per-item error object, and the retry policy for those is a separate decision with its own cost.
• No streaming, and no early stop. You pay for whatever was generated, and you cannot cut a runaway answer off part way. max_tokens is doing more work here than it does synchronously.
• Debugging is slower. A prompt bug found six hours after submission has already been paid for across the whole batch. Run a fifty-item batch first, always, and diff the output shape before submitting fifty thousand.
• Ordering is not guaranteed. Responses come back keyed by your custom id, not in submission order. If your ingester assumes order, it will corrupt data quietly rather than failing.