Batching Requests From a Queue Before Calling a Model API

Three unrelated techniques get called batching, and only one of them reduces what you pay per token. Getting them confused produces a worker that receives ten messages at a time, makes ten separate model calls, and reports itself as batched.

Three different things called batching

• Receive batching. Pulling several messages per API call to the broker. Saves broker requests. Does nothing at all to the provider bill.
• Array-input endpoints. One HTTP request carrying many units of work, where the provider’s endpoint accepts a list. Saves round trips and request-rate quota. Token cost is unchanged.
• Asynchronous batch APIs. A separate, slower endpoint with its own pricing. This is the one that changes the token price, and it costs you latency measured in hours.

Decide which of the three your problem is before writing anything. “We are hitting the requests-per-minute limit” is the second. “The nightly classification job costs too much” is the third. “Our SQS bill is noticeable” is the first, and is almost never true.

Draining the queue into a window

Whichever of the last two you are aiming at, you need to accumulate work before you act on it. The broker-side limits are small: a single SQS message batch request can include a maximum of 10 messages, so MaxNumberOfMessages caps at 10 and so do DeleteMessageBatch and SendMessageBatch. Building a batch of 200 means twenty receive calls, and the loop that does it needs a deadline as well as a target size, or a quiet period leaves work sitting in a half-full buffer forever.

import time, boto3

sqs = boto3.client("sqs")

def drain(queue_url, target=100, max_wait=30):
    items, deadline = [], time.monotonic() + max_wait
    while len(items) < target and time.monotonic() < deadline:
        resp = sqs.receive_message(
            QueueUrl=queue_url,
            MaxNumberOfMessages=10,          # documented maximum
            WaitTimeSeconds=5,               # long poll; see the polling page
            AttributeNames=["ApproximateReceiveCount"],
        )
        batch = resp.get("Messages", [])
        if not batch:
            break
        items.extend(batch)
    return items

Two things about the accumulated buffer. It is not durable — the messages are in flight, so if the process dies they return after the visibility timeout, which is another reason that number matters. And the visibility timeout now has to cover the drain window plus the combined call, not just the call.

Endpoints that genuinely take an array

Embeddings are the clear case: the input parameter accepts a list of strings and returns one vector per element, in order, from a single request. Combining a hundred documents into one call turns a hundred requests-per-minute against your quota into one, which on an embeddings-heavy pipeline is the difference between working and throttling.

Chat and completion endpoints are the other case, and here the honest answer is that they do not take an array. You can put ten questions into one prompt and ask for ten answers, and people do, but that is not batching — it is a different prompt with different failure modes. The answers interfere with each other, a single malformed item can derail the rest, the output token limit is now shared between ten answers, and you have to parse a delimiter out of free text. It is occasionally worth it for tiny classification items. It is not a general technique and should not be presented to a team as one.

The asynchronous batch APIs

Both major providers ship a separate asynchronous endpoint at half price, and the shape is close enough that one worker can target either.

Anthropic documents its Message Batches API as reducing costs by 50%, with a batch limited to either 100,000 Message requests or 256 MB in size, whichever is reached first. Results are accessible when all messages have completed or after 24 hours, whichever comes first; batches expire if processing does not complete within 24 hours, and results remain available for download for 29 days after creation. Each request carries a custom_id that must be 1 to 64 characters of alphanumerics, hyphens and underscores. Results come back as JSONL with a result type of succeeded, errored, canceled or expired — and expired requests are documented as not billed.

OpenAI documents its Batch API at a 50% discount against the synchronous endpoints, taking a .jsonl file where each line carries custom_id, method, url and body. The completion window can only be set to 24h. The documented limits are 50,000 requests per batch and a 200 MB file, with a separate ceiling of 50,000 embedding inputs across all requests in an embeddings batch.

The design consequence is that a batch API is not a drop-in replacement for your worker. It is a second pipeline: submit, poll, retrieve, reconcile by custom_id, and handle the expired-after-24-hours case by requeueing to the synchronous path. Neither publisher guarantees result ordering, which is precisely why the id field exists.

What one bad item does to the whole batch

This is where naive batching hurts. With a hundred items in one array request, a validation error on item 57 can reject the request, and the other ninety-nine were fine. With a queue behind it, a naive worker then returns all hundred messages to the queue, retries, and fails again on the same item forever — and if the endpoint charged for the successful ninety-nine, you pay again each time.

The pattern that survives this is bisect-on-failure: on a batch-level rejection, split the batch in half and retry each half, recursively, until the failing item is isolated to a batch of one and can be dead-lettered on its own. It costs a logarithmic number of extra requests in the rare failing case and nothing at all in the common one.

For queue settlement, delete per item rather than per batch. Both SQS and most brokers let you acknowledge selectively, and if you are on Lambda the mechanism is the partial batch response described in SQS to Lambda for asynchronous inference.

Building it

1. Decide which of the three batchings you need. If the answer is “cheaper tokens”, you are building the asynchronous path and the rest of this list is secondary.
2. Write the drain loop with both a target size and a wall-clock deadline. Cap MaxNumberOfMessages at 10 because that is the documented maximum, and loop.
3. Raise the queue’s visibility timeout to cover the drain window plus the combined call plus the settle loop.
4. Assign every item a stable id at intake, and use it as the custom_id. Reconcile on it. Never rely on response order.
5. Implement bisect-on-failure before you go to production, not after the first poison item.
6. For the asynchronous path, add a reaper: anything still unresolved after the provider’s expiry window goes back to the synchronous queue, or the job silently never completes.