Batching Eval Calls in CI to Reduce Per-Request Overhead

A batch endpoint trades latency for a published discount, which is an excellent trade for an eval nobody is watching. The reason teams bounce off it is not the API; it is that batching changes a job that blocks and asserts into two jobs separated by hours.

Where batch fits, and where it does not

OpenAI’s Batch API documents a 50% cost discount compared with the synchronous APIs, with each batch completing within a 24-hour window and often sooner, a single batch holding up to 50,000 requests, and an input file up to 200 MB. It accepts a set of endpoints including chat completions and embeddings. The current details are in OpenAI’s batch guide. Anthropic ships an equivalent Message Batches API with its own discount and window; see the Message Batches API.

The 24-hour window is the whole fit criterion. It rules out anything that gates a merge, because no one will wait, and it makes the endpoint close to free of downside for anything that runs on a schedule and reports a number.

• Good fit: a nightly eval over a golden set; a weekly regression sweep across several models; a one-off re-scoring of a whole dataset after changing a rubric; generating synthetic test data.
• Bad fit: pull-request contract tests; any check a human is waiting on; anything whose result must exist before a deploy in the same pipeline run.

The pipeline consequence is the part to plan for. A batched eval is two jobs: one that builds and submits, and one that polls, collects and asserts. Between them the result lives at the provider, so the batch identifier has to be persisted somewhere the second job can read it — an artefact, a workflow output, or a small store. A batch submitted by a job whose identifier was never written down is money spent on a result nobody will collect.

custom_id is the join key

Batch input is JSON Lines, one request per line, each carrying a custom_id you choose. Results come back as a file of lines that are not guaranteed to be in submission order, and custom_id is the only thing linking a result to the case that produced it. Treat it as a primary key, not a label.

# build the batch input
import json

with open("batch.jsonl", "w") as fh:
    for case in cases:
        fh.write(json.dumps({
            "custom_id": case.id,          # stable, unique, meaningful
            "method": "POST",
            "url": "/v1/chat/completions",
            "body": {
                "model": MODEL,
                "temperature": 0,
                "max_tokens": 512,
                "messages": build_messages(case),
            },
        }) + "\n")

Make the identifier stable across runs — the case id from your dataset, not an enumeration index — so that yesterday’s results and today’s can be joined to each other as well as to the cases. Make it unique, because a duplicate silently discards one of the two results at collection time. And keep it meaningful, since it is the only thing you will have in a failure report.

Submitting and collecting

1. Build the JSONL file from your case set, one line per case, with the model and parameters pinned exactly as the synchronous suite pins them. Write the file as a build artefact so the submitted content is inspectable later.
2. Upload the file with the batch purpose, then create the batch against the endpoint you are calling with the 24-hour completion window. Record the returned batch id and the input file id as job outputs.
3. Exit. Do not poll in the submitting job — a runner sitting idle for hours is a bill of its own, and CI runners have their own timeouts.
4. In a second scheduled job, retrieve the batch by id and branch on its status. The lifecycle runs through validating, in progress, finalising and completed, with failed, expired, cancelling and cancelled as the terminal alternatives. Expired is the one to handle explicitly: it means part of the work did not finish inside the window.
5. Download the output file, parse one JSON object per line, index by custom_id, and run your assertions over the joined set. Download the error file too — batches report per-request failures separately from the batch status.
6. Publish the aggregate: pass rate, token totals, and the list of case ids with no result. Store it, because the value of a nightly eval is the series, not the run.

Keep the request bodies identical to the ones the synchronous suite builds, generated by the same function. The point of a nightly batch run is to tell you something about the configuration you ship, and a batch file assembled by a separate code path drifts from it silently — a different system prompt, a stale tool schema, a parameter somebody changed in one place. Share the builder, not the shape.

Partial failure is the trap

A batch can reach a completed status with some of its requests having failed. Those failures appear in a separate error file, and the output file simply does not contain a line for them. If your collection step iterates over the results and asserts on each one, the cases that failed are not iterated over, and the suite reports a perfect pass rate on a partial run.

This is the single most important assertion on the page, and it is one line: the number of results must equal the number of cases submitted.

results = {json.loads(line)["custom_id"]: json.loads(line)
           for line in output_file.splitlines()}

missing = [c.id for c in cases if c.id not in results]
assert not missing, f"{len(missing)} of {len(cases)} cases returned no result: {missing[:10]}"

for case in cases:
    body = results[case.id]["response"]["body"]
    assert_properties(case, body)

The same reasoning applies to the aggregate you publish. A pass rate computed over the results present is a different quantity from a pass rate computed over the cases submitted, and only the second one is comparable to yesterday’s. Compute the denominator from the case set, always.

What it does to rate limits

The second benefit, and often the one that actually unblocks a team. Batch queues are accounted separately from synchronous request and token limits, so a large eval sweep stops competing with production traffic on the same key. A suite that previously had to sleep between calls, or serialise itself to avoid 429s, submits one file instead.

That removes an entire class of test flakiness. Rate-limit retries are a common source of both intermittent failures and surprise cost — a retried call is a paid call — and moving the bulk work to a queue that has its own capacity removes the contention rather than backing off around it. The constraint that replaces it is a limit on enqueued tokens rather than on requests per minute, which is a much easier shape to plan a nightly job around. Check the current numbers in the batch guide, since they are per-tier and change.