Migrating Long-Running Batch Jobs Between Provider Batch APIs

A batch job is three interactions, not one: you submit work, you poll until the job stops moving, and you fetch results. Providers disagree on all three, and the disagreements are not symmetrical — the hardest one is at collection, where a job that reports success can still contain nothing but failures.

Two shapes of batch API

Batch endpoints exist because the provider can schedule your work against spare capacity instead of serving it interactively, and the usual bargain is a large discount against a completion window measured in hours rather than seconds. The bargain is similar everywhere. The plumbing is not.

File-based. OpenAI’s Batch API takes a JSONL file uploaded through the Files API first. Each line is a complete HTTP request — a custom_id, a method, a url and a body — and the batch object references that file by id. Results come back as another file you download and parse.

Inline. Anthropic’s Message Batches API takes the requests in the create call itself, as a requests array of { custom_id, params } objects, with no file upload step at all. Results are streamed from a results endpoint as JSONL.

The difference is not cosmetic. A file-based API separates upload succeeded from contents were valid, so malformed lines surface hours later inside a job status. An inline API validates the request array when you post it, so a schema mistake is a 400 you see immediately — but your submitter now has to chunk the array itself against a request-count and payload-size ceiling that the file-based API handled as a file size.

Stage one: submission

Here is the same work, twice. File-based, the input file is JSONL, one request per line:

{"custom_id": "doc-0001", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "MODEL_ID", "messages": [{"role": "system", "content": "Classify the ticket."}, {"role": "user", "content": "Card declined twice"}], "max_tokens": 64}}
{"custom_id": "doc-0002", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "MODEL_ID", "messages": [{"role": "system", "content": "Classify the ticket."}, {"role": "user", "content": "Where is my invoice"}], "max_tokens": 64}}

Uploaded with purpose: "batch", then referenced when the batch is created — alongside an endpoint that must match the url on every line, and a completion_window.

POST /v1/batches
{
  "input_file_id": "file-...",
  "endpoint": "/v1/chat/completions",
  "completion_window": "24h"
}

Inline, the same two units of work go in the create call:

POST /v1/messages/batches
{
  "requests": [
    {
      "custom_id": "doc-0001",
      "params": {
        "model": "MODEL_ID",
        "max_tokens": 64,
        "system": "Classify the ticket.",
        "messages": [{"role": "user", "content": "Card declined twice"}]
      }
    },
    {
      "custom_id": "doc-0002",
      "params": {
        "model": "MODEL_ID",
        "max_tokens": 64,
        "system": "Classify the ticket.",
        "messages": [{"role": "user", "content": "Where is my invoice"}]
      }
    }
  ]
}

Three things changed that a mechanical rewrite misses. The system instruction moved out of the message list into a top-level system field, so a submitter that maps messages one-for-one drops it. max_tokens is required rather than optional, so a job that relied on a provider default now fails validation — or worse, gets given a value by your migration script that truncates outputs the old job did not truncate. And there is no per-line url, because the endpoint is fixed by the batch type; a mixed batch of chat and embedding requests, legal in one shape, has to become two batches in the other.

Stage two: polling

Both sides give you a job id and expect you to poll. What you poll for differs enough that this deserves its own page; the short version is that one side’s status enum encodes the outcome and the other’s does not, so a translation of the form “completed means ended” is wrong in the direction that costs you money. The value ranges, and which of them are terminal, are tabled in mapping batch job status values between providers.

What both agree on: poll with backoff, not on a tight loop; the job id is the only handle you have, so persist it before you start polling rather than holding it in memory; and a completion window is a deadline, not an estimate, so your poller needs to handle the case where the window elapses with work unfinished. Store the id, the submission time and the window alongside the input manifest, so a restarted poller can resume a job it did not submit.

Stage three: collection

This is where migrations break, for three reasons that all appear at once.

• Order is not preserved. Neither API promises results in submission order. This is why custom_id exists on both sides and why it is the one field you must never generate as a loop index at write time and re-derive at read time — make it a stable key from your own data, and join on it.
• Per-request outcome lives in a different place. In the file-based shape, an output line carries a response object with its own status_code, plus an error field, and errors may additionally be split into a separate error file. In the inline shape, each result line carries a result object whose type is the outcome. A reader written for one will treat the other’s failures as successes with missing content.
• Partial failure is normal. A batch of ten thousand with forty failures is an ordinary result, not an incident. Your collector needs a re-submission path for the failed subset keyed on custom_id, or you will end up re-running the whole batch and paying for it twice. See testing partial failures in batch inference for how to force that path in a test.

Rewriting the script

1. Split the existing script into three functions if it is not already — submit(manifest), poll(job_id), collect(job) — with a persisted job record between them. A batch script written as one straight line cannot be migrated a stage at a time, and cannot be resumed.
2. Make custom_id a stable business key. If it is currently a row number, change it first, on the old provider, before you touch anything else. This one change is what makes the rest of the migration verifiable.
3. Write the request translator as a pure function from your internal job record to the target request body. Handle the system-prompt move, the required max_tokens, and stop sequences explicitly. Unit-test it against a handful of records with no network involved.
4. Submit a batch of twenty on the new provider with the same twenty records you have old results for. Do not skip straight to the full run; the twenty is what tells you the translation is faithful rather than merely valid.
5. Diff old against new on the join key. You are not looking for identical text — you will not get it. You are looking for records present in one set and absent from the other, outputs that hit the length limit on one side only, and empty completions.
6. Run the full batch with the failed-subset path armed, and check that your collector reports the same count it submitted. Submitted minus succeeded minus failed should be zero; if it is not, you are dropping results, not losing them at the provider.