Batch vs Streaming for AI Workloads

This argument is usually conducted as a preference. It is not one. There is a single number that decides it, most teams have never written it down, and once written down it makes the rest of the design mechanical.

The decision rule

The number is staleness tolerance: how long may a change in the source take to become visible to a query? Write it as a target and a hard limit, the same way you would write any other service objective, and get whoever owns the feature to agree to it.

The last row is worth dwelling on because it is where an enthusiastic design ends up. If indexing is synchronous with the write, then an embedding provider being slow makes your product’s save button slow, and an embedding provider being down makes it impossible to save anything. That is a very large availability cost for freshness nobody asked for.

Two clarifications make the rule usable. First, the tolerance is per content type, not per system: a price list and a five-year-old policy document do not need the same freshness, and forcing the whole corpus to the tightest requirement is how a two-minute pipeline ends up running against two hundred thousand documents that change annually. Split the sources and run two schedules. Second, deletions usually need a tighter tolerance than updates — a document that should no longer be retrievable is a different kind of problem from one that is slightly out of date, and it is reasonable to propagate removals immediately while batching everything else overnight.

Why batch is cheaper, structurally

Three effects, and none of them depends on a particular provider or price.

• Amortised fixed costs. A run has overhead — connection setup, model loading, index opening, warm caches. Spread over 100,000 documents it disappears; paid per document it dominates for small documents.
• Request batching. Embedding APIs accept many inputs per request, and self-hosted inference is bounded by memory bandwidth rather than arithmetic, which is exactly the regime where batching is nearly free. Throughput and latency trade against each other here, and batch is the end of that trade where throughput wins.
• Asynchronous tiers. Providers offer discounted processing in exchange for latency tolerance — you submit a job and collect results within a window. A batch pipeline can use these by construction; a streaming one cannot. The economics of that trade are a large part of the case for batch.

There is also a fourth effect that is not about money: a batch job is restartable. It reads a set of inputs and writes a set of outputs, and if it fails you run it again. A streaming pipeline that fails halfway through has partially applied state, and recovering from that is a design problem you have to solve up front.

The micro-batch middle

The unadvertised answer to most requirements: keep the batch job, and run it every two minutes over a queue of changed ids. You get near-streaming freshness with batch code, batch restartability and most of the batching efficiency.

def micro_batch(queue, max_items=500, max_wait_s=120, embed_batch=96):
    """Drain up to max_items, or whatever arrived within max_wait_s."""
    ids, deadline = set(), time.monotonic() + max_wait_s
    while len(ids) < max_items and time.monotonic() < deadline:
        got = queue.pop_many(max_items - len(ids), timeout=5)
        if not got:
            break
        ids.update(got)              # a set: five edits to one doc = one job

    docs = [load(i) for i in ids]
    for i in range(0, len(docs), embed_batch):
        upsert(embed(docs[i:i + embed_batch]))
    queue.ack(ids)

The set is the whole trick. A document edited five times in two minutes is embedded once, and under a burst — a bulk import, a find-and-replace across a wiki — the collapse ratio gets better exactly when you need it to. A per-event streaming pipeline embeds it five times and pays five times.

Getting the change events at all

Both micro-batch and streaming need to know what changed, and there are three ways to find out, in descending order of reliability.

• Change data capture. Read the database’s replication log — Debezium over Postgres logical decoding or the MySQL binlog. Catches every change including ones made by a migration script or by somebody in a psql session, which application-level hooks miss by definition.
• The outbox pattern. The application writes a row to an outbox table in the same transaction as the change, and a relay publishes it. Slightly invasive, and it gives you exactly-once semantics with respect to the source transaction, which webhooks never do.
• Polling a modified-since column. Simple, and it misses deletes entirely and anything that changed without touching the column. Fine as a supplement, dangerous as the only mechanism — pair it with a reconciliation pass that compares id sets.

When you need both

The mature shape is a fast path and a slow path over the same code. The fast path applies changes as they arrive; the slow path re-derives everything from source on a schedule and reconciles. The slow path is what protects you from every bug in the fast path — a dropped message, a failed retry, an event processed out of order — and it is the reason you can run the fast path without the paranoia that would otherwise be appropriate.

Two disciplines make the pairing safe. Both paths must write through the same idempotent function keyed on content, so applying a change twice is a no-op. And the reconciliation must report the discrepancies it fixed rather than silently fixing them — a rising fix count is your only early warning that the fast path has developed a leak.

Watch freshness itself as well as the pipeline’s health. The metric that matters is the age of the oldest unprocessed change — not queue depth, which conflates a backlog with a burst, and not throughput, which looks healthy right up until the consumer is draining a queue it will never catch up with. Oldest-pending-age maps directly onto the tolerance you wrote down at the top of this page, which makes it the one number worth alerting on.