Queues and Background Jobs for Slow AI Calls

“Put it on a queue” is correct often enough to be a reflex and wrong often enough to be worth a rule. The rule is about your tail latency and your timeout budget, and once you have applied it the interesting work is not the queue at all — it is the job record.

The decision, as a rule you can evaluate

Do not reach for a queue because a call feels slow. Reach for one when a specific inequality holds. Take the p99 duration of the operation, including retries and any fallback rungs, and compare it against the shortest timeout on the path in front of you. That is rarely your own server’s: it is usually a load balancer or CDN idle timeout, and it is frequently sixty seconds or less on managed platforms. Then:

• p99 comfortably under the shortest hop timeout, and the user is waiting anyway — keep it synchronous and stream. A queue adds a hop, a poll and a state machine, and buys nothing if the user is going to sit on the page regardless.
• p99 within a factor of two of the shortest hop timeout — asynchronous. You are one bad day from a class of failure that looks like a 504 to the user and a completed, billed call to the provider.
• The work outlives the session — asynchronous, regardless of duration. Batch imports, document processing and anything fanned out over many items belong on a queue even if each item is fast, because the unit of work is the batch.
• The result is wanted but not awaited — asynchronous. Enriching a record after creation, generating embeddings on upload, writing a summary for later search. Nobody is watching; do not make the request path pay.

Note what is not in the list: cost. A queue does not make a call cheaper. It changes who waits and what happens when the call fails, and it lets you control concurrency centrally, which is a different benefit worth having on its own.

The job is a state machine

The queue is infrastructure and you should think about it as little as possible. The job record is yours, it lives in your database, and it is what the user interface reads. Give it explicit states rather than a pair of booleans.

create table ai_job (
  id            uuid primary key,
  request_key   text not null unique,   -- caller-supplied idempotency key
  state         text not null,          -- queued|running|succeeded|failed|abandoned
  attempts      int  not null default 0,
  lease_until   timestamptz,            -- who owns it, until when
  input_hash    text not null,          -- detects a reused key with new input
  result        jsonb,
  error_class   text,                   -- normalised, not the vendor string
  cost_cents    numeric not null default 0,
  created_at    timestamptz not null default now(),
  updated_at    timestamptz not null default now()
);

Three columns there are the ones people add later, after an incident. attempts lets you stop; without it a poison message runs forever and each run is billable. cost_cents accumulates across attempts, which is how you discover that your failure path is more expensive than your success path. And error_class is a normalised code rather than the vendor’s message, because a dashboard grouped by raw provider strings groups by nothing.

abandoned deserves to be a state distinct from failed. Failed means the work was attempted and did not succeed. Abandoned means you gave up before it was worth trying again — deadline passed, budget exhausted, the user cancelled. They are the same to the queue and completely different to a human reading the table.

Visibility timeouts and the double charge

Here is the failure that is specific to this dependency. Nearly every queue worth using is at-least-once: a worker takes a message, the message becomes invisible for some lease period, and if the worker has not acknowledged by the end of that period the message goes back on the queue for someone else. This is a good design and it is why the queue is reliable.

Now put a model call inside the worker. The call takes long enough to be interesting — that is why it is on a queue — and the lease was set to the platform default, which was chosen for jobs that resize images. The lease expires mid-call. The queue hands the message to a second worker. Both workers now run the same generation. Both are billed. If the job writes its result on completion, the second write wins, or they interleave, and you also have a correctness problem on top of the money.

Three defences, and you want all three:

• Set the lease from the deadline, not from the default. The lease must exceed the worst case of the whole attempt including retries and fallbacks. If your queue supports extending a lease while work is in progress, extend it on a heartbeat rather than guessing a large number up front.
• Claim before you call. Take the job row with a conditional update — set state=’running’ and lease_until only where the current lease has expired — and make the model call only if you won the update. A duplicate delivery then costs a wasted database round trip instead of a generation. The general form of this is the subject of the idempotency page.
• Bound total attempts by spend, not just by count. Five attempts of a cheap call and five attempts of a long document summarisation are not the same risk.

Getting the result back to the user

Once the work is off the request path, the user needs to learn that it finished. Polling a job endpoint is the boring answer and it is right more often than it gets credit for: it survives reconnects, works through every proxy, needs no server-side session state, and the client can back off as the job ages. Return the state, a monotonic updated_at and, if you have one, a progress fraction.

Push mechanisms are better when the wait is long enough that polling would be either wasteful or laggy, and the trade-offs among them — server-sent events, WebSockets, and polling done well — are their own decision. What matters here is that whichever you pick, the job record remains the source of truth. A push that is missed must be recoverable by reading the row, or a dropped connection turns into a job that finished and a user who never found out.

The dead letter queue nobody reads

A dead letter queue that nobody looks at is a way of converting incidents into silence. Two habits make it useful. First, alert on the arrival rate into the dead letter queue, not on its depth: a depth alert fires forever after one bad hour and gets muted. Second, make replay a first-class operation that goes through the same claim path as a normal run, so replaying a batch of two thousand jobs cannot duplicate the ones that actually succeeded before failing on the write.

And keep the failed jobs. The most useful artefact after a bad deployment is a table of jobs with their normalised error class, their input hash and their accumulated cost, because it answers “what did this cost and which inputs caused it” in one query.