SQS to Lambda for Asynchronous Model Inference

The interesting part of this pattern is not the trigger. It is that a model call can take ninety seconds, and SQS, Lambda and API Gateway all have opinions about how long anything is allowed to take. Get those opinions to agree and the rest is twenty lines of handler.

The shape, and why the API does not call the model

Three components: an intake path that accepts a request and returns a job id immediately, a queue that holds the work, and a worker Lambda that does the model call and writes the answer somewhere the client can poll or be notified from. A DynamoDB table with the job id as its partition key is the usual place for the answer, because you need somewhere to record PENDING before the work starts anyway.

The reason the intake path does not call the model directly is not elegance. API Gateway’s REST and HTTP APIs enforce an integration timeout that is far shorter than a long generation, and even where you can raise it, holding an HTTP connection open for a minute means a client disconnect turns into an orphaned, already-paid-for call. Once the work is on a queue, a disconnected client is a client that comes back later and reads a row.

Intake can write to SQS directly through an API Gateway service integration with no Lambda in the middle, which removes a cold start from the path a human is waiting on. If you do keep an intake function, it should do exactly two things: write the PENDING row, and call SendMessage with the job id in the body.

The three timeouts that have to agree

This is the part that bites. Lambda’s maximum configurable function timeout is 15 minutes, which is generous for a model call, so the binding constraint is the queue rather than the function.

AWS documents two rules for the relationship, and they are different rules. The hard one: your function timeout must be less than or equal to the queue’s visibility timeout, and Lambda validates this when you create or update the event source mapping and returns an error if it does not hold. The soft one, which is guidance rather than validation: set the source queue’s visibility timeout to at least six times the function timeout, so there is room for Lambda to retry a batch that was throttled mid-flight. If you also set a batch window, AWS’s recommendation becomes six times the function timeout plus the value of MaximumBatchingWindowInSeconds.

Work it through for a 120-second function: visibility timeout 720 seconds. That is well inside SQS’s documented maximum visibility timeout of 12 hours, so nothing is at risk, but it does mean a genuinely stuck message takes twelve minutes to come back — which is the trade you are making and should be a deliberate one. The mechanism behind the six is worked through in setting visibility timeout for slow model calls.

The handler and the record it receives

Batch size for a standard queue can be up to 10,000 records, and for a FIFO queue the maximum is 10. Anything above 10 requires you to also set MaximumBatchingWindowInSeconds to at least 1. For this workload you almost certainly want a small number. Lambda passes the whole batch to one invocation, that invocation has one timeout, and ten sequential ninety-second model calls do not fit inside 15 minutes. A batch size of 1 to 5 with concurrent calls inside the handler is the usual answer.

The whole batch also has to fit the 6 MB synchronous invocation payload quota, and both Lambda and SQS attach metadata to every record, so the effective record count can come out below your configured batch size.

import json, os, boto3

ddb = boto3.resource("dynamodb").Table(os.environ["RESULTS_TABLE"])

def handler(event, context):
    failures = []
    for record in event["Records"]:
        try:
            body = json.loads(record["body"])
            answer = call_model(body["prompt"])       # your provider SDK
            ddb.update_item(
                Key={"job_id": body["job_id"]},
                UpdateExpression="SET #s = :s, answer = :a",
                ExpressionAttributeNames={"#s": "status"},
                ExpressionAttributeValues={":s": "DONE", ":a": answer},
            )
        except Exception:
            failures.append({"itemIdentifier": record["messageId"]})
    return {"batchItemFailures": failures}

Partial batch failures and the dead-letter queue

By default, if Lambda encounters an error at any point while processing a batch, every message in that batch returns to the queue. On a batch of five model calls where the fourth threw, that means paying for the first three a second time. The fix is the partial batch response: set the event source mapping’s FunctionResponseTypes to include ReportBatchItemFailures, and return the shape above, where batchItemFailures is a list of objects each carrying an itemIdentifier equal to the failed record’s messageId. Only those come back.

The two ways to get this wrong are worth naming. Returning an empty list when something did fail silently deletes the failure. And throwing out of the handler after some items succeeded discards the partial report entirely, because Lambda never sees the return value — so catch per record, as above, rather than around the loop.

For messages that fail repeatedly, configure a redrive policy pointing at a dead-letter queue. AWS recommends setting maxReceiveCount on the source queue to at least 5, which gives Lambda a few attempts before a message is moved aside. Set it to 1 or 2 and a single throttled batch sends real work to the DLQ.

Because SQS is at-least-once and the whole design assumes retries, the worker must be safe to run twice on the same message. That is a separate piece of work and it is the subject of idempotency keys for a queued model request. Do not skip it on the grounds that duplicates are rare; they are rare and expensive, which is the worst combination for something you only notice on the invoice.

Building it

1. Create the results table with job_id as the partition key, and the queue plus its dead-letter queue. Set the source queue’s VisibilityTimeout to six times your intended function timeout and RedrivePolicy with maxReceiveCount of 5.
2. Attach the AWSLambdaSQSQueueExecutionRole AWS managed policy to the worker’s execution role. If the queue is encrypted with a customer managed key, add kms:Decrypt as well. Add dynamodb:UpdateItem on the results table.
3. Deploy the handler with a timeout that is honestly larger than your provider’s worst case, not its median, and set the SDK’s own client-side read timeout below the function timeout so a hung socket surfaces as your error rather than as a Lambda timeout.
4. Create the event source mapping with BatchSize of 1 to start, MaximumBatchingWindowInSeconds of 0, and FunctionResponseTypes set to ReportBatchItemFailures.
5. Publish one message, confirm the row flips to DONE, then publish one that you know will fail and confirm it lands in the DLQ after five receives rather than immediately or never.
6. Set the mapping’s maximum concurrency, or use provisioned mode, before you turn on real traffic. Lambda will otherwise scale up against a provider rate limit and convert a backlog into a wall of 429s.

aws lambda create-event-source-mapping \
  --function-name inference-worker \
  --event-source-arn arn:aws:sqs:eu-west-1:123456789012:inference-jobs \
  --batch-size 1 \
  --function-response-types ReportBatchItemFailures \
  --scaling-config MaximumConcurrency=20