Fan-Out From SNS to Multiple Model Processing Queues

One document arrives and three different model calls should happen to it. You can write three SendMessage calls, or you can publish once and let SNS do the delivery. The second is not just tidier — it is the version where adding a fourth consumer does not require redeploying the producer.

Why a topic instead of three sends

The argument is about failure, not about lines of code. With three sends from the producer, the producer must decide what to do when the second succeeds and the third times out. Retry all three and the first consumer sees a duplicate. Retry only the third and you now have partial-failure bookkeeping inside a request handler.

With a topic, the producer makes one Publish call that either succeeds or does not. SNS owns delivery to each subscriber and retries each independently. Each consumer gets its own queue, its own visibility timeout sized to its own model call, its own dead-letter queue and its own backlog. A slow summarisation worker no longer delays moderation.

The persistence point matters too: SNS on its own is push-only and a subscriber that is down misses the message. Fanning out to SQS rather than to Lambda or HTTPS endpoints gives every consumer a durable buffer, which is what makes a consumer deployment a non-event.

The envelope, and raw message delivery

By default, the message that lands in the queue is not what you published. Amazon documents SNS as wrapping the payload in a JSON notification document, so the SQS message body is an object with Type of Notification, a MessageId, the TopicArn, an optional Subject, your payload as a string in Message, a Timestamp, and signature fields including SignatureVersion, Signature, SigningCertURL and UnsubscribeURL.

Your JSON is therefore a string inside a JSON document, which means two parses. This is the single most common surprise in a first fan-out, and the reason a consumer that worked against SQS directly breaks the moment a topic is inserted in front of it.

import json

def payload_from(record, raw_delivery: bool):
    body = json.loads(record["body"])
    if raw_delivery:
        return body                      # your object, unwrapped
    return json.loads(body["Message"])   # SNS notification envelope

If you do not need the envelope, set the subscription attribute RawMessageDelivery to true and the queue receives your payload unchanged. The trade is that you lose the topic ARN, the timestamp and the signature fields, so a consumer subscribed to two topics can no longer tell which one a message came from. Publish that information as a message attribute if you turn raw delivery on.

The envelope also costs you payload budget, and the two services do not agree on how much there is. Amazon documents SQS’s maximum message size as 1,048,576 bytes — 1 MiB — while SNS documents a current maximum of 256 KB, with the Extended Client Libraries offloading to S3 for payloads up to 2 GB. So a document that fits comfortably in an SQS message can be rejected by the topic in front of it, and the fan-out is where you discover this rather than in the direct-send version you tested with. Publish a reference — a bucket and key, or a row id — and let each consumer fetch the body. That is worth doing on principle for inference work anyway, since four subscribers each receiving a full copy of a large document is four copies of it moving through the system for no reason.

The queue policy that makes it work

Subscribing succeeds even when the queue will not accept the delivery, which is why a fan-out that looks correctly configured can deliver nothing. SNS needs permission to call sqs:SendMessage on each queue, and that permission lives on the queue’s own resource policy, not on any role you control.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowTopicToSend",
    "Effect": "Allow",
    "Principal": { "Service": "sns.amazonaws.com" },
    "Action": "sqs:SendMessage",
    "Resource": "arn:aws:sqs:eu-west-1:123456789012:embed-jobs",
    "Condition": {
      "ArnEquals": { "aws:SourceArn": "arn:aws:sns:eu-west-1:123456789012:doc-events" }
    }
  }]
}

The Condition is not optional in any meaningful sense. Without it you have granted every SNS topic in the world the right to write to your queue. With it, only the named topic can. Use ArnEquals on aws:SourceArn rather than a wildcard, and add a second statement rather than loosening the first when you add a topic.

When you subscribe from a queue in a different account, both sides need configuring, and the failure is silent in the same way. Check delivery by publishing a test message and reading ApproximateNumberOfMessagesVisible on each queue, rather than by confirming the subscription exists.

Filter policies, so each queue sees only its work

Fan-out to four queues means four copies of every event, and if the moderation worker only cares about user-generated documents it is now receiving and discarding the rest — paying receive requests, and worse, occasionally failing to discard them correctly.

A subscription filter policy pushes that decision into SNS. Set the FilterPolicy attribute on the subscription and SNS evaluates it before delivering. By default the policy is evaluated against message attributes; set FilterPolicyScope to MessageBody to evaluate against the payload itself, which is usually what you want when the producer is publishing a domain event rather than a transport-annotated one.

The pattern that keeps this maintainable is to publish one rich event and let each subscriber narrow it, rather than publishing four consumer-specific events to four topics. The moment the producer knows the names of its consumers, you have the coupling back that the topic was supposed to remove.

At-least-once, multiplied by the subscribers

Fan-out does not introduce duplicate delivery, but it does multiply the number of places it can happen and it removes any hope of handling it centrally. SNS retries each subscriber independently, so a delivery failure to the moderation queue produces a second copy there and no second copy anywhere else. Every consumer therefore needs its own idempotency check, and none of them can rely on another having done it.

The detail that is genuinely easy to get wrong is the shape of the key. The obvious choice — the event id — is wrong here, because three consumers are doing three different pieces of work in response to one event. If all three claim the key evt-8841 against a shared idempotency table, the first to arrive wins and the other two conclude the work is already done. The key has to identify the operation as well as the event: evt-8841:summarise, evt-8841:embed, evt-8841:moderate, or a separate table per consumer. Either works; silently sharing a namespace does not, and the failure looks like two of your three pipelines mysteriously never running. The mechanism itself is in idempotency keys for a queued model request.

One more consequence of independence: each subscription has its own delivery retry policy and its own dead-letter queue, set on the subscription rather than on the topic. A subscriber whose queue policy is wrong, or whose queue was deleted, fails delivery quietly while every other path works — which is exactly the failure that a single producer making three sends would have surfaced immediately. Give each subscription a redrive policy pointing at a dead-letter queue, and alarm on NumberOfNotificationsFailed per topic in CloudWatch. Without that alarm the only evidence that one of four consumers stopped receiving work is that its output stopped appearing, and nobody watches for the absence of a thing.

Building it

1. Create the topic and one queue per consumer, each with its own dead-letter queue and its own visibility timeout sized to that consumer’s model call.
2. Put the resource policy above on every queue, with the topic ARN in the condition. Do this before subscribing so the first test message actually lands.
3. Subscribe each queue with aws sns subscribe --protocol sqs --notification-endpoint QUEUE_ARN, then set RawMessageDelivery deliberately — on or off, but decided, and the same across all subscribers if you can manage it.
4. Add a FilterPolicy to any subscription that does not want every event, with FilterPolicyScope set explicitly.
5. Publish one message and confirm it appears in every queue you expect and none you do not. This is the step that catches the missing resource policy.
6. Give each consumer its own idempotency check. Fan-out multiplies at-least-once delivery by the number of subscribers, and each path retries independently.