Triggering an Embedding Pipeline on S3 Object Upload

A document lands in a bucket and should be chunked, embedded and indexed. The wiring is one API call. The two properties of that wiring which decide whether the pipeline is correct are documented in a warning box and a single sentence, and both are easy to read past.

The wiring

S3 event notifications are a bucket subresource. You give S3 a list of event types, an optional filter, and a destination — an SQS queue, an SNS topic, a Lambda function, or EventBridge:

aws lambda add-permission \
  --function-name embed-document \
  --statement-id s3-invoke \
  --action lambda:InvokeFunction \
  --principal s3.amazonaws.com \
  --source-arn arn:aws:s3:::corpus-inbound \
  --source-account 111122223333

aws s3api put-bucket-notification-configuration \
  --bucket corpus-inbound \
  --notification-configuration '{
    "LambdaFunctionConfigurations": [
      {
        "Id": "embed-on-upload",
        "LambdaFunctionArn": "arn:aws:lambda:us-east-1:111122223333:function:embed-document",
        "Events": ["s3:ObjectCreated:*"],
        "Filter": {
          "Key": {
            "FilterRules": [
              { "Name": "prefix", "Value": "inbound/" },
              { "Name": "suffix", "Value": ".pdf" }
            ]
          }
        }
      }
    ]
  }'

The add-permission call has to come first — PutBucketNotificationConfiguration validates that the destination will accept the invocation and fails if it will not. Both --source-arn and --source-account should be present: without them the resource policy allows any bucket in any account to invoke your function, which is a confused-deputy problem rather than a theoretical one.

s3:ObjectCreated:* covers Put, Post, Copy and CompleteMultipartUpload. That last one is why you want the wildcard rather than s3:ObjectCreated:Put: any upload above the SDK’s multipart threshold, which is most documents of interest, completes as a multipart upload and does not fire Put at all. A pipeline that works in testing with small files and silently ignores large ones in production is almost always this.

Note also that PutBucketNotificationConfiguration replaces the entire configuration rather than appending to it. Two teams each adding “their” notification through the CLI will silently delete each other’s. And overlapping prefix filters within one configuration are rejected, so you cannot have both inbound/ and inbound/pdf/ as separate rules.

The loop that bills you until you notice

AWS puts this in a warning box, and it is the single most expensive mistake available here: if the notification writes to the same bucket that triggers it, the function indirectly triggers itself. An embedding pipeline is unusually prone to it, because writing extracted text, chunk manifests or a JSON sidecar back beside the source document is the obvious thing to do.

The loop does not announce itself. Each generation of objects is smaller than the last only if you are lucky; if the function writes one sidecar per input, it is a constant-rate loop that runs until your concurrency limit, your embedding provider’s rate limit or your patience gives out. Every iteration is a paid model call.

Two mitigations, and AWS names both. Use two buckets — inbound and derived — which is unambiguous and cannot be broken by a later prefix change. Or configure the trigger to apply only to a prefix used for incoming objects, which works but is one careless put_object away from failing. If you take the prefix route, write outputs under a prefix that could not possibly match the filter, and add a guard at the top of the handler that returns immediately when the key does not look like an input.

At least once, in no particular order

AWS states that event notifications are designed to be delivered at least once, and that they typically arrive within seconds but can sometimes take a minute or longer. Both halves matter.

At least once means duplicates are normal operation, not an incident. The same object can be delivered twice, and your function will happily embed it twice, pay twice, and write two sets of vectors for the same content — which then both come back in a similarity search and crowd out other results. The next section is about that.

No ordering guarantee means a delete and a re-upload of the same key can arrive in either order. If they arrive reversed, the delete handler removes vectors that the create handler had just written, and the document silently disappears from the index while sitting perfectly intact in the bucket. Where that matters, enable bucket versioning and use the version id to establish order, or route through EventBridge to an SQS FIFO queue — AWS notes that FIFO queues are not supported as a direct S3 notification destination but are reachable through EventBridge.

Making the embed step idempotent

The fix for at-least-once delivery is a deterministic identity for each chunk, so that a second delivery overwrites the first instead of adding to it:

1. Read the identity out of the event record. With versioning on, the triple of bucket, key and s3.object.versionId is exact. Without it, use s3.object.eTag, which changes when the content changes.
2. Derive each vector’s key from that plus the chunk index: sha256(bucket + key + versionId) + ":" + index. The same input document always produces the same set of keys.
3. Upsert rather than insert. A vector store keyed on that string replaces on a second delivery; one that assigns its own ids accumulates. This is the whole decision — everything else is wiring.
4. Delete the previous version’s chunks by prefix before writing the new ones, because a shorter revision produces fewer chunks and the surplus from the old version would otherwise survive.

Skip the naive alternative of a “have I seen this event id” table. It is another round trip, it needs its own TTL, and it answers a weaker question than the content hash does — the hash also gives you the re-upload case for free.

When Lambda is the wrong consumer

Direct Lambda invocation is the shortest path and it has a hard ceiling: fifteen minutes, and concurrency that scales with the upload rate rather than with what your embedding provider will accept. Upload a thousand documents at once and S3 will cheerfully invoke a thousand concurrent functions, all of which will hit the provider’s rate limit simultaneously and fail together.

• Put SQS in between. Notify a queue rather than the function, and set the event source mapping’s MaximumConcurrency to something your provider tolerates. The queue absorbs the spike, and a failed embed is redelivered rather than lost. Pair it with a dead-letter queue and scale the consumer on queue depth.
• Hand long documents to Step Functions. A 300-page PDF that needs OCR, chunking and several hundred embedding calls does not belong in one function invocation. A workflow gives you per-step retries and catch branches and a Map state whose concurrency you control.
• Use EventBridge when the routing gets interesting. Enable it on the bucket and you get content-based filtering, multiple independent targets for one event, and a path to FIFO ordering — at the cost of an extra hop and EventBridge’s own quotas.

Whichever consumer you pick, the destination for the vectors is a separate decision; S3 Vectors is the low-cost option that keeps everything in one service, and a Bedrock knowledge base is the option that replaces this pipeline with a managed one.