Generating Presigned S3 URLs for a RAG Upload Flow

A presigned URL lets a browser upload straight to S3 with no AWS credentials of its own and no bytes through your API. It is a bearer token with a signature, and almost everything that goes wrong with one comes from the credential that signed it rather than from the URL.

The shape of the flow

Three parties. The browser asks your API for permission to upload. Your API — which does have AWS credentials — decides whether this user may, picks the object key, and signs. The browser uploads directly to S3 using the signature. Your API never sees the file.

The capability of the URL is bounded by the permissions of the principal who created it. AWS is explicit that anyone with valid credentials can create a presigned URL, but that access only succeeds if the creator had permission for the operation. So the signing role should hold exactly s3:PutObject on exactly the ingestion prefix, and nothing else — a broad role signing a narrow URL is one code change away from signing a broad one.

Pick the key server-side. If the client supplies it, a user can write over another tenant’s document by guessing a path, and no amount of expiry tuning helps. A key of inbound/<tenantId>/<uuid>/<sanitised-filename> keeps the original name for the user and makes collisions and traversal impossible.

What actually decides expiry

The rule that explains most confused bug reports: a presigned URL expires at its configured expiry or when the credentials that signed it expire, whichever comes first. AWS documents the practical consequences by credential type:

• IAM user credentials with SigV4 — valid up to 7 days, the protocol maximum.
• STS AssumeRole credentials — the URL expires when the role session ends. AWS notes the default session is one hour.
• EC2 instance profile credentials — metadata credentials rotate with a maximum validity of roughly 6 hours.
• ECS task role credentials — AWS states these typically rotate every 1–6 hours.

Read that against the normal architecture. Your signing API runs in Lambda or on ECS, so it is using role credentials, so the seven-day expiry you passed is a ceiling you will never reach. Asking for 604,800 seconds from a Lambda produces a URL that stops working in under an hour with ExpiredToken, and the code looks completely correct. There is no configuration that fixes this, because it is the point of temporary credentials.

Which is fine, because an upload URL should be short-lived anyway. Sign for fifteen minutes, issue it at the moment the user picks a file, and re-request rather than caching. Where you genuinely need a long-lived link — a share URL for a report, say — the honest options are to re-sign on demand behind your own authorisation check, or to front the object with CloudFront and a signed URL that has its own key-pair-based expiry.

PUT signs a request; POST signs a policy

Two mechanisms share the name, and they are not interchangeable. A presigned PUT URL signs one specific request — method, bucket, key, and whichever headers you chose to sign. A presigned POST signs a policy document: a set of conditions the eventual multipart form must satisfy.

import boto3, uuid

s3 = boto3.client("s3")

# PUT: simplest, but nothing here can bound the file size.
put_url = s3.generate_presigned_url(
    ClientMethod="put_object",
    Params={
        "Bucket": "corpus-inbound",
        "Key": f"inbound/{tenant_id}/{uuid.uuid4()}/{safe_name}",
        "ContentType": "application/pdf",
    },
    ExpiresIn=900,
)

# POST: a signed policy, with conditions S3 enforces on upload.
post = s3.generate_presigned_post(
    Bucket="corpus-inbound",
    Key=f"inbound/{tenant_id}/{uuid.uuid4()}/${filename}",
    Fields={"Content-Type": "application/pdf"},
    Conditions=[
        {"Content-Type": "application/pdf"},
        ["content-length-range", 1, 26214400],   # 1 byte to 25 MiB
        ["starts-with", "$key", f"inbound/{tenant_id}/"],
    ],
    ExpiresIn=900,
)

The difference that decides which to use is content-length-range. A presigned PUT has no way to express a maximum object size: the signature covers the request, and the body length is not part of it. Hand someone a presigned PUT URL for an ingestion bucket and they can upload five gigabytes of anything, which your pipeline will then dutifully try to chunk and embed. The POST policy condition is enforced by S3 itself and rejects the upload before a byte is stored.

The cost is client complexity: the browser must build a multipart/form-data body containing every field S3 returned, in order, with the file last. That is a dozen lines with FormData, and it is the right trade for an endpoint the public can reach.

Constraining what can be uploaded

• Bound the size in the policy, not in the client. Client-side validation of file size is a UX affordance. The content-length-range condition is the control.
• Sign the content type and mean it. A signed Content-Type must match at upload, which stops the trivial substitution. It does not inspect the bytes — a declared PDF can be anything — so the extraction step still needs to fail safely on a file that is not what it claims.
• Pin the prefix with starts-with. Where the key template contains a client-supplied portion, a starts-with condition on $key keeps the upload inside the tenant’s space regardless of what the client sends.
• Age out old signatures at the bucket. The s3:signatureAge condition key lets a bucket policy deny any presigned request whose signature is older than a chosen number of milliseconds, independent of what expiry the signer chose. It is a useful backstop against a signing path that hands out longer expiries than you intended.
• Remember the URL is reusable. AWS documents that a presigned URL may be used multiple times until it expires. If one upload should mean one object, either make the key unique per issue — as above — or accept that an overwrite is possible and make the downstream idempotent.

CORS, and what happens next

A browser uploading cross-origin needs a CORS configuration on the bucket, or the request fails at the preflight with an error that says nothing about signatures:

aws s3api put-bucket-cors --bucket corpus-inbound --cors-configuration '{
  "CORSRules": [
    {
      "AllowedOrigins": ["https://app.example.com"],
      "AllowedMethods": ["PUT", "POST"],
      "AllowedHeaders": ["content-type", "x-amz-*"],
      "ExposeHeaders": ["ETag"],
      "MaxAgeSeconds": 3000
    }
  ]
}'

Exposing ETag is worth doing: it gives the client a content hash it can report back, which is a cheap way to confirm the upload matched what was picked. If a signature error persists after CORS is right, check the clock — AWS lists NTP drift first among the causes of SignatureDoesNotMatch, along with corporate proxies that rewrite headers a signature covered.

The moment the object lands, this flow is over and the ingestion side begins. Wire the bucket notification as described in triggering an embedding pipeline on S3 object upload — and note that the two-bucket rule from that page starts here: if your upload target and your derived-output target are the same bucket, the loop is already built.