Handling Files Users Upload to an AI Feature

The moment your AI feature accepts a file, you are running an untrusted input pipeline. The interesting failures are not exotic: a 4 GB upload that fills the disk, an HTML file served back from your own domain, a PDF that expands to forty gigabytes of text, and a quota that two simultaneous uploads walk straight past.

The shape of a safe upload

1. Client asks your API for permission to upload, declaring the filename and byte size. Your API checks the user’s quota and the declared size, records an intent row, and returns a presigned upload target scoped to one key in a quarantine bucket.
2. Client uploads directly to object storage. The bytes never touch your application servers, which removes request-size limits, memory pressure and a whole class of denial of service from your API.
3. Client notifies your API, or a bucket event does. Do not trust the notification alone — verify the object exists and read its real size from the store rather than from the client.
4. Your worker inspects, scans and promotes the object from quarantine to the main bucket, then records the document row and enqueues extraction.

Everything after this section is a detail of one of those four steps, and each detail is one that has produced an incident somewhere.

The size cap that is actually enforced

This is the trap. A presigned PUT URL authorises a write to a key; unless you signed the Content-Length header specifically, the client may upload any number of bytes to that key. A client that declares 2 MB and uploads 4 GB succeeds, and your first indication is the storage bill or the extraction worker running out of memory.

A presigned POST policy is different: the policy document can contain a content-length-range condition, and the storage service rejects an upload outside it before storing anything.

# Presigned POST with an enforced size range — this is the one
# that actually caps the upload.
import boto3

s3 = boto3.client("s3")
presigned = s3.generate_presigned_post(
    Bucket="uploads-quarantine",
    Key=f"{tenant_id}/{upload_id}/{safe_name}",
    Fields={"Content-Type": declared_type},
    Conditions=[
        {"Content-Type": declared_type},
        ["content-length-range", 1, 25 * 1024 * 1024],   # 1 byte .. 25 MB
    ],
    ExpiresIn=300,      # five minutes; long enough to upload, short
                        # enough that a leaked URL is nearly worthless
)
# presigned["url"] and presigned["fields"] go to the client as a
# multipart/form-data POST target.

If your storage provider or client library requires PUT, the fallback is defence in depth rather than prevention: sign a short expiry, and have the worker read the object’s real size from a HEAD request before doing anything with it, deleting and rejecting anything oversized. That means you pay to store the oversized object briefly, which is much better than processing it.

head = s3.head_object(Bucket=QUARANTINE, Key=key)
if head["ContentLength"] > MAX_BYTES:
    s3.delete_object(Bucket=QUARANTINE, Key=key)
    reject(upload_id, "file exceeds the 25 MB limit")
    return

Sanitise the filename before it becomes part of a key. Take the basename, strip anything that is not alphanumeric, dot, dash or underscore, cap the length, and keep the original in a database column for display. A filename is user input and it has been an attack surface since long before object storage existed.

Deciding what the file is

Three sources claim to tell you the type, and only one of them is evidence.

MAGIC = {
    b"%PDF-":            "application/pdf",
    b"PK\x03\x04":       "application/zip",   # also .docx, .xlsx, .pptx
    b"\x89PNG\r\n\x1a\n": "image/png",
    b"\xff\xd8\xff":     "image/jpeg",
    b"GIF87a":           "image/gif",
    b"GIF89a":           "image/gif",
}

def sniff(first_bytes: bytes) -> str | None:
    for magic, mime in MAGIC.items():
        if first_bytes.startswith(magic):
            return mime
    return None

# Fetch only the first 512 bytes; you do not need the file to decide.
head = s3.get_object(Bucket=QUARANTINE, Key=key, Range="bytes=0-511")
detected = sniff(head["Body"].read())
if detected not in ALLOWED_TYPES:
    reject(upload_id, "unsupported file type")

Allow-list, never deny-list. A deny-list is a claim that you have thought of every dangerous format, which nobody has. And note the zip signature covers .docx, .xlsx and .pptx as well as plain archives, so accepting Office documents means accepting a zip container and everything that implies — see the extraction section.

Two rules for serving files back, both of which prevent stored cross-site scripting. Set Content-Disposition: attachment on anything a user uploaded, and serve user content from a different origin than your application. A user-uploaded HTML file served inline from your own domain executes with your session cookies, which is about as bad as it gets.

Quarantine, scan, promote

Two buckets. Uploads land in quarantine, which nothing except the worker can read. Only after inspection does the object move to the bucket the application serves from.

# ClamAV, streamed, with a size cap so a large file cannot
# exhaust the scanner's memory.
clamdscan --fdpass --stdout /tmp/quarantine/<key>

# Exit codes: 0 clean, 1 infected, 2 error.
# Treat 2 as "do not promote" — a scanner that errored has told
# you nothing, and "we could not check" is not "it is fine".

Malware scanning genuinely matters for a document pipeline, and not only for your own machines. Files uploaded to an AI feature are frequently downloaded again by other users of the same workspace, which makes you a distribution channel if you do not check.

Record the state transitions in the database so a stuck upload is visible and a retry is safe:

CREATE TABLE uploads (
  id            uuid PRIMARY KEY,
  tenant_id     uuid NOT NULL,
  user_id       uuid NOT NULL,
  original_name text NOT NULL,
  declared_type text NOT NULL,
  declared_bytes bigint NOT NULL,
  actual_bytes  bigint,
  detected_type text,
  sha256        bytea,
  state         text NOT NULL DEFAULT 'intent',
                -- intent -> uploaded -> scanned -> promoted -> extracted
                -- with terminal states: rejected, failed
  reject_reason text,
  quarantine_key text NOT NULL,
  object_key    text,
  created_at    timestamptz NOT NULL DEFAULT now(),
  updated_at    timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX uploads_state_idx ON uploads (state, created_at)
  WHERE state NOT IN ('promoted', 'extracted', 'rejected');

The partial index is small because it covers only in-flight rows, and it makes “what is stuck” a fast query on a table that grows forever.

Quotas that cannot be raced

The natural implementation is to sum the user’s current usage, compare it with the limit, and then insert. Two uploads arriving together both read the pre-insert total and both pass, which is a textbook race and it is exploited by accident constantly — a user selecting forty files in a picker issues forty concurrent requests.

Make the check and the increment one atomic statement, with the limit expressed as a constraint the database enforces:

CREATE TABLE storage_quota (
  user_id     uuid PRIMARY KEY,
  used_bytes  bigint NOT NULL DEFAULT 0,
  limit_bytes bigint NOT NULL,
  CONSTRAINT within_quota CHECK (used_bytes <= limit_bytes)
);

-- Reserve space. Either this updates one row, or the CHECK fails
-- and the transaction rolls back. There is no window between the
-- test and the write, because they are the same statement.
UPDATE storage_quota
SET used_bytes = used_bytes + $2
WHERE user_id = $1
RETURNING used_bytes, limit_bytes;
-- On constraint violation (SQLSTATE 23514), return 413 to the client
-- and do not issue a presigned URL.

Reserve at intent time using the declared size, and reconcile at promotion time with the actual size — releasing the difference, or releasing the whole reservation if the upload was abandoned. A scheduled job releases reservations for intent rows older than an hour, which is what keeps an abandoned upload from consuming a user’s quota forever.

-- Reconcile at promotion.
UPDATE storage_quota
SET used_bytes = used_bytes - $declared + $actual
WHERE user_id = $1;

-- Release abandoned intents, hourly.
UPDATE storage_quota q
SET used_bytes = greatest(0, q.used_bytes - u.declared_bytes)
FROM uploads u
WHERE u.user_id = q.user_id
  AND u.state = 'intent'
  AND u.created_at < now() - interval '1 hour';

greatest(0, …) is there because a reconciliation bug that drives the counter negative should degrade to zero rather than to a quota nobody can exceed. The check constraint catches the other direction.

Extraction is where the bombs go off

A 200 kB file can become forty gigabytes of extracted text. The size cap on the upload does not bound the output, and the extraction worker is usually the process with the least defensive code in the system.

• Cap the output, not just the input. Stop reading at a byte limit and mark the document truncated. Every extraction library that streams can be wrapped in a counter; one that only returns a complete string should be run against a bounded temporary file.
• Cap the page and entry count. A PDF with fifty thousand pages, or a zip with a million entries, exhausts memory through object count rather than bytes. Refuse above a threshold rather than discovering it.
• Never follow references out of the document. XML external entities, remote images, embedded URLs. An XML parser with entity resolution enabled will fetch a URL of the attacker’s choosing from inside your network, which is server-side request forgery delivered as a document.
• Run it in a sandbox with a wall-clock limit. Extraction libraries parse hostile input in C. Give the process its own container, no network, a memory cap and a timeout, and treat a crash as a rejection rather than as a retry.
• Treat extracted text as untrusted input to the model. A document containing instructions aimed at your assistant is prompt injection with a delivery mechanism, and the upload path is the most common one. Extraction is a data boundary, not a trust boundary.

Retention and deletion

Decide the retention period before launch, because retrofitting one to a bucket with three years of files in it is a project. Two mechanisms, and you want both, because they fail in different ways.

{
  "Rules": [
    { "ID": "quarantine-expire",
      "Filter": { "Prefix": "" },
      "Status": "Enabled",
      "Expiration": { "Days": 1 } },
    { "ID": "abort-incomplete",
      "Filter": { "Prefix": "" },
      "Status": "Enabled",
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 1 } }
  ]
}

The quarantine bucket expires everything after a day: anything not promoted in that time was abandoned or rejected, and a bucket of unscanned user files with no expiry is a liability that grows on its own. The main bucket’s lifecycle follows your product’s retention promise, backed by a database job that deletes the row, the object and everything derived from it — deletion that reaches the vector index is the rest of that procedure, and it includes the step people miss, which is that a versioned bucket keeps the old version until you delete the version rather than the object.

One more thing belongs in the retention conversation and rarely makes it: what the model provider keeps. A file the user uploaded, extracted into text and sent as part of a prompt has left your storage boundary, and the provider’s own retention window now applies to it independently of yours. If your product promises deletion within thirty days, that promise covers a copy you do not control unless you have checked the terms and, where it is offered, opted out of retention. GDPR and AI APIs covers the processor relationship; the operational point here is that the retention policy for uploads has at least two systems in it and the documentation should name both.