Object Storage for Documents, Weights and Artefacts

Object storage is where the documents, the model weights and the evaluation artefacts go, and the bill for it has four line items of which only one is the bytes you stored. This page covers key layout that survives a reprocessing run, a lifecycle rule per artefact class, and the arithmetic of the two rules — minimum billable size and minimum storage duration — that make archiving small objects cost more than keeping them.

What belongs in a bucket and what does not

The rule is simple and frequently broken: bytes in the bucket, facts in the database. The PDF goes in S3; its SHA-256, its page count, its extracted text length, its processing status and the key it lives at go in Postgres. Anything you might need to query, filter, join or count belongs in a row.

Do not use the bucket as an index. ListObjectsV2 is a paginated scan of a lexicographically ordered keyspace, a thousand keys per request; answering “how many documents does tenant 42 have” by listing a prefix is a query that gets linearly slower forever and costs a request per page. The database answers it with an index scan.

Nor should the bucket be your source of truth for what has been processed. A key existing does not tell you whether the write finished, whether the content is what you think, or whether a retry wrote it twice. Those are transactional facts and they belong somewhere with transactions.

Key layout

Object keys are immutable in practice — renaming means copy plus delete, at full cost — so a layout is a decision you make once. What works:

raw/{tenant}/{document_uuid}/original.pdf
raw/{tenant}/{document_uuid}/meta.json

derived/{tenant}/{document_uuid}/{pipeline_version}/text.txt
derived/{tenant}/{document_uuid}/{pipeline_version}/pages/0001.png

weights/{model_name}/{version}/model.safetensors
weights/{model_name}/{version}/config.json

evals/{suite}/{run_id}/results.jsonl

Three properties make this layout worth copying.

• Raw is immutable and derived is disposable. Anything under derived/ can be deleted and regenerated. That single split is what lets you write one lifecycle rule that is aggressive and one that is conservative, instead of one that is a compromise.
• The pipeline version is in the path. Reprocess with a new extractor and you write to a new prefix; nothing is overwritten; the old outputs remain until you decide otherwise; and a rollback is a change to which prefix you read. Overwriting derived artefacts in place is the mistake that makes reprocessing scary.
• The tenant is high in the path. A tenant deletion becomes a prefix operation, and IAM policies can be written against raw/acme/*. If the tenant is buried three levels down, neither is possible.

The advice to prefix keys with a random hash for performance is obsolete. S3 has scaled per-prefix since 2018 — 3,500 write and 5,500 read requests per second per prefix, and it splits prefixes automatically under sustained load — so a readable hierarchy costs you nothing and a hashed one costs you the ability to reason about your own data.

Lifecycle rules per artefact class

One rule per prefix, expressing what that class of object is actually worth over time:

{
  "Rules": [
    {
      "ID": "derived-expire",
      "Filter": { "Prefix": "derived/" },
      "Status": "Enabled",
      "Expiration": { "Days": 90 }
    },
    {
      "ID": "raw-archive",
      "Filter": { "Prefix": "raw/" },
      "Status": "Enabled",
      "Transitions": [
        { "Days": 30,  "StorageClass": "STANDARD_IA" },
        { "Days": 180, "StorageClass": "GLACIER_IR" }
      ]
    },
    {
      "ID": "abort-incomplete-uploads",
      "Filter": { "Prefix": "" },
      "Status": "Enabled",
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
    }
  ]
}

That third rule is the one everybody discovers late. A failed multipart upload leaves its parts in the bucket, billed as storage, invisible to ListObjectsV2 because they are not objects yet. A pipeline that uploads large files and crashes occasionally will accumulate them for years. Enable the abort rule on every bucket you create, on day one, before it has any data.

Model weights get no lifecycle rule at all. A 40 GB checkpoint you might need to roll back to is worth its storage cost several times over, and the retrieval fee and latency of pulling it out of an archive class during an incident are exactly what you do not want.

The four line items

Model the egress before it happens, because it is the item with no natural ceiling. The shape is always:

egress_gb_per_month
  = documents_served × avg_size_gb × serves_per_document
  + backup_copies_to_other_region_gb
  + weights_pulled_gb × pulls_per_month

egress_cost = egress_gb_per_month × rate_per_gb_out   [rate: your price list]

Worked shape, 40 GB of weights pulled by a fleet:
  40 GB × 20 nodes × 4 deploys/month = 3,200 GB/month of egress
  — if the nodes are in a different region from the bucket.
  Same region: zero. This single architectural fact is usually
  worth more than every other optimisation on this page combined.

The request line is the one that catches AI pipelines specifically, because reprocessing multiplies it. Requests are billed per operation regardless of object size, so the cost is driven entirely by how many objects your pipeline touches — a quantity that is a design decision, not a property of your data.

requests_per_reprocessing_run
  = documents × (1 read of the original
               + writes_per_document
               + 1 lifecycle transition per object written)

Chunk-granularity layout, 100,000 documents × 40 chunks:
  100,000 reads + 4,000,000 writes = 4.1 M billable requests
  per reprocessing run.

Document-granularity layout, same corpus:
  100,000 reads + 100,000 writes = 0.2 M billable requests.

A factor of twenty, from one decision about what an object is.
Multiply by the PUT rate from your price list; on any plausible
one, the first arrangement costs more in requests than the
second costs in total.

The same arithmetic explains why ListObjectsV2 in a loop is expensive: it is billed at the write-class rate, a thousand keys per call, so listing a bucket with ten million objects is ten thousand billable requests and takes minutes. Every time you are tempted to answer a question by listing, check whether the database could answer it instead — it almost always can, and it does so for free.

The small-object trap, worked

Infrequent-access and archive classes carry two rules that make them the wrong choice for small objects, and the interaction is not obvious until you do the arithmetic.

Minimum billable object size. Objects in infrequent-access classes are billed at a floor — commonly 128 KB — so a 4 KB object is billed as 128 KB, a factor of 32.

Minimum storage duration. Each class carries a minimum billable period — 30 days for infrequent access, 90 for the instant-retrieval archive tier, 180 for deep archive. Delete before it elapses and you are billed for the remainder anyway.

Now take a realistic AI pipeline: ten million chunk records at 3 KB each, transitioned to infrequent access after 30 days.

Actual data:        10,000,000 × 3 KB      =  30 GB
Billed as:          10,000,000 × 128 KB    = 1,280 GB

Transition requests: 10,000,000 lifecycle transitions,
                     billed per thousand at the PUT-class rate.

So the "saving": pay the IA rate on 1,280 GB instead of the
standard rate on 30 GB, plus ten million transition requests,
plus a retrieval fee every time anything reads them back.

For any plausible price list, this is several times more
expensive than doing nothing at all.

The fix is not a different storage class. It is not to have ten million small objects: pack chunks into a columnar or line-delimited file per document or per batch — one Parquet or JSONL object of a few megabytes instead of thousands of tiny ones — and keep the per-chunk facts in the database where they belong. That change reduces storage cost, request cost and retrieval latency at once, and it is the reason the key layout above puts derived artefacts at document granularity rather than chunk granularity.

The general rule this is an instance of: an object should be something you fetch whole. If you never read one chunk without reading its neighbours, they were always one object. If you never read a page image without the document it came from, likewise. Objects sized to how you read them get the request count, the transfer volume and the minimum-billable-size rule all pointing the same way; objects sized to how you happened to produce them get all three pointing against you.

Consistency and the database link

S3 has been strongly read-after-write consistent since December 2020, which removes a large class of historical workarounds: a GET after a successful PUT returns the new object, and a listing reflects it. What remains is the harder problem, which is that the bucket and the database are two systems with no shared transaction.

Write the object first, then the row. That ordering leaves orphaned objects when the process dies in between — an object nobody references — which is recoverable by a sweeper that lists a prefix and deletes keys with no row. The other ordering leaves a row pointing at a key that does not exist, which is a broken document surfacing in a user request. One of those failure modes is a background job and the other is an incident.

-- The row that makes the sweeper possible.
CREATE TABLE documents (
  id           uuid PRIMARY KEY,
  tenant       text NOT NULL,
  object_key   text NOT NULL UNIQUE,
  sha256       bytea NOT NULL,
  bytes        bigint NOT NULL,
  content_type text NOT NULL,
  uploaded_at  timestamptz NOT NULL DEFAULT now()
);

-- Anything in the bucket older than a day with no row here is garbage.
-- Anything here whose key 404s is an incident.

The sha256 column is what makes reprocessing idempotent: it identifies content independently of the key it was stored under, so a re-upload of the same file is detectable and a document that was moved between prefixes is still recognisable. It is also what makes the sweeper safe to run: deleting an unreferenced key is only defensible if you can prove the key is unreferenced, and the proof is a query against a column that does not depend on the key. Give the sweeper a grace period of at least a day, so that an object written seconds before a crash is not deleted before the retry that would have claimed it. A documents table that survives re-indexing builds the rest of that schema.