Metadata Design for Retrieval

Every retrieval system eventually needs a field it did not capture. The useful way to choose what to store is not “what will we need?” — nobody knows — but “what becomes unrecoverable the moment ingestion finishes?”

The only question that matters

Sort every candidate field into one of three buckets, and the answer falls out.

• Unrecoverable. Facts about the act of ingestion: where the bytes came from, when you fetched them, what the source said the last modification was, which parser version produced the text. Once the crawl is over, this information does not exist anywhere. Capture all of it, always, even the parts with no current use.
• Expensive to recompute. Anything that required a model call: a summary, an entity list, a topic label. Recoverable in principle, at the cost of running the whole corpus through again. Capture it if you have any reason to think you will want it.
• Cheap to recompute. Token counts, language detection, hashes, headings parsed from the text. Derivable from stored text in one pass. Capture for convenience, not from fear.

Almost every regret in this area is a bucket-one field somebody left out. “Which version of the parser produced this text?” is unanswerable a year later, and it is the question you need when deciding which documents a parser fix should re-run.

The counter-argument is that unused fields are clutter, and it is a real cost — a schema with forty columns nobody reads is harder to understand than one with twelve. The resolution is to keep the unrecoverable fields on the document, where they are written once and read rarely, and to be much stricter about what gets copied onto the chunk, which is the row that exists millions of times and that every query touches. Storage on the document table is cheap; a field on the chunk table is a decision about the size of your hot working set.

The fields, by recoverability

Pre-filter, post-filter, and why it matters

Metadata earns its place at query time by narrowing the search, and there are two ways to combine a filter with a vector search that behave very differently.

Post-filtering retrieves the top k by similarity and then discards the ones that fail the filter. Simple, and it has a failure mode that is severe and quiet: if the filter is selective — one customer out of five thousand — the top 100 by similarity may contain zero matching rows, and the query returns nothing at all despite plenty of relevant documents existing.

Pre-filtering restricts the candidate set before the vector search. Correct, and harder to implement, because a graph index like HNSW navigates by following edges — restricting the node set can disconnect the graph and make the search fail to reach the region it should. Vector stores handle this differently and it is worth knowing which yours does; with pgvector you also have the option of an exact scan over a small filtered subset, which is often faster than an approximate search anyway.

-- pgvector: the filter is a WHERE clause, so the planner decides.
-- With a selective tenant filter and a btree index on tenant_id, an
-- exact scan of that tenant's rows can beat the HNSW index entirely.
SELECT c.id, c.text, e.vec <=> $1 AS distance
FROM   chunks c JOIN embeddings e ON e.chunk_id = c.id
WHERE  c.tenant_id = $2
  AND  c.language  = 'en'
  AND  c.effective_from <= now()
ORDER  BY distance
LIMIT  20;

Design the fields for this. Low-cardinality categoricals (language, document type, tenant) filter well; free text does not. The filtering strategies in more depth are worth reading before you decide what the schema looks like, because the schema is what constrains them.

There is a design habit that follows from the post-filtering failure above, and it is worth adopting as a rule: any field a query will filter on must be on the chunk row, not looked up through a join to the document. Not for performance — the join is usually cheap — but because the filter has to be expressible in whatever query language the vector store offers, and most of them cannot join. Denormalising tenant, language and access labels onto the chunk is redundancy you accept on purpose, and the price of it is that changing one of those values means updating every chunk of the document rather than one row.

Three different dates

Time is where metadata design most often goes wrong, because one updated_at column is asked to mean three things:

• Ingestion time — when your pipeline processed it. Answers “how fresh is the index?” and nothing about the content.
• Source modification time — when the document last changed upstream. Drives what to re-fetch.
• Effective time — when the content is true about. A policy dated 1 January, ingested in March, revised in April. This is the one users mean by “the current policy”, and it is the only one that cannot be derived from file metadata: it comes from the document body, from the URL, or from a human.

Store all three. Ranking by ingestion time when the user meant effective time is a bug that produces confidently wrong answers, and it is invisible until somebody notices the assistant citing a superseded policy that happens to have been re-crawled yesterday.

Access labels belong on the chunk

If any document in the corpus is not visible to every user, the access label must live on the retrievable unit and be applied as a pre-filter. Filtering after generation is not a control at all — the content reached the model, and therefore the answer, and therefore possibly the logs.

Store the label as a stable identifier — a group id, a classification level — rather than a resolved user list, which goes stale the moment somebody changes teams. Resolve the user’s groups at query time and filter on the intersection. And store the source system’s own identifier for the permission alongside it, so that when the answer to “why could this person see that?” is needed, it can be traced back rather than reconstructed. Tenant isolation has the same shape and the same consequence for getting it wrong.

One more field belongs in the same category and is almost always forgotten: the reason a document is in the corpus at all. A collection or ingest_reason tag — which crawl, which upload, which integration put it here — is what lets you remove a source cleanly when a contract ends, a customer leaves, or somebody discovers that an automated import has been running against the wrong folder for two months. Without it the removal is a guess reconstructed from URL patterns, and guesses either leave documents behind or delete somebody else’s.