Metadata Filtering: The Part of RAG Everyone Skips
5 min read · updated August 3, 2026
A surprising share of retrieval failures are not similarity failures at all. The system found a genuinely relevant chunk — about the wrong product version, the wrong year, the wrong customer. No embedding model fixes that, because the chunk really is similar.
Most bad retrievals are the wrong scope
Go through a log of bad answers and classify them. A large fraction will be scope errors: a deprecated document that reads exactly like the current one, a v2 migration guide answering a v3 question, an archived policy that was never removed from the index. These are invisible to similarity because similarity is measuring the thing they have in common.
The fix is structured: attach version, effective_date, status, tenant_id, doc_type, language at ingest and constrain the search. It is unglamorous and it typically removes more wrong answers per hour of work than any change to the embedding model.
Three places the filter can go
| Approach | Description |
|---|---|
| post-filter | Retrieve top-n by similarity, then discard the rows that fail the predicate. Works with any store, requires no index support, and silently returns fewer than k results — sometimes zero — whenever the filter is selective. |
| pre-filter | Evaluate the predicate first to get an allowed id set, then search only within it. Exact by construction. Cheap when the set is small enough to score by brute force; expensive or impossible when it is millions of rows, because you have given up the approximate index. |
| filtered traversal | What production vector stores actually do: carry the predicate into the graph or inverted-list search so that only permitted nodes count as results, while still traversing through non-matching nodes to stay connected. Pinecone describes its version as single-stage filtering; the research line includes Filtered-DiskANN (Gollapudi et al., WWW 2023) and later predicate-aware traversal work. |
The over-fetch arithmetic
Post-filtering is the default in hand-rolled pipelines because it is three lines of code, and its cost is easy to underestimate. Model it as sampling. If a fraction p of the corpus passes the filter and the filter is roughly independent of similarity, then fetching n candidates yields about n × p survivors.
want k survivors, filter selectivity p -> fetch n ≈ k / p k = 10 p = 0.50 -> n = 20 fine k = 10 p = 0.10 -> n = 100 noticeable k = 10 p = 0.01 -> n = 1000 you are scanning the index k = 10 p = 0.001 -> n = 10000 post-filtering has failed
And n × p is only the expectation. At p = 0.01 and n = 100 you expect one survivor, but the variance means a fair number of queries return zero — which surfaces as an intermittent “I don’t have information about that” on a question the corpus clearly answers. Intermittency is the signature. If your bug report says the same question sometimes works, look at the filter before you look at the model.
Note also that the independence assumption is optimistic. A tenant’s documents are usually more similar to each other than to the corpus average, which can help; a filter on status = current correlates with recency and therefore with phrasing, which can hurt. Neither direction is predictable, which is another reason to prefer a filter the index understands.
Why the index cannot just intersect
The natural question is why a vector store cannot simply skip disallowed nodes. The answer is in the data structure. HNSW is a navigable small-world graph: search starts at an entry point and walks greedily toward the query, and the walk depends on the graph being well connected. Delete 99% of the nodes and the surviving 1% is typically not a connected graph at all — the greedy walk gets stuck in a local pocket and returns whatever it could reach, with recall that can collapse without any error being raised.
Filtered traversal implementations get around this by letting the walk pass through non-matching nodes while only collecting matching ones, and by falling back to brute force when the predicate is selective enough that scanning the allowed set is cheaper than navigating. The practical consequence for you is that filter behaviour is a per-store property with real recall implications, and it is worth reading your vendor’s documentation on it rather than assuming exactness. Some stores document a recall degradation under selective filters; some fall back silently; the difference matters.
Designing the metadata
- Low cardinality, high selectivity. The useful fields are the ones with a handful of values that carve the corpus into meaningful regions — tenant, language, doc type, status. A field with a million distinct values is an identifier, not a filter.
- Denormalise onto the chunk. Metadata lives on the chunk row, not on a document table you join to afterwards, because a join after retrieval is a post-filter by another name.
- Store dates as numbers. Range predicates on an integer epoch are supported by every store; range predicates on an ISO string are supported by some.
- Make the security filter non-optional in code. A tenant or ACL predicate that any caller can forget to pass is a data breach with a deploy schedule. The filter belongs inside the retrieval function’s signature, not in its arguments.
- Keep an unfiltered escape hatch for debugging. When a query returns nothing, the first thing you need to know is whether the chunk was missing or excluded, and that is one query with the predicate removed.
The remaining question is where the metadata comes from. Three sources, in descending order of reliability. The best is the system of record — a CMS already knows a page’s status, author, product and locale, and copying those fields at ingest is free and correct. The second is the document itself: a file path, a front-matter block, a heading hierarchy or a filename convention will carry most of what you need if the corpus has any discipline at all. The third is extraction with a model at ingest, which is the only option for unstructured sources and the least trustworthy — treat an extracted effective date as a hint, not as a fact you are willing to filter exact matches on.
One filter deserves special mention because it is nearly always available and nearly always skipped: recency as a soft signal rather than a hard predicate. Instead of excluding old documents, multiply the retrieval score by a decay factor, or rerank with the date in the reranker’s input. A hard cut-off is brittle — it silently loses the one still-authoritative document from 2019 — while a soft preference degrades gracefully and is much harder to get badly wrong.