Multi-Tenant RAG Without Leaking Between Customers

Multi-tenant retrieval has one catastrophic failure mode and it is not subtle: customer A is shown customer B’s document. What is subtle is that the leak usually does not happen in the retrieval query, which is the only place anybody looks.

Three isolation levels

The important distinction between the first two is what happens when the tenant is absent. A filter is an optional narrowing: omit it and you get everything. A namespace is part of the lookup: omit it and the call fails. Prefer designs where the failure mode of forgetting is an exception rather than a superset.

Make the unfiltered query impossible

If your retrieval function can be called without a tenant, it will be — in a background job, a migration script, an admin tool, or the endpoint someone adds in a hurry. The fix is not a code review checklist. It is a type.

class TenantIndex:
    """The only object in the codebase that can run a search."""

    def __init__(self, store, tenant_id: str):
        if not tenant_id:
            raise ValueError("tenant_id is required")
        self._store, self._tenant = store, tenant_id

    def search(self, query, k=10, where=None):
        scoped = {"tenant_id": self._tenant, **(where or {})}
        #        ^ tenant first, then caller filters; a caller CANNOT
        #          override it because we rebuild the dict either way
        return self._store.query(query, k=k, filter=scoped)


# Anywhere else in the application:
#   index = TenantIndex(store, request.tenant_id)
# The raw store is private to this module. There is no other door.

Note the dict ordering. Writing where first and tenant_id second would let a caller pass tenant_id in where and override the scope, which is a real bug that has shipped in real systems. Building the tenant key last is one character of difference and the whole guarantee.

Then enforce it structurally: the raw store client is module-private and no other module imports it. A grep for the store’s import across the codebase should return exactly one file, and that invariant is easy to assert in CI.

Four leaks that are not the query

• The cache keyed on the query text. This is the one that gets people. A retrieval or answer cache keyed on sha256(query) will serve tenant A’s cached answer to tenant B who asked the same question — and generic questions (“what is our refund policy?”) are exactly the ones with high cache hit rates. Every cache key in a multi-tenant system must include the tenant. Every one.
• The reranker. If you send a batch of candidates to a reranking service, and any batching or caching layer in between groups requests, you have created a path where documents from different tenants travel together. Check that your reranker client does not batch across requests.
• Evaluation and logging. Your evaluation set was built from real queries and real chunks, and it is now sitting in a repository that every engineer can read, and possibly in a prompt you send to a third-party judge model. Same for traces containing full retrieved context. This is not a retrieval bug; it is still a data leak.
• Error messages and counts. “No results in your 4,182 documents” leaks a number. A stack trace containing a namespace name leaks a customer list. Aggregate telemetry that spans tenants is fine; anything returned to a user must be scoped.

Shared documents, the awkward case

Most real products have three tiers of content: global (your product documentation), group (everything belonging to an organisation), and private (one user’s uploads). A single equality filter cannot express that, and the naive fix — an OR across scopes — is precisely the query shape most likely to be written wrong.

Two workable designs. Either store a repeated scopes field on each chunk and filter with a set-membership predicate against the caller’s allowed scope list, which keeps it to one query at the cost of needing a store that supports array containment. Or run one search per scope and fuse the rankings with RRF, which is more calls but has a useful property: you can guarantee the global corpus never crowds out the tenant’s own documents by reserving slots per scope.

That second property matters more than it sounds. A shared corpus is usually much larger than any one tenant’s, so a single fused top-k drifts toward global content, and the customer’s own uploaded document — the entire reason they are using the feature — ranks eleventh.

Checks worth automating

• An integration test that seeds two tenants with deliberately similar documents and asserts that a query from one never returns a chunk id belonging to the other. Run it on every commit; it is the single highest-value test in the system.
• A CI assertion that the vector store client is imported in exactly one module.
• A test that every cache key construction includes a tenant component. If your cache keys are built in one helper, this is one assertion; if they are built in fifteen places, fix that first.
• A periodic audit that samples chunks and confirms their tenant_id matches the document’s owner in the source system. Ingest bugs put the wrong tenant on a chunk, and no query- side control catches that — the filter works perfectly and returns the wrong document.

Two operational consequences follow from the isolation choice and are worth deciding up front rather than discovering. The first is cost attribution: with a shared index you cannot say what a given customer costs without tagging every request, and you will be asked, either by finance or by the first customer who wants usage-based pricing. Emit the tenant on every retrieval and generation call as a metric dimension from the beginning; retrofitting it means a quarter with no history.

The second is the largest tenant. Shared indexes degrade unevenly — one customer with fifty times everyone else’s documents makes their own queries slower and, depending on the store, everybody else’s too. Watch the distribution of chunk counts per tenant rather than the total, and have a plan for promoting an outlier to its own namespace or index before it becomes an incident. Migration of a single tenant is a routine operation if the manifest and the sync path were built per document; it is a project if they were not.