Enterprise Search: Permissions and Access Control

In enterprise search the corpus is not a fixed thing that everyone queries. Every user has their own, defined by permissions that change without telling you, and the difference between filtering before and after retrieval decides whether the product works.

Everyone sees a different corpus

A document in a company knowledge base is visible to some set of principals: users, groups, roles. A given employee can typically see a small fraction of the whole — a few percent is normal in a large organisation, and much less for anyone outside the central functions. Every ranking decision has to be conditioned on that, and the conditioning has to be exactly right, because the failure is not a bad result, it is a disclosure.

Two implementation orders are possible and they are not equivalent. Retrieve then filter, or filter then retrieve. The next section is why that ordering is the whole design.

Why post-filtering fails, in numbers

Let p be the fraction of the corpus this user can see, and k the number of candidates retrieved before the permission check. If permission is roughly independent of relevance:

E[visible results] = k * p
P(zero visible)    = (1 - p)^k

Take a user who can see 1% of the corpus and a retriever that returns 100 candidates:

p = 0.01, k = 100

E[visible]      = 100 * 0.01 = 1 result
P(zero visible) = 0.99^100   = 0.366

Better than a third of queries return nothing at all, and the
average query returns one result.

To expect 10 visible results:  k = 10 / 0.01 = 1,000 candidates.

And that is the optimistic model. Permission is usually correlated with topic — the documents most relevant to a finance query are the ones finance can see — so for a user asking about something outside their area the effective p is far below their average. The empty page appears exactly when the query is most interesting.

Raising k is the obvious response and it is a bad one. It multiplies retrieval cost by the reciprocal of the visibility fraction, it still fails for the least-privileged users, and for approximate nearest-neighbour retrieval it does not even reliably work, because a graph index does not simply return more of the same distribution as k grows. The filtered-search behaviour of HNSW is the reason vector databases grew explicit filter support rather than leaving it to the caller.

Putting the ACL in the index

The fix is to make permission part of the query rather than a check after it. Store an array of allowed principals on each document and intersect it with the querying user’s principal set:

document
  id:          "doc-4812"
  allow:       ["group:finance", "group:leadership", "user:alice"]
  deny:        ["user:bob"]

query for a user whose expanded principals are
  ["user:bob", "group:eng", "group:all-staff"]

  filter:  allow INTERSECTS principals
      AND  NOT (deny INTERSECTS principals)

Now the retriever only ever considers documents the user can see, and the recall arithmetic above disappears entirely: k candidates means k usable results. Three details make the difference between this working and this being slow or wrong.

• Deny is evaluated after allow, and wins. An allow list composes with OR; a deny list composes with AND NOT and must be applied last. Getting the order wrong produces a filter that is correct on every test case anyone thinks to write and wrong on the one that matters.
• Group membership must be expanded transitively. Groups contain groups. A user’s effective principal set is the transitive closure, and it can be large — a query carrying a thousand-term OR clause is slow enough to matter. Cache the expanded set per user with a short TTL, and treat the TTL as a security parameter, because it is exactly how long a revoked group membership keeps working.
• Denormalise to opaque tokens where you can. Rather than storing organisational structure in the search index, store opaque access tokens computed by the source system, and give the user the set of tokens they hold. The index then knows nothing about your org chart, which is both faster and one fewer copy of sensitive structure to keep in sync.

The multi-tenant version of this same problem — where the filter is a tenant rather than a principal set, and a leak crosses a customer boundary — is worked through in multi-tenant RAG, and the pre-filter versus post-filter trade-off in a vector index is in metadata filtering.

The index is always slightly wrong

The permissions in your index are a copy, and a copy of a mutable thing is stale between updates. Somebody is removed from a group at 09:00 and the reindex runs at 09:15; for fifteen minutes the index believes something false, and it believes it in the unsafe direction.

The standard answer is late binding: treat the indexed ACL as a pre-filter whose job is recall and performance, and re-check the top results against the source of truth before rendering them. Checking ten documents against an authorisation service is affordable in a way that checking a million is not, which is what makes the two-layer design work at all. Budget for it in the latency plan — it is a synchronous call in the request path, and it belongs in the arithmetic in search latency.

Two supporting habits. Drive permission updates by change events from the source system rather than by a periodic full crawl, so the window is seconds rather than the crawl interval. And make deletion and revocation a priority path that jumps the indexing queue: adding a document late is an inconvenience, removing one late is an incident.

Leaks that survive a correct filter

A perfectly correct permission filter still leaks through channels around the result list, and these are the ones that get found in audits.

• Snippets from the index. If the snippet is generated from indexed text rather than fetched at render time, a document deleted or restricted at the source can still show its content in a result whose link 404s. The link being dead is not the protection — the snippet was the payload.
• Autocomplete built from everyone’s queries. A suggestion index built from the whole organisation’s query log will happily complete a prefix into the codename of a project the typist has never heard of. Suggestions need the same k-anonymity threshold as any public system and, in an enterprise, scoping to something narrower than the whole company — the autocomplete page covers the threshold argument.
• Facet counts. “Department: Legal (3)” tells you three documents exist that you cannot see. Counts must be computed over the filtered set, which is more expensive and is the only correct answer.
• Total result counts and pagination. The same disclosure at a coarser grain. Reporting a total computed before filtering tells the user how much is being hidden.
• Timing. If a query for a restricted term is measurably slower than one for a nonexistent term, the difference is an oracle. This matters in genuinely adversarial settings and is usually acceptable risk internally — but it should be a decision somebody made, not an accident.