Build an Internal Assistant Over Confluence and Drive

An internal knowledge assistant is a retrieval system where the hard requirement is not relevance but authorisation: it must answer from what this person is allowed to see, which is a different set from what exists. Get that wrong and the assistant surfaces a salary review to the whole engineering team — quietly, plausibly, and with a citation.

The constraint that shapes everything

In a public docs bot, retrieval quality is the product. Here, authorisation is the product and retrieval quality is a feature. That inversion changes three architectural decisions:

• The index is not a single corpus. Every chunk carries who may see it, and that field is used in the query, not after it.
• The identity of the asker is required. There is no anonymous mode, no shared service account, no “we will add auth later”. A prototype without identity teaches you nothing about the real system.
• Freshness has a security meaning. A stale document is an inconvenience; a stale permission is a breach. They need different refresh paths, which is the one way this differs from conventional enterprise search only in degree.

A permission model that fits in an index

Source systems express permissions differently — a wiki has space and page restrictions, a file store has per-file grants and inherited folder grants, a group directory has nested groups. Normalise all of it to one thing at index time: a set of opaque principal ids that may read this chunk.

chunk = {
  "id": "c_8811",
  "doc_id": "conf:SPACE/12345",
  "text": "...",
  "vec": [...],
  # Everything above is normal. This is the field that matters:
  "acl": ["user:u_112", "group:g_eng", "group:g_leadership"],
  "acl_source": "confluence",
  "acl_synced_at": 1754300000
}

Two rules keep this honest. Deny-by-default: a chunk with an empty or missing acl is visible to nobody, so an ingestion bug fails closed. And no negative entries: express “everyone except X” by materialising the positive set at index time, because an exclusion evaluated at query time is a rule you have to get right in two places.

Nested groups have to be flattened. If g_eng contains g_platform, then a document granted to g_eng must list g_platform too, or you must expand the user’s groups transitively at query time. Pick one — expanding the user side is usually cheaper because a person has tens of groups and a group has thousands of documents.

Ingesting with the ACL attached

This is the point at which the page stops naming methods. Both Confluence and Google Drive expose the permissions of an item through their APIs, and both change those APIs. The requirement list is stable even when the endpoints are not:

Whatever the endpoints turn out to be, wrap each source behind one interface returning documents with {id, text, acl, version, deleted}. Everything downstream then treats Confluence and Drive identically, and adding a third source is a new adapter rather than a new pipeline.

Filtering before the search, not after

Retrieve the top k from the subset the user may see. Do not retrieve the global top k and then remove what they may not see — that produces a user who is authorised for one relevant document receiving zero results because forty documents they cannot see outranked it.

def answer(user_id, question):
    principals = expand_principals(user_id)      # ['user:u_112','group:g_eng',...]
    qvec = normalise(embed([question])[0])

    # Pre-filter in the store. In SQLite, a join against a principals table;
    # in a vector database, a metadata filter applied during search.
    rows = DB.execute("""
        SELECT c.id, c.doc_id, c.text, c.vec
        FROM chunk c
        WHERE EXISTS (
          SELECT 1 FROM chunk_acl a
          WHERE a.chunk_id = c.id AND a.principal IN (%s))
    """ % ",".join("?" * len(principals)), principals)

    hits = top_k_by_cosine(rows, qvec, k=8)
    if not hits:
        return NO_RESULT, []
    return generate(question, hits), hits

Whatever store you use, confirm that its filtering happens during the nearest-neighbour search rather than after it. Approximate indexes traverse a graph, and a filter applied to the output of that traversal can return far fewer than k results — or, in the worst implementations, silently miss authorised documents. Ask the vendor which one theirs does; the answer is not always in the documentation.

Write the test now, not later: create a document visible only to group A, then assert that a user in group B gets zero chunks from it, that the answer does not mention it, and that no citation refers to it. Then run that test in CI on every change to retrieval. The multi-tenant version of this problem is identical in structure and has the same test.

Revocation, and how stale your index is

Here is the number nobody computes. If permissions sync nightly, then the window between somebody losing access and your assistant enforcing it is up to 24 hours. For a departing employee whose account is disabled centrally, single sign-on closes that hole. For a person moved off a project whose account stays active, it does not.

1. Check the user’s principals live, not from the index. Resolve group membership at query time against the directory, with a cache measured in minutes. This closes the common case — someone removed from a group — without re-indexing anything.
2. Sync document ACLs on a change feed rather than a nightly crawl, so a document made confidential propagates in minutes.
3. Re-check at citation time. Before returning an answer, verify the user can still read every document being cited, using a live call for the small number of documents involved. This is the cheap belt-and-braces check and it catches everything the index missed.
4. Record the staleness. Store acl_synced_at and alert when the oldest exceeds your policy. “Permissions are at most 15 minutes old” is a claim you can make to a security reviewer only if you measure it.

Leaks that are not in the documents

Three channels leak information without ever showing a restricted passage, and all three are easy to miss.

• The shape of the refusal. If “no such document” and “you cannot see that document” are different messages, the difference confirms the document exists. Return one message for both.
• Autocomplete and suggestions. Query suggestions built from all users’ queries will suggest “acquisition of…” to someone who should not know there is one. Scope suggestions to the asker, or drop the feature.
• Shared caches. A semantic cache keyed on the question alone serves one user’s answer to another. Any cache in front of a permission-aware system must include the principal set in its key, and that mostly destroys the hit rate — which is the honest trade-off, not a bug to engineer around.

The conversation history is a fourth. If a user was shown a passage they were entitled to on Monday and lost access on Tuesday, a follow-up question that resends the history re-exposes it. Store retrieved passages by reference and re-fetch under current permissions rather than carrying the text forward.

What it costs to keep the index warm

The recurring cost of an internal assistant is not the questions. It is keeping the index and the ACLs current, and that cost scales with how much the company writes, not with how much it asks.

Corpus: 40,000 documents, mean 1,800 words = 6 chunks each
        = 240,000 chunks

Initial embed:  240,000 x 300 tokens = 72M tokens
                at $0.02/M           = $1.44          (one-off)

Storage:        240,000 x 1,536 dims x 4 bytes = 1.5 GB of float32
                plus text and ACL rows          ~ 2.5 GB total

Steady state, if 2% of documents change per week:
                800 docs x 6 chunks = 4,800 chunks/week
                = 1.44M tokens/week ~ $0.03/week

Permission sync is the other half, and it is API calls rather than tokens:
40,000 documents polled for effective permissions is 40,000 requests if
there is no change feed, which is why "does it have a change feed" is the
question that decides the shape of this system.

Questions: 800 staff x 2/day x 2,500 tokens = 4M input tokens/day
           at $0.15/M                        = $0.60/day

Prices are illustrative and will move; the ratios will not. Embedding the corpus once is trivial, keeping it current is trivial, and the questions cost less than the coffee. What actually costs is the connector: permission APIs that must be polled, rate limits on the source systems, and an engineer keeping up with their changes.

Budget accordingly, and be suspicious of any plan whose cost model is all tokens. The failure mode of an internal assistant is not that it became expensive; it is that nobody maintained the connectors, the index went stale, people got a wrong answer twice, and they stopped asking.

Getting it approved

1. Start with one space or one shared drive that has simple, uniform permissions and a willing owner. The pilot tests the pipeline, not the permission model.
2. Add the permission tests to CI before adding the second source. Retrofitting them is how the gap gets discovered by someone else.
3. Log every query with the asker, the principals used, the documents retrieved and the documents cited — and make that log itself restricted, because it is a record of what everybody is asking about — and prompt logs are personal data under most regimes that would review this.
4. Publish what the assistant can see, in plain language, before people use it. The most damaging outcome is not a leak; it is employees discovering afterwards that a system had been reading a space they considered private.