Building an Internal AI Assistant for Your Company
5 min read · updated August 3, 2026
Almost every internal assistant project starts by building one index over everything the company has written. That is the moment the project acquires its defining problem, because a single index over documents with different audiences is a permission boundary that has been deleted.
The index is a permission boundary
Documents in a company are not uniformly readable. Compensation bands, performance reviews, an unannounced restructuring, a deal folder, an investigation, a customer’s data under a contract that says only the account team sees it — each has an audience, and the source systems enforce those audiences. Copy the text into a vector store and the enforcement stays behind in the source system.
What makes this worse than a normal access-control bug is that retrieval surfaces things nobody would have found by browsing. Nobody discovers a mis-shared salary spreadsheet in a folder they never open. An assistant asked a natural question about pay bands retrieves it on the first try and quotes the relevant paragraph, helpfully. The most common source is not a misconfigured system: it is a link somebody set to “anyone with the link” three years ago for a meeting, which has been technically company-readable ever since and practically invisible.
So the first artefact of the project is not a prototype. It is a report of what is in scope and who can currently read it, produced from the source systems’ own permission APIs. That report routinely stops the project for a month, and it is cheaper than the alternative.
Three enforcement designs
| Design | Description |
|---|---|
| acl-on-chunk | Store the source document's access list with each chunk and filter by the asking user's groups before ranking. Fast, works with any store that supports metadata filters. Failure window: permissions change in the source and the copy is stale until the next sync — an access revoked this morning may still be honoured this afternoon. Sync frequency is a security parameter, not an ops preference. |
| check-at-read | Retrieve candidates, then call the source system to confirm this user can read each one before it enters the prompt. No staleness at all, and it is the design that survives an audit. Costs a permission call per candidate on every query, so it needs caching with a short time-to-live and it constrains how many candidates you can consider. |
| public-corpus | Index only material explicitly published to everyone — the handbook, policies, published runbooks — and nothing else. Trivially correct, no per-user machinery, and it answers a surprisingly large share of real questions. The right starting point for almost everyone, and the one teams skip because it feels unambitious. |
Whichever is chosen, two rules hold. The generated answer may only rest on chunks that passed the filter for this user, which means permission filtering happens before generation and never as a post-hoc redaction of an answer already written. And conversation history is subject to the same rules: a summary carried forward from an earlier turn can smuggle content into a context where it no longer belongs, and shared or exported threads carry it further. The general isolation problem is multi-tenant retrieval, and the exfiltration risk once an assistant can both read internal documents and reach outside is the lethal trifecta — which is a live concern the moment anyone adds a browsing or email tool.
The corpus is worse than you think
The second discovery of every project is that the company’s written knowledge is mostly wrong. Not maliciously — it is old. The wiki has four pages on the deployment process from four different years, none marked obsolete, and the accurate one is the least viewed because the outdated one ranks higher internally after years of links.
Retrieval has no way to know which is current, and recency of the document’s modified date is a poor proxy, because a typo fix bumps it. What helps is unglamorous corpus management: an owner and a review date per document class, archiving rather than leaving superseded pages in place, and — the highest-value single intervention — excluding whole spaces that nobody maintains. A smaller curated corpus beats a complete stale one, every time, and this is the thing that determines answer quality far more than model choice. Index freshness and metadata filtering cover the mechanics; the hard part is organisational.
Chat archives deserve a specific warning. A thread where three people debated an approach and the conclusion arrived as a reaction emoji, or in a call, reads to a retriever as three confident contradictory statements. Treat conversation history as weak evidence, prefer documents, and if you must index chat, index resolved threads only.
Adoption is decided in the first week
Internal tools have one launch. An employee who asks three questions and gets two wrong answers has formed a view that a later improvement will not reverse, because they will not come back to test it. That asymmetry should determine the rollout more than any technical consideration.
- Launch narrow. One domain, with a curated corpus and a named owner who cares whether the answers are right. Expand when that domain is boring.
- Build the question set from real questions. Mine the internal help channels for what people actually ask. It is free, it is representative, and it is a better evaluation set than anything invented in a planning meeting — golden datasets explains how to keep it honest.
- Grade citation correctness, not answer correctness. Whether the cited source is the right source is far more tractable to judge than whether a paragraph is true, and it is what makes an answer verifiable in a company where somebody will always want to check. Answers that cite nothing should be labelled as such rather than presented alongside answers that do.
- Route negative feedback to the corpus owner. A thumbs-down almost always means a document is wrong, missing or outranked by an obsolete one. Sending it to the engineering team produces a ticket; sending it to the person who owns the page produces a fix.
Logging queries is watching employees
Query logs are necessary — you cannot improve retrieval without seeing what failed — and they are also a record of what individual employees asked, which includes questions about parental leave, grievance procedures, redundancy terms and their own pay. That is a different category of data from application telemetry, and treating it as ordinary logging is how a useful tool becomes a trust problem.
The workable posture is to separate the two purposes. Aggregate, identifier-free query analysis for improving the system, retained normally. Identifiable logs, if kept at all, on a short retention with restricted access and a stated purpose. Tell people which is which before launch rather than after somebody asks. In several jurisdictions, systems capable of monitoring employees also involve consultation with employee representatives before deployment, so this is a conversation to start early rather than a compliance step at the end — see PII in logs and what to log. This is not legal advice.