Turning a Wiki Into Something Machines Can Query

An internal wiki holds the answers to questions nobody can query: which services depend on which, who owns what, which regions a feature is enabled in. Turning that into structured data is a real project with a knowable cost, and the first thing to do is work out whether the cost is acceptable — because the extraction is the cheap half.

The arithmetic that decides the project

Extraction is cheap and review is not. Do this calculation before writing any code, with your own numbers:

corpus:                3,000 pages
pages that hold facts
worth extracting:      3,000 x 0.35  =  1,050    (from a sample, not a guess)

extraction, per page:
  ~1,500 input tokens + ~400 output tokens
  1,050 pages -> ~2.0M tokens total, one pass, a few hours of wall clock
  -> a small bill, and rerunnable

review, per page:
  4 minutes to confirm a clean extraction
  12 minutes to correct a bad one

at 90% acceptable:  1,050 x (0.9 x 4 + 0.1 x 12) / 60  =  84 hours
at 70% acceptable:  1,050 x (0.7 x 4 + 0.3 x 12) / 60  = 112 hours
at 40% acceptable:  1,050 x (0.4 x 4 + 0.6 x 12) / 60  = 154 hours

Two conclusions follow, and both are the opposite of the instinct. Review dominates the budget completely, so effort spent raising the acceptable rate pays back several times over — the gap between 90% and 40% is 70 hours of somebody’s life. And the sensible move is to convert the subset with the highest acceptable rate first, prove the value, and let the difficult pages wait, rather than running the whole corpus and generating a backlog nobody works through.

Get the 0.35 from a sample. Take 50 random pages, classify them by hand, and you have a defensible number in an hour. Guessing it is how projects are approved on figures that turn out to be wrong by a factor of three.

Triage: which pages convert

Not all wiki content should become structured data, and the test is not a percentage — it is a property of the page. A page converts when its value is in facts that recur across pages in the same shape. It stays prose when its value is in argument, exception, judgement or narrative.

The last row is the one that costs people. A page nobody has edited in four years, extracted into a graph, becomes a fact with today’s timestamp and no visible age. Carry the source page’s last-edited date onto every extracted fact as provenance, and let consumers filter on it.

A cheap classifier does the triage: table density, heading structure, template usage, page length, edit recency, and whether the title matches a known naming pattern. Route the uncertain cases to a person; it is a five-second decision per page.

Take the tables first

The highest-value, lowest-risk step, and it needs no model at all. Tables, infoboxes and templated fields are already structured; the only work is mapping columns onto schema fields.

1. Export properly. Use the wiki’s API to get the source markup or clean HTML rather than scraping rendered pages — navigation chrome and edit links produce columns that do not exist.
2. Parse tables to rows, keeping the page id, the section heading and the table index. That triple is the provenance for every fact you derive, and reconstructing it later is impossible.
3. Map headers to fields once per table shape, not per table. Most wikis have a handful of recurring shapes and a long tail; handle the shapes and send the tail to review.
4. Normalise values through the glossary so that “prod”, “Production” and “PRD” become one value. See a glossary your company and your model both use.
5. Flag the merged and spanning cells. A rowspan silently shifts every column after it in naive parsers, which produces confidently wrong rows. Detect them and route those tables to review rather than trusting the parse.

Expect this pass to produce a large share of the total facts for a small share of the total effort, and to have a much higher acceptable rate than anything a model does — which is exactly why it goes first.

Extracting from the prose that remains

Everything in extracting a knowledge graph with an LLM applies: a closed schema, evidence spans required, and a validation pass that rejects anything not grounded in the page text. Three things are specific to wikis and worth handling explicitly.

Links are entity mentions, and better ones than text

A wiki link is a human’s assertion that this phrase refers to that page. That is a resolved entity mention, free, with none of the ambiguity of a string. Use links preferentially over extracted names, and treat the target page id as the entity key wherever a link exists.

Section headings are context

A fact under “Deprecated approaches” means the opposite of the same sentence under “Current setup”. Feed the heading path into the extraction prompt and store it with the fact, and add heading patterns that suppress extraction entirely — historical, deprecated, proposed, draft, rejected.

Hedged sentences are not facts

Wikis are full of “we should probably move this to X” and “this may still be running on Y”. An extractor that ignores modality turns a plan into a fact. Require the model to emit a modality field — asserted, planned, deprecated, uncertain — and load only the asserted ones, keeping the rest as annotations on the page.

The review interface

The 84 hours in the arithmetic above assume review is confirmation, not correction. That is a property of the interface as much as of the extractor:

• Source text and extracted fields side by side, with the evidence span highlighted in the source. The reviewer’s job is to compare two things on one screen, not to read a page.
• Approve the page, not the field. One key to accept everything, with per-field correction only where something is wrong. Requiring a click per field turns four minutes into fifteen.
• Order by confidence, ascending. Reviewers find the problems while they are fresh, and the tail of high-confidence pages can be spot-checked rather than fully reviewed once the error rate on the reviewed portion is known.
• Group by page type. Fifty service pages in a row is far faster than fifty mixed pages, because the reviewer holds one schema in their head.
• Track corrections by field. One field accounting for most corrections is a prompt or schema problem, fixable once, and it is invisible unless you count.

Where the wiki lives afterwards

The question that decides whether the project holds: if the wiki stays the place people edit, the structured data drifts within months; if the structured data becomes the source and the wiki is generated, people stop writing the prose that made the wiki valuable. Neither pure answer works.

The arrangement that does work is to split the page by kind. Fields move to the structured store and are rendered into the page from it, so editing a field means editing the record. Prose stays in the wiki and is edited there. The page a person reads is the two composed, which means nobody has to visit two places and nobody is asked to maintain the same fact twice.

After that, the wiki becomes one more source in a change pipeline, with the same freshness monitoring and conflict rules as every other source — which is the subject of keeping a knowledge graph fresh. The conversion is not a project that finishes. It is the point at which the wiki starts being maintained as data.