Building a Document Corpus You Can Actually Search

Every step here has its own page in this cluster. This one is about the sequence, because the most common way a corpus ends up unsearchable is not a bad step — it is a correct step performed before the one it depended on.

The order of operations

1  inventory      what exists, how much, which formats     -> counts by format
2  extract        bytes -> text, per format                -> text + failures
3  normalise      one encoding, NFC, whitespace, quotes    -> canonical text
4  clean          boilerplate, headers/footers, furniture  -> article text
5  dedupe         exact, then near, then substring         -> representatives
6  enrich         language, dates, sections, access labels -> metadata
7  chunk          split, with offsets back into the text   -> chunks
8  index          embed + write; keyword index too         -> searchable
9  verify         known-item retrieval on a real question  -> pass/fail

Each step reads the previous step's artefacts and writes its own.
Nothing is done in memory across a step boundary.

The dependencies that make this ordering non-negotiable: dedupe needs normalised text or it finds nothing; dedupe needs cleaning first or it matches on shared boilerplate rather than on content; chunking needs cleaning or every chunk contains navigation; and everything needs the offsets that only exist if normalisation happened once, early, and was recorded.

Steps 1–3: get the text

Inventory before anything. A count by detected format, a size distribution, and a sample you have opened with your own eyes. Half a day here changes the design: a corpus that is 70% scanned images is a different project from one that is 70% HTML, and you would rather know before choosing a parser. The triage script produces exactly this table.

Extract with a fallback cascade and record which rung produced each document, because that field is what lets you re-run a class later. Text layer first, OCR for the documents that fail the assertions, quarantine for the rest.

Normalise once, at the boundary. Decide the encoding where you have both the bytes and the HTTP headers, apply Unicode normalisation, collapse whitespace, and never touch it again. Record the character offsets from this point forward; every citation you will ever show is an offset into this text.

Checkpoint: no document in the corpus has an empty extraction, a U+FFFD, or a letter ratio under 0.5 without being quarantined. Count the quarantine. That count is your coverage number and somebody should sign off on it before you continue.

Steps 4–6: make it a corpus

Clean before you dedupe. This is the ordering that costs people a rebuild. Two pages from the same site share their navigation, so a near-duplicate detector run on uncleaned HTML finds that every page on a site is a duplicate of every other page — and the fix is not a threshold change, it is doing step 4 first.

Dedupe in three passes, cheapest first: exact hashes over canonical text, then MinHash with LSH for near-duplicates, then substring detection for the boilerplate that survived cleaning. Keep the losers as pointers rather than deleting them.

Enrich while the structure still exists. Section paths, effective dates and access labels come from the document tree and from the source system, both of which you still have at this point and neither of which you can recover from flat text later. The recoverability test is how to decide what goes in here.

Checkpoint: duplicate rate is plausible for the source (a wiki: low; an email archive: high), every document has a language with a confidence, and every document has an access label — even if that label is “public”, written explicitly rather than absent.

Steps 7–9: make it searchable

Chunk with offsets and section paths. The chunk id is derived from content, so re-chunking is additive — the schema is what makes this reversible.

Index both ways. Build the keyword index alongside the vector index. It costs almost nothing on top of what you have already done, and it is what makes exact identifiers, product codes and rare names findable at all — hybrid retrieval is not an optimisation to add later, it is the default that a vector-only index is a subset of.

Verify with known-item retrieval. Not a similarity score. Take twenty questions to which you personally know the answering document, run them, and check whether that document is in the top results. This catches the things nothing else does: a filter on a field that is null, an index built over the wrong model’s vectors, a chunk boundary that split every answer away from its question.

The acceptance test

Before anyone calls the corpus done, these should all pass, and they should be a script rather than a checklist:

• Every source document is in exactly one terminal state — indexed, quarantined or skipped — and the three counts sum to the inventory from step 1. This one assertion catches most silent losses.
• No orphans: every chunk resolves to a document version, every vector to a chunk, and the row counts are consistent in both directions.
• Twenty known-item queries retrieve their known document in the top ten. Store the questions; this is the regression test for every future change.
• A random sample of ten chunks, read by a human, is coherent text with a plausible section path. Ten chunks takes five minutes and catches chunking bugs no metric reports.
• A citation from a retrieved chunk resolves back to a highlighted span in the original document. If it does not, the offsets were lost somewhere and you will not find out later at a convenient moment.

The four orderings people get wrong

All four have the same underlying shape: a stage was run before the stage that produces its input assumptions. That is also why the artefact-per-stage discipline from the pipeline overview is worth the extra plumbing — when you discover the ordering was wrong, it decides whether the fix is re-running two stages or rebuilding everything from the source system.

Build the smallest version of all nine steps before improving any one of them. A thousand documents through the whole path, verified end to end, teaches you more about which stage deserves the work than any amount of planning — and it is usually not the stage you expected.