Data Quality Checks Before You Train or Index

A data quality dashboard is a place where problems are displayed. A data quality gate is a place where they stop. The checks below are the same either way; what differs is whether a bad batch can reach the index while everyone is looking at something else.

Where the checks belong

Two positions, and both are needed. Per-document checks run inside the extract stage and route individual failures to quarantine — they are cheap, local and never block the batch. Corpus-level checks run after the batch is assembled and before it is published, compare it with the previous batch, and block on a large enough change.

The distinction matters because the two catch different bugs. A broken parser produces thousands of individually plausible documents that are collectively wrong — every one passes the per-document checks and the corpus checks catch it in one line.

There is a third position people reach for and should not: checking after indexing. By then the bad data is retrievable, the vectors are paid for, and any cache that was warmed holds the wrong content. The cost of moving a check earlier is almost always a few seconds of pipeline time; the cost of moving it later is a rebuild plus however long the bad answers were being served.

It is also worth being explicit about what a check is for. Not “is this document good?” — that is a judgement nobody can automate — but “is this document the kind of thing the rest of the pipeline assumes?” Every stage downstream has assumptions: the chunker assumes sentence boundaries exist, the embedding model assumes a language it was trained on, the retriever assumes the metadata it filters on is populated. A quality check is one of those assumptions written down and made enforceable, which is why the useful way to derive the list is to walk the downstream stages and ask what each one would be surprised by.

Per-document checks

import re, unicodedata
from dataclasses import dataclass

@dataclass
class Doc:
    id: str; text: str; lang: str; raw_bytes: int; source: str

def check_document(d: Doc) -> list[str]:
    fail = []

    # 1. Non-trivial text. Prevents: unretrievable zero-length rows from
    #    scanned PDFs that parsed without raising.
    if len(d.text.strip()) < 50 and d.raw_bytes > 20_000:
        fail.append("empty_after_extraction")

    # 2. Decodable. Prevents: U+FFFD inside chunks, which tokenizes badly
    #    and makes exact-match search fail on the affected words.
    if "\ufffd" in d.text:
        fail.append("replacement_characters")

    # 3. Prose-shaped. Prevents: navigation dumps and glyph soup being
    #    embedded as if they were content.
    letters = sum(c.isalpha() for c in d.text)
    if letters / max(len(d.text), 1) < 0.5:
        fail.append("low_letter_ratio")

    # 4. Expected language. Prevents: a multilingual embedding model
    #    silently doing worse on documents nobody realised were German.
    if detect_language(d.text) != d.lang:
        fail.append("language_mismatch")

    # 5. Not one repeated line. Prevents: a paginated export whose every
    #    page is the header, dominating retrieval for its terms.
    lines = [l.strip() for l in d.text.splitlines() if l.strip()]
    if lines and len(set(lines)) / len(lines) < 0.3:
        fail.append("repetitive")

    # 6. Control characters. Prevents: JSON that fails to serialise three
    #    stages later, in a queue consumer with no context.
    if any(unicodedata.category(c) == "Cc" and c not in "\t\n\r"
           for c in d.text):
        fail.append("control_characters")

    return fail

Check 4 is the one people leave out and regret. Language is silent when it is wrong: nothing errors, the embedding model returns a vector, the vector is simply less useful, and the symptom is a vague “retrieval is worse for some customers” six months later.

Corpus-level checks

These are comparisons against the previous accepted batch. Each has a threshold, and the threshold is a decision to be written down rather than a number in somebody’s head:

The last one is the sharpest. A retrieval filter on a field that is now null returns nothing, and “returns nothing” is not an error anywhere in the stack — it is an empty result set, which looks exactly like a question with no answer.

Set the thresholds from history rather than from intuition. Run the checks in report-only mode for a few weeks, look at the distribution of each delta across normal runs, and put the threshold outside the range you observed. A threshold tighter than normal variation produces failures nobody believes, and a check nobody believes is a check that gets bypassed with a flag and then deleted. Widening a threshold after a false alarm is a normal and healthy thing to do, provided the widening is a commit rather than an argument.

Two of these checks need a subtlety to be useful. Compare medians and percentiles rather than means, because one enormous document moves a mean and tells you nothing. And compare against the last accepted batch rather than the last attempted one, or a bad batch that slipped through becomes the new baseline and the next bad batch looks normal — the ratchet failure that makes gradual degradation invisible.

The two leaks worth a dedicated check

Evaluation data in the index

If your golden evaluation set is built from documents in the corpus, and the corpus is what retrieval searches, then a question drawn verbatim from a document will be answered by retrieving that document — which measures nothing. Keep the hashes of evaluation source documents in a list and assert they are absent from any index the evaluation runs against, or accept that the evaluation is measuring lookup rather than retrieval.

Secrets and personal data

Corpora assembled from internal drives contain credentials. Not occasionally — routinely, because a share drive is where people put runbooks. Run a detector over extracted text before it is embedded, because after embedding the content exists in a vector store, in a cache, and in whatever logs recorded the request:

SECRET_PATTERNS = {
    "aws_key":      r"AKIA[0-9A-Z]{16}",
    "private_key":  r"-----BEGIN (?:RSA |EC |OPENSSH )?PRIVATE KEY-----",
    "jwt":          r"eyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}",
    "slack_token":  r"xox[baprs]-[0-9A-Za-z-]{10,}",
    "pg_url":       r"postgres(?:ql)?://[^\s:@]+:[^\s@]+@",
}

Regexes catch the structured ones. Names, addresses and account numbers need a real detector, and the decision about what to do with a hit belongs in the redaction stage rather than here — this check’s job is only to make sure nobody can claim they did not know.

Making it a gate

The mechanism that makes this real is publishing by pointer. Build the new batch under a version, run the checks against it, and only then move the alias that readers resolve:

build  corpus/2026-08-03/            # nothing reads this yet
verify python -m quality.corpus corpus/2026-08-03 --against corpus/current
promote corpus/current -> 2026-08-03  # one atomic pointer swap
                                      # rollback is the same swap, backwards

A failing check should print the two numbers and the threshold, not just fail. “median tokens 412 vs 1,830, threshold ±25%” identifies the parser regression immediately; “quality check failed” sends somebody to read the code. And keep an override that requires a reason string to be recorded — a gate nobody can open is a gate that gets deleted the first time it is wrong.

The promote step is also the natural place to write the run’s manifest: which inputs went in, which checks ran, what each of them measured, and who approved any override. That record is what makes a later question — “when did the corpus start containing this?” — answerable by looking rather than by guessing, and it costs one file per run. Versioning the corpus alongside the code is the same discipline applied to the whole build.

Keep the alias swap genuinely atomic. A promotion implemented as “delete the old rows, insert the new ones” has a window in which readers see a half-empty corpus, and that window is exactly as long as the largest table. A pointer, an alias or a view definition changes in one statement, and the rollback is the same statement with the previous value — which means the answer to a bad promotion is ten seconds of work rather than a restore.