Skip to content
Multigrid

Data engineering for AI

Getting documents out of the formats they arrive in and into a store you can index, update and reproduce — the part of an AI feature that is most of the work and none of the demos.

An AI feature is usually presented as a model plus a prompt. In the repository it is a model, a prompt, and several thousand lines that turn whatever the business actually has — a share drive of PDFs, a CMS, a vendor’s nightly CSV, a decade of scanned contracts — into text a model can be shown, with enough structure around it that a citation points somewhere and a deletion request can be honoured.

That work is unglamorous and it is where the failures live. A parser that silently returns an empty string, an encoding guessed wrong three stages upstream, a re-embedding run that has to redo everything because nothing was content-addressed: none of these look like AI problems, and all of them present as the model being bad at its job.

These pages are about that layer. They are heavy on code and on format detail, because the format detail is the subject — a PDF defeats naive extraction for reasons written down in its specification, and once you know the reason you know which documents you cannot parse and what to do with them instead.

The Data Pipeline Behind Every AI Feature

The eight stages between a file arriving and a model being able to cite it, and the one contract between stages that makes the whole thing rerunnable.

5 min read

Ingesting Documents at Scale: Formats and How Each One Fails

The specific failure mode of every format you will be handed — PDF, HTML, DOCX, CSV, email, images — with the check that detects each one before it reaches the index.

6 min read

PDF Parsing: Why It's Still Hard

What a PDF actually stores instead of text, which class of document each extraction strategy must therefore fail on, and a bake-off script to score parsers on your own hard files.

6 min read

HTML Extraction and Boilerplate Removal

Getting the article out of the page, why the naive approach costs you tokens on every request, and the arithmetic of what removal is worth.

6 min read

OCR Pipelines: The Preprocessing That Decides Your Accuracy

The image operations Tesseract's own quality guidance asks for, in order, with the code — and a character-error-rate harness so you can see what each one is worth on your documents.

6 min read

Web Scraping for AI: The Legal and Technical Constraints

What robots.txt actually is as a standard, the AI-specific user-agent tokens publishers now use, and the polite-crawler code that keeps you welcome.

6 min read

Deduplication in Large Text Corpora

Exact, near and substring duplication are three different problems; MinHash with LSH solves the middle one, and here is the implementation with its parameters explained.

6 min read

Data Quality Checks Before You Train or Index

Nine assertions to run between extraction and indexing, each written as code and each paired with the specific downstream failure it makes impossible.

6 min read

Incremental Processing: Only Redo What Changed

Content hashing per stage, why mtime is not good enough, and the arithmetic that shows what a full rebuild costs against a selective one.

5 min read

Schema Evolution in AI Pipelines

Which upstream changes are safe, which are not, and the compatibility rules from Avro and the schema-registry world that tell you which is which before you deploy.

6 min read

Metadata Design for Retrieval

The fields to capture at ingest because they are impossible or expensive to reconstruct later, and what each one buys you at query time.

6 min read

Storing Documents and Chunks: Table Design

A relational schema for documents, chunks and embeddings that survives re-chunking and re-embedding without invalidating stored citations.

6 min read

Handling Updates and Deletes in a Search Index

Why deletes in vector and inverted indexes are tombstones rather than removals, what that means for correctness and for disk, and how to make an update atomic from the reader's side.

5 min read

Batch vs Streaming for AI Workloads

One question decides it — how stale is the index allowed to be — and everything else is a consequence of the answer.

5 min read

Orchestrating Multi-Stage AI Pipelines

Retries that do not double-charge you, idempotency keys that make at-least-once delivery safe, and partial reruns that touch only the stage you changed.

5 min read

Data Versioning for Reproducible AI

What you must pin to reproduce a result from six months ago, which of those things are not naturally versioned, and the manifest that records all of them.

5 min read

Cost of Data Movement: Egress, Storage and Compute

Every point in a document pipeline where bytes move and somebody is charged, and the cost model those points combine into.

6 min read

Privacy-Preserving Data Pipelines

Where to put redaction so that every later stage inherits it, how to keep redacted text usable, and the surfaces that keep a copy after you thought you had removed it.

5 min read

Labelling Data Without a Labelling Team

Model pre-labelling with human adjudication of the disagreements, why that is the cheap configuration, and the arithmetic for how much cheaper.

5 min read

Building a Document Corpus You Can Actually Search

The whole path in one place — ingest, clean, dedupe, chunk, index, verify — as an order of operations with the checkpoint after each step.

6 min read

Other topics

Data engineering for AI · Multigrid