Domain-Specific Corpora: Building One for Your Industry

Almost every “we need a corpus for our industry” project starts by scraping and ends by discovering that the expensive part was neither collection nor cleaning but deciding what the corpus was for.

Decide the purpose before the sources

Three different artefacts get called a domain corpus, and they have almost nothing in common except the subject matter. Getting this wrong means collecting the right documents for the wrong job.

The order to try things in is usually the reverse of the order people attempt them: retrieval first, because it is cheapest and needs no training; then a task corpus for fine-tuning; and continued pretraining only when the domain’s language is genuinely unlike anything in general pretraining data. The fine-tuning versus retrieval comparison is the decision underneath this, and Gururangan and colleagues’ Don’t Stop Pretraining (ACL 2020) is the standard reference for when domain-adaptive and task-adaptive pretraining pay off — with the notable finding that task-adaptive pretraining on a small, task-relevant corpus was often competitive with domain-adaptive pretraining on a much larger one.

The sourcing tiers

Ordered by how defensible they are, which is not the same as how easy.

• Your own data. Support tickets, internal documentation, contracts, resolved cases, product manuals. Best quality-per-effort by a wide margin, unambiguous licensing, and uniquely matched to your distribution. The constraints are internal rather than legal: it contains personal data, and it needs detection and redaction before it goes anywhere near a training job.
• Government and regulator publications. Legislation, case law, regulatory guidance, standards bodies, patents, clinical trial registries, company filings. Frequently the highest-value public source for a regulated industry, usually explicitly reusable, and reliably well structured because someone was required to publish it.
• Open research and open-licensed publishing. Preprint servers, open-access journals, institutional repositories. Check per-article licences rather than the platform’s — a repository hosts articles under a range of terms and the aggregate tag means nothing.
• Licensed commercial data. Trade press archives, standards documents, market data. Costs money and buys certainty. The contract question to ask before signing is whether it permits model training specifically, and whether it permits distributing the resulting model.
• The open web, filtered. Industry blogs, forums, vendor documentation. Highest volume, lowest average quality, and the most complicated position on rights. Treat as a source that must earn its place through the quality filter below rather than as a default.
• Generated to fill gaps. Legitimate as a supplement for a slice you can name and cannot source, with everything the rest of this cluster says about gating a generator applying in full. Not legitimate as the foundation.

Filtering by a quality classifier

The technique that made the largest visible difference to public pretraining corpora is not a better crawler. It is a classifier that decides which documents are worth keeping.

The pattern, used in several published corpus efforts: assemble a small set of reference documents that exemplify what you want, train a cheap classifier — historically a fastText model over n-grams, more recently a small model scored against an explicit rubric — to distinguish those from a random sample of the crawl, then score the full corpus and keep the top slice. The FineWeb-Edu work is the clearest recent example: documents were scored for educational quality by a language model, that scoring was distilled into a small classifier, and the classifier was run over the whole corpus. The phi series makes the same argument from the other end — Textbooks Are All You Need reported strong coding results from a small model trained on a modest volume of filtered and synthesised textbook-style material.

For a domain corpus the same recipe applies at much smaller scale, and the reference set is the part that carries the information:

# 1. Two hundred documents a domain expert calls exemplary, and a random
#    sample of the raw crawl as negatives. The positives are the spec.
train = [(d, 1) for d in expert_selected] + [(d, 0) for d in random_crawl]

# 2. A cheap classifier. Embeddings + logistic regression is enough; this is
#    a ranking device, not a product.
clf = LogisticRegression(max_iter=1000).fit(embed([d for d, _ in train]),
                                            [y for _, y in train])

# 3. Score everything, keep a FRACTION rather than a threshold — thresholds
#    drift between runs, fractions do not.
scores = clf.predict_proba(embed(corpus))[:, 1]
cutoff = np.quantile(scores, 0.70)          # keep the top 30%
kept   = [d for d, s in zip(corpus, scores) if s >= cutoff]

# 4. READ the boundary. Fifty documents just above the cutoff and fifty just
#    below. This is where you discover the classifier learned "long" or
#    "has a table of contents" rather than "good".
inspect(sorted(zip(scores, corpus))[int(0.68*len(corpus)):int(0.72*len(corpus))])

Step 4 is not optional. A quality classifier trained on a small reference set will happily learn a proxy — document length, formality, the presence of citations — and filtering a million documents on a proxy removes an entire register of writing from the corpus. Reading the boundary is how you find out which proxy it learned, and it takes twenty minutes.

The build order

• Inventory before collecting. List candidate sources with estimated volume, licence status and extraction difficulty. Half of them will turn out to be redundant with each other, and finding that out on a spreadsheet is cheaper than finding out after extraction.
• Sample fifty documents per source and read them. Before writing a single extractor. This is the step that reveals the source is 80% boilerplate, or duplicated from another source, or written in a register nobody in your domain uses.
• Extract, then normalise, then attach provenance. Every document carries its source, retrieval date and licence from the moment it enters the pipeline — retrofitting is the hard path.
• Deduplicate globally, across sources. Domain corpora are extraordinarily redundant: the same press release, the same regulation, the same specification appears in a dozen places with different framing. Cross-source deduplication typically removes far more than within-source dedup does.
• Filter by quality classifier. As above, with the boundary read by hand.
• Decontaminate against your evaluation set. Before training, not after the results look good.
• Document it. Sources, volumes, filters, removals, gaps. The datasheet is what makes the corpus reusable by the next team rather than rebuilt by them.

The licensing questions that decide it

These are the questions to answer before the engineering, because a negative answer to any of them can invalidate the work rather than adjust it. None of them is answerable retroactively at reasonable cost: each depends on facts — what a page said, what a licence was, what a user agreed to — that are true on the day of collection and only recoverable afterwards by luck.

• What does the source permit, from the source itself? Not from an aggregator’s metadata field. The published audit of dataset licensing found those tags to be unreliable often enough that they cannot be the basis of a decision — the provenance page covers what the audit found.
• Does the jurisdiction have a text-and-data-mining exception, and did the rightsholder opt out? The EU has such an exception with a machine-readable reservation mechanism; other jurisdictions differ substantially. This is the question a lawyer answers, and it is worth the hour.
• Is any of it share-alike? Share-alike terms can propagate obligations to derived work. One CC BY-SA source in a corpus is a decision to make deliberately, not a footnote.
• Does it contain personal data? Domain corpora are full of it — names in case law, authors in filings, customers in tickets. That triggers a different regime with different rights attached; see the GDPR page.
• Where did the generated portion come from? Model output carries the provider’s terms, and those terms often speak directly to training. Record the model and version per row.