Text Classification Without an LLM

Classification is the task language models most often get used for and least often deserve. Putting a document into one of eight buckets is a closed-output problem, which is exactly the shape classical methods were built for, and the gap in running cost is three orders of magnitude.

The four approaches

1. Rules

Keyword lists and regular expressions with a priority order. Zero training data, zero inference cost, completely auditable, and you can fix a specific error in one line. They fail on synonyms, negation and anything requiring judgement, and they accumulate exceptions until nobody will touch the file. Still the correct answer for a handful of unambiguous, high-precision categories — and always worth writing first, because building them tells you what the categories actually are.

2. Linear model over sparse features

TF-IDF or character n-grams into logistic regression or a linear SVM. Trains in seconds on a laptop, predicts in well under a millisecond, and the learned weights are directly inspectable — you can print the twenty features driving a class. Wang and Manning’s Baselines and Bigrams (ACL 2012) is the standing reminder that a carefully-built version of this remained competitive with far more elaborate systems on sentiment and topic tasks. fastText (Joulin, Grave, Bojanowski and Mikolov, 2016) extended the same idea with subword features and reported training on more than a billion words in under ten minutes on a multicore CPU, and classifying half a million sentences among hundreds of thousands of classes in under a minute — those figures are from the paper and they are the reason this row exists.

3. Fine-tuned small transformer

A 22M to 110M parameter encoder — MiniLM, DistilBERT, a small multilingual model — fine-tuned on your labels. This is the accuracy option among the local approaches: it reads context, so negation and word order are handled. It needs a GPU for comfortable training, a few hundred to a few thousand labels, and it costs single-digit milliseconds per document on a CPU at inference. The relevant page is whether fine-tuning is the right move.

4. Prompted language model

Describe the categories, send the document, parse the label. No training data, works on day one, handles categories you invented this morning, and copes with nuance the others cannot. It is also the only option with a per-document price, a network round trip, and an output space that is not actually closed — constraining it to your label set is a separate piece of work.

The cost, derived

Nobody benchmarked these. What follows is arithmetic from labelled assumptions; replace them with yours and the conclusion may move.

• Assumption: 1 million documents to classify, averaging 300 tokens — 300 million input tokens.
• Assumption: a 4-vCPU instance costs $0.05 per hour.
• Assumption: a linear model classifies a document in 0.2 ms; a small fine-tuned encoder in 8 ms on CPU.
• Assumption: a hosted model costs $0.10 per million input tokens at the cheap end and $1.00 at the mid range, with output small enough to ignore.

The ratio between the second and fourth rows is roughly 3,000× to 30,000× on running cost, and it is a ratio rather than a difference: it holds at ten million documents and at a hundred million. Against that, the hosted model has essentially no fixed cost, which is the entire reason it wins at low volume. The crossover is what the break-even derivation computes.

What a label budget actually buys

The objection to every local option is “we do not have labelled data”, and it is usually a smaller obstacle than it sounds.

Labelling a document into one of a few buckets takes seconds, not minutes. At an assumed 10 seconds per document, 1,000 labels is under three hours of one person’s time — and a linear model over a thousand well-chosen examples is a real system, not a toy. Rough guidance worth holding loosely: a few hundred examples per class is enough to fine-tune a small encoder usefully; a thousand or two per class is where a linear model stops improving quickly; ten thousand is diminishing returns for most business taxonomies.

There is also a shortcut that makes the choice less binary: use a hosted model to label a few thousand documents once, have a human review and correct them, and train a local model on the result. You pay the API once instead of forever, and you end up with a component that runs in-process. That is distillation in the ordinary sense — the general pattern is the same — and for classification it is unusually easy because the output is one token.

Do not skip the human review step. Training on unreviewed model output bakes its systematic errors into a component that will now make them faster and cheaper, and you will have no evaluation set that is independent of them.

Latency, determinism, auditability

Cost is the least interesting of the four differences.

Latency. Sub-millisecond in-process against hundreds of milliseconds over the network. That is not a percentage difference, it is an architectural one: below a millisecond, classification is a function call inside a request handler. At 400 ms it needs a queue, a worker, timeouts and a retry policy, and it either blocks the user or becomes asynchronous.

Determinism. A fitted model with pinned weights returns the identical label for the identical input indefinitely. A hosted endpoint can be updated without notice — silent model updates are a documented operational hazard — and the same prompt can produce different labels across calls unless you have pinned the sampling.

Auditability. For a linear model you can produce the exact features and weights that led to a decision. If the classification affects a person — moderation, eligibility, prioritising a complaint — that difference is not academic; it is the difference between an explanation and a plausible story generated afterwards.

The order to try them in

Ascending cost, and stop at the first one that clears your bar on a held-out set of your own documents. Write the rules, because they define the categories. Fit a linear model on whatever labels you have, because it takes an afternoon and it is the baseline every later decision is measured against. Fine-tune a small encoder if the linear model’s errors are context-dependent. Reach for a hosted model when the categories genuinely need reading comprehension, when volume is low enough that the bill is noise, or when you need something working this week.

And then consider the cascade rather than the choice: a local model handles the confident majority, and only documents below a confidence threshold escalate. At a 5% escalation rate the API line in the table above drops to $1.50 to $15 per million documents, and the hard cases still get the better treatment.