Classification With Embeddings Instead of an LLM

Routing a support ticket into one of forty categories does not obviously need a language model. It needs a function from text to a label, and if you have a few thousand labelled examples there is a much cheaper function available: embed the text once, then run a linear classifier over the vector.

The mechanism

An embedding is a feature vector. Anything you would do with feature vectors works, and the two useful options sit at opposite ends of the effort scale.

Centroid classification needs no training at all: average the embeddings of the labelled examples for each class, and assign a new item to the nearest centroid. Ten labelled examples per class is enough to try it, and for well-separated categories it is often enough to ship. It is also trivially updatable — a new example is an incremental change to one mean.

Logistic regression on the embeddings is the version that wins. It learns a decision boundary rather than assuming spherical classes, it handles imbalanced and overlapping categories, and it gives you calibrated-ish probabilities you can threshold. Fitting it over 1,536-dimensional features and a few thousand examples takes seconds on a laptop CPU, and inference is one matrix multiply against a matrix of size n_classes × 1536. For forty classes that is 61,440 multiply-adds — microseconds, and no network call.

The important structural property is that the expensive step happens once per document, not once per decision. Add a second classifier — urgency, language, sentiment, whether it needs a human — and it reuses the same vector at zero marginal inference cost. The LLM approach pays the full price again for each one.

The two cost models, worked

Take a million items a month, averaging 300 tokens each, into 40 categories. The comparison people make ignores the prompt, which is the dominant term:

EMBEDDING ROUTE
  input tokens   = 1e6 * 300                    = 300,000,000
  at $0.02 / 1M  (OpenAI text-embedding-3-small,
                  published launch rate, Jan 2024)  = $6.00
  classifier inference                              = $0 (local matmul)
                                                     ------
                                                      $6.00

LLM ROUTE
  per item: 800 tokens of taxonomy + instructions
          + 300 tokens of item                   = 1,100 input tokens
  input tokens   = 1e6 * 1100                    = 1,100,000,000
  output tokens  = 1e6 * 5                       =     5,000,000
  at $0.15 / 1M in and $0.60 / 1M out            = $165.00 + $3.00
                                                     ------
                                                    $168.00

Twenty-eight times, and the ratio is driven mostly by the 800 tokens of taxonomy resent with every single item. Prompt caching narrows that gap where the provider offers it — the taxonomy is a perfect cached prefix — so run the arithmetic with the cached rate if you have it. Even then the shape holds, because the embedding route has no prompt at all.

Latency separates them further and in a way that is not about money. An embedding call plus a local matmul is one network round trip; a classification call to a chat model is a round trip plus generation. For a synchronous path — routing a ticket while the form is submitting — that difference is the product decision, not the bill.

The accuracy question, honestly

Nobody can tell you which is more accurate on your taxonomy, and any page that gives you a single number for “embeddings versus LLM classification” has made it up. What can be said is where each approach has a structural advantage, and how to settle it in an afternoon.

Embeddings plus a linear model have the advantage when categories are topical — distinguished by what the text is about — and when you have labelled data. They learn your actual label boundaries, including the idiosyncratic ones your taxonomy has for historical reasons that no prompt would ever capture. A language model has the advantage when the distinction requires reasoning over the content rather than recognising its subject: whether a message contains a commitment, whether a review describes a safety issue, whether the policy was violated. Those are not directions in embedding space.

The settling procedure: hold out 20% of your labelled data, run both, and compare macro-F1 rather than accuracy — with 40 imbalanced categories, accuracy is dominated by the three largest and will tell you nothing. Then read the confusion matrix. If the errors concentrate in a few genuinely subtle category pairs, a cascade fixes it. If they are spread everywhere, your taxonomy is ambiguous and no classifier will save it.

There is one asymmetry worth planning around regardless of the result. The linear model needs labelled data and the language model does not, so a new category is a prompt edit in one design and a labelling exercise in the other. If your taxonomy changes monthly, that difference outweighs the cost ratio. If it is stable — and most operational taxonomies are, because they are wired into downstream routing — the labelled set is a one-time cost that keeps paying, and it also becomes the training data for fine-tuning the embedding model itself if you ever want to.

Building one in an afternoon

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
import numpy as np

X = np.load("embeddings.npy")     # (n, 1536), already normalised
y = np.load("labels.npy")

clf = LogisticRegression(max_iter=2000, C=1.0, class_weight="balanced")
print(cross_val_score(clf, X, y, cv=5, scoring="f1_macro").mean())

clf.fit(X, y)
proba = clf.predict_proba(X_new)
conf  = proba.max(axis=1)         # the number the cascade below uses

class_weight="balanced" is not optional on a real taxonomy; without it the model learns to predict your three largest categories and reports a flattering accuracy. Five hundred to a thousand labelled examples per class is comfortable; a hundred is usually enough to know whether the approach works at all, which is the question you are answering this afternoon.

The cascade that gets you both

The classifier emits a confidence. Route on it: take the linear model’s answer when it is confident, and send only the rest to a language model.

if conf >= 0.85:   use the linear model's label
else:              call the LLM with the top 3 candidate labels

if 88% of traffic clears the threshold:
    LLM cost = 0.12 * $168 = $20.16
    plus the $6 of embeddings          -> $26.16/month vs $168
    and the escalated prompt is shorter: 3 candidate labels, not 40

Choose the threshold from the held-out set by plotting accuracy against the fraction of traffic auto-handled, and pick the point where accuracy on the auto-handled slice meets the standard you would have demanded of the language model. That plot also gives you the operational lever: the day quality matters more than cost, you lower the threshold and more traffic escalates, with no retraining and no deployment.