Putting a Classical Model and an LLM in One System

The useful question is not which is better. It is which part of the request each one should handle. There are three arrangements that repeatedly work in production — the classical model as a router, as a consumer of extracted features, or as a verifier — and each has a different economic case.

What each side is actually good at

Arrangement 1: the classical model routes

A cheap model handles what it can and passes the rest on. It is the same pattern as the cheap filter and model cascading, with a gradient-boosted classifier as the first stage instead of a smaller language model.

1. Score every incoming item with the classical model. Support ticket triage, moderation, document classification, lead qualification — anything with a high-volume easy majority.
2. Resolve the confident cases directly. Two thresholds rather than one: above the upper, act as positive; below the lower, act as negative; between them, escalate. Both thresholds come from costs, exactly as in the threshold page, with the escalation cost as the third option in the comparison.
3. Send only the middle band to the language model. These are the ambiguous items, which is where its ability to read context earns the money it costs.
4. Log both decisions on the escalated items. The language model’s answers on the hard cases are labelled training data for the next version of the classical model, which widens the confident band over time.

The last step is what makes the arrangement improve rather than just exist. Each month the cheap model absorbs a little more of the traffic, and the expensive path shrinks towards the genuinely hard residue.

The routing arithmetic

Whether the router pays is a calculation, and it converts the vague question “is the filter accurate enough” into a price per mistake. Rates below are illustrative placeholders; substitute your own and the structure holds.

Assume, per month:

  volume                          1,000,000 requests
  cost of one language-model call    £0.002        (illustrative)
  cost of one classical inference    negligible
  fraction the filter resolves         85%

BEFORE
  1,000,000 x £0.002                              = £2,000 / month

AFTER
  150,000 escalated x £0.002                      =   £300 / month
  850,000 x classical inference                   ~     £0
                                                     -------
  saving                                          = £1,700 / month

WHAT THE FILTER'S MISTAKES MAY COST

  Suppose the filter resolves 2% of its 850,000 wrongly:
    850,000 x 0.02 = 17,000 wrong decisions per month

  Break-even cost per wrong decision:
    £1,700 / 17,000 = £0.10

So the router pays if and only if a wrong auto-resolution costs less
than 10 pence. For a misrouted support ticket that is plausibly true.
For a wrongly auto-approved refund it is obviously false, and the
correct design is a much narrower confident band -- or none.

That last line is the point of the derivation. The threshold is not set by the filter’s accuracy; it is set by what its errors cost against what its coverage saves, and those two numbers come from different people in the organisation.

Arrangement 2: the LLM makes features

A tabular model cannot read a support ticket, a contract or a product description. A language model turns unstructured input into columns, and the gradient-boosted model — which remains the thing making the decision — gets a wider table.

raw:      free-text complaint, 400 words

extracted (one LLM call, structured output, cached by content hash):
  issue_category          enum of 12
  product_mentioned       enum, nullable
  severity_expressed      ordinal 1-5
  refund_requested        boolean
  previous_contact        boolean

joined onto the existing feature table:
  account_age_days, plan_tier, tickets_90d, spend_12m, ...

model: HistGradientBoostingClassifier over all of it

Three properties make this the strongest of the three arrangements. Extraction is cacheable, because the same text always yields the same fields, so cost is per unique document rather than per decision. Extraction runs offline, so the language model is off the latency path entirely. And the decision stays with a model that is auditable, calibratable and stable — which matters when the decision has to be explained.

• Constrain the output to a schema. Enumerations, not free text; a fixed list of twelve categories rather than whatever phrase the model produced. A feature whose value space drifts is a feature that breaks the model silently. Structured output and enums over free text cover the mechanics.
• Version the extraction prompt with the model. A changed prompt is a changed feature distribution. If the extractor is updated without a refit, you have manufactured train-serve skew in one commit.
• Pin and monitor the extractor. A provider updating the model underneath you shifts every extracted feature at once. Keep a fixed evaluation set of documents with known correct extractions and re-run it on a schedule; a drop there is the earliest possible warning.
• Embeddings are the other option. Where the text has signal you cannot enumerate, an embedding reduced to a handful of dimensions can go straight into the table. Classification over embeddings covers when this beats extraction, and it costs interpretability that the enumerated version keeps.

Arrangement 3: the classical model verifies

Reverse the order. The language model produces an answer; a small supervised model, trained on human review outcomes, predicts whether this particular answer will be accepted. Below a threshold, the item goes to a person.

The verifier’s features are cheap and available: output length, whether the output contains a value that appears in the source document, the number of retries the call took, the extraction confidence for each field, and any structural check you can run deterministically — does the total equal the sum of the lines, is the date within range, does the referenced identifier exist.

It is worth building for one specific reason: it produces a calibrated probability, which a language model’s self-reported confidence does not. That means the escalation threshold can be derived from the cost of a review against the cost of a wrong answer, using exactly the arithmetic in the threshold page. Related mechanisms sit in teaching a model to abstain and the verification pattern.

The training data comes free if you are already reviewing output. Every human decision on a generated item is a label. After a few thousand reviews the verifier is usually good enough to cut the review queue substantially at a fixed error rate, which is a straightforward headcount argument.

Three arrangements that do not work

• Replacing a working tabular model with a language model. Feeding rows as text to be classified is slower by orders of magnitude, more expensive by more, less accurate on the task the tabular model was fitted for, and produces no calibrated probability. It is occasionally the right call for a cold start with no labels at all — and then the first thing to do is collect the labels and fit the classifier.
• Asking a language model for the numeric prediction. “Estimate this customer’s churn probability” returns a plausible number with no relationship to any frequency. Every downstream expected-value calculation is then arithmetic on a fabricated input.
• Chaining without a budget. Extractor, then reasoner, then verifier, then summariser: four calls where the tabular model needed none, with the error rates compounding and the latency adding. Count the calls per decision and put a ceiling on it before the architecture sets.

Operating a hybrid

1. Two different retraining cadences. The classical model is retrained on your schedule; the language model changes on the provider’s. Only one of those is under your control, which is a reason to pin versions and to keep the extraction evaluation set described above.
2. One monitoring surface, not two. The router’s escalation rate, the extractor’s field distributions and the verifier’s score distribution all belong on the same dashboard as the model’s own drift metrics. A shift in the escalation rate is the earliest signal available that something upstream changed, and the drift page treats it as one series among the rest.
3. Fail towards the classical path. When the language model is unavailable, the system should degrade to the tabular decision plus a wider human queue, not to an error. This is the argument for keeping the classical model in the loop even where the language model is more accurate.
4. Attribute cost per decision, not per month. The number that governs the design is cost per resolved item, split by path. A monthly total hides a small band of escalations consuming most of the budget, which is the usual shape.