Pattern: The Cheap Filter in Front of the Expensive Model

Most expensive model calls are made about inputs that did not need one. The pattern is to put something cheap in front that decides, and the entire question is whether the cheap thing is cheap enough and accurate enough to be worth its own existence. That is an inequality, and you can write it down.

The shape

A filter sits between the request and the expensive call and answers one question: does this input need the expensive call at all? It is not a smaller version of the task. It is a different, much easier task, and that asymmetry is the whole source of the win.

• Relevance. Is this support message about anything the assistant handles? Is this document one of the types we extract from? Most of what arrives at an inbox-shaped feature is not.
• Sufficiency. Can this be answered from a cache, a lookup table or a template? A large share of questions to any assistant are the same twenty questions.
• Triviality. Is this input empty, one word, a duplicate of the previous one, or otherwise not worth reasoning about? This is the most embarrassing traffic to pay frontier prices for and the easiest to remove.
• Abuse. Is this the same user submitting their hundredth request in a minute? A filter here is also a denial-of-wallet control, which is a different justification for the same component.

The filter need not be a model. A regular expression, a length check, a hash lookup against recent inputs and an embedding similarity search are all filters, and they are two to four orders of magnitude cheaper than a generation. Reach for a small classifier only when the deterministic options genuinely cannot express the question.

When it pays: the inequality

Here is the derivation, with every symbol something you supply. Let C be the cost of the expensive call, c the cost of the filter, and p the fraction of traffic the filter correctly removes.

without filter, per request:   C
with filter, per request:      c + (1 - p) * C

it pays when:                  c + (1 - p) * C  <  C
                        <=>    c                <  p * C
                        <=>    c / C            <  p

  "the filter must cost a smaller fraction of the expensive call
   than the fraction of traffic it removes."

Worked with a filter costing one hundredth of the expensive call
(c/C = 0.01): it pays as soon as it removes more than 1% of traffic.
Worked with a filter that is itself a model call at one fifth the cost
(c/C = 0.2): it must remove more than 20% of traffic to break even,
which is a much harder bar and the reason a model-based filter is
often not worth it where a deterministic one is.

Two consequences follow immediately. First, the ratio matters far more than the accuracy: a filter that is a thousand times cheaper pays for itself on almost any removal rate, which is why the deterministic checks belong first even when they only catch obvious cases. Second, the inequality ignores errors, and errors are where the real cost is — which is the subject of the next section, and the reason accuracy cannot be optimised in the abstract.

There is a second term worth adding for latency-sensitive features. The filter adds its own latency to every request, including the ones that pass. If the filter takes a meaningful fraction of the expensive call’s time, the passed traffic gets slower in exchange for the filtered traffic getting cheaper, and for an interactive feature that trade is not automatically good.

Which way to be wrong

A filter has two error types with wildly asymmetric costs, and treating it as a balanced classification problem is the standard mistake.

The asymmetry is not close. One error costs a fraction of a cent; the other costs a user experience and is undetectable. So the filter must be tuned to pass when uncertain, not to be accurate — set the threshold so that borderline inputs go through to the expensive path, and accept a lower removal rate as the price of that. The inequality above tells you how much removal you can afford to give up: at c/C = 0.01, a great deal.

There is one important exception. If the filter’s decision is reversible by the user — a “this did not answer my question” control that escalates to the full path — then the false negative becomes cheap and recoverable, and you can bias harder. Building that escape hatch is usually a better investment than improving the classifier.

Building the filter

In cost order, cheapest first. Add layers only while each one still satisfies the inequality on the traffic that reaches it — the second filter sees only what the first passed, so its removal rate is computed on that residual, not on total traffic.

• Structural checks. Length, emptiness, language, content type, an exact-hash match against a recent-inputs table. Microseconds. This layer is pure profit and is skipped embarrassingly often.
• Lookup. An exact or normalised match against known questions and their stored answers. Not semantic yet — normalise whitespace and case and check a map. In any assistant with real traffic, a repeated-question rate exists and is worth measuring before building anything cleverer.
• Embedding similarity. One embedding call, one nearest-neighbour search. Cheap relative to generation and it catches paraphrases the lookup misses. This is also the natural place for a semantic cache; the layering and invalidation questions are the same ones a cache has.
• A small classifier. A single-token classification with a small model, or a trained classifier if you have labels. Last because it is the most expensive and the most likely to fail the inequality on its own.

Instrument every layer with what it removed and what it passed, from the first day. The removal rate per layer is the only way to know whether a layer still earns its place, and it drifts as traffic changes.

How it fails

Three failure modes, in rough order of how often they bite.

• The filter drifts and nobody notices. Traffic changes, the removal rate falls, and the filter becomes a pure cost with no benefit. A removal-rate metric with an alert on a sustained drop is the whole defence, and it takes an afternoon.
• The filter becomes the product. Once it exists, people add conditions to it — routing, personalisation, business rules — until it is a second implementation of the feature that nobody evaluates. Keep it to the single question of whether the expensive call is needed. Anything else belongs downstream.
• The filter is the failure that has no error. When something goes wrong with it, the symptom is that quality dropped for a subset of users, with no exception and no alert. Log the filter decision on every request alongside the outcome, so a complaint can be traced back to “this request never reached the model”. Without that field, this is one of the hardest production problems in the cluster to diagnose.