Selective Attention: Filtering Before You Fill

Shrinking blocks to fit is the second-best move. The first is deciding that a block should not be there at all — filtering is free in window terms, and truncation never is.

Filtering happens before allocation

Order of operations matters here and it is frequently reversed. Assembly should gather candidates, filter them, and only then allocate room among the survivors. Filtering after allocation is pointless — the damage is done — and truncating instead of filtering produces the worst possible artefact: several partial documents, each missing its conclusion, none of them useful.

Compare the two dispositions of a 30,000-token budget across ten candidate documents of 8,000 tokens each. Truncate everything to fit: ten documents at 3,000 tokens, each cut mid-argument, nine of them irrelevant to the question and all ten unreliable. Filter to the three that scored above threshold and give them the budget: three complete documents, 10,000 tokens each, whole. The second is better on every axis and it is the same number of tokens.

The strong form of the rule: prefer fewer complete things to more partial things. It is also what the allocator’s floor encodes — below the floor, drop rather than shrink.

The gate

A filter is a scoring function plus a threshold plus a cap. All three parts are needed: a score with no threshold is a ranking, and a threshold with no cap still admits forty documents on the day the corpus is unusually relevant.

type Candidate = { id: string; kind: string; tokens: number; text: string };

async function gate(cands: Candidate[], request: Request, budget: number) {
  const scored = await Promise.all(cands.map(async c => ({
    c,
    score: await relevance(c, request),   // 0..1
  })));

  const kept: Candidate[] = [];
  let used = 0;
  for (const { c, score } of scored.sort((a, b) => b.score - a.score)) {
    if (score < THRESHOLD) break;             // hard floor on relevance
    if (kept.length >= MAX_ITEMS) break;      // cap regardless of score
    if (used + c.tokens > budget) continue;   // skip; a later one may fit
    kept.push(c); used += c.tokens;
  }
  return { kept, dropped: scored.length - kept.length };
}

continue rather than break on the budget line is deliberate: a document too large to fit should not terminate the loop, because a smaller high-scoring one behind it may fit. This is a knapsack problem being solved greedily, which is the right amount of effort for the decision it is making.

The relevance function has three common implementations with very different costs. Lexical overlap or a keyword match is free and surprisingly effective for filtering out the obviously irrelevant. Embedding similarity costs one small model call and catches paraphrase. A cheap model asked “could this document help answer this question? yes/no” is the most accurate and the most expensive. Which is right is what the next section is for. The quality of the ranking itself — reranking, cross-encoders, recall metrics — is the retrieval cluster’s subject; here the concern is only whether the gate is worth running.

Does the filter pay for itself?

A filter has a cost. If it is a model call, it may cost more than the tokens it removes. The comparison, with F the filter’s cost in equivalent input tokens, T the tokens it removes from the main call, and m the ratio between the main model’s price and the filter model’s:

filter pays when    T  >  F / m

Assume (ASSUMPTIONS): the main model costs 20x the filter model (m = 20),
the filter reads all 80,000 candidate tokens and emits a little (F ≈ 80,000
input tokens on the cheap model), and it removes 50,000 tokens.

  F / m = 80,000 / 20 = 4,000 equivalent main-model tokens
  T     = 50,000
  -> pays by roughly 12x

Now change one assumption. If the filter is another call to the same model, m = 1, and the filter must remove more tokens than it reads — which a filter that reads every candidate cannot do. That is why relevance filtering with a full-price model is almost always a loss on cost, and only worth it when the win is accuracy rather than money.

Embedding-based filters change the shape entirely, because embeddings are priced far below generation and, more importantly, the corpus side is embedded once and reused. The per-request cost is one embedding of the query plus a vector search, which is small enough that the arithmetic is rarely close.

When stuffing is correct

The honest cases, because a page that claims filtering always wins is selling something.

• The whole corpus is small. If everything fits in 5,000 tokens, filtering adds a failure mode — the filter drops the one relevant document — in exchange for a saving that does not matter. Send all of it.
• The corpus is static and the prefix is cacheable. A stable block charged at the cached rate may be cheaper carried whole than filtered per request, because filtering makes it volatile and therefore uncacheable. Filtering can genuinely make a prompt more expensive by this route, which is counter-intuitive enough to catch people twice.
• The question needs aggregation. “How many of these contracts mention indemnity?” cannot be answered from a filtered subset, because the filter has pre-empted the count. Any question whose answer depends on the whole set defeats relevance filtering by construction.
• A miss is unacceptable. Compliance boundaries, safety rules, refusal policy. A filter is a recall risk, and for material whose absence changes what the system is allowed to do, the correct recall is 100%.

The asymmetry that decides the threshold

Setting THRESHOLD is a trade between two errors that are not equally bad, and knowing which way it leans is most of what makes a filter usable.

A false negative — dropping something relevant — produces a wrong or incomplete answer, and it is invisible: nothing in the output says the model was missing a document. A false positive — keeping something irrelevant — costs tokens and adds a distractor, which degrades the answer somewhat and is at least recoverable, because the material is present and the model may simply ignore it.

The errors are not symmetric, so the threshold should not be set as if they were: lean permissive, and control volume with MAX_ITEMS and the budget rather than by raising the score floor. A permissive gate that admits eight documents of which five are useless is usually a better system than a strict one that admits three and misses the right one on 5% of requests — and unlike the strict gate, its failures are visible in the token count rather than in the answer quality.

One caveat pulls the other way and belongs on the same page: distractors are not entirely harmless. Material that is topically similar but wrong is the most dangerous thing a filter can admit, because it is close enough to be used. If your false positives are near-misses rather than obvious noise, tighten the threshold — the asymmetry argument assumes irrelevant means recognisably irrelevant.