Testing Few-Shot Example Selection Logic Independent of the Model Call

Dynamic few-shot selection is a ranking problem sitting in front of a language model, and ranking problems are ordinary software. Almost every bug in this layer — wrong order, a duplicate, a blown token budget, the answer leaking into its own examples — is findable with no model call at all.

Find the seam: selection is not generation

The refactor comes first. Selection must be a function from a query and an example bank to an ordered list of example ids. Not to a rendered prompt, and certainly not to a completion. If your code embeds the query, searches, formats the examples and calls the model in one function, none of the assertions below can be written without mocking the model, and a mocked model test is testing the mock.

Return ids rather than example objects. It makes assertions readable — expect(ids).toEqual(["ex-14", "ex-3", "ex-91"]) is a test somebody can debug from the failure message, whereas a diff of three example bodies is a wall. It also decouples the test from incidental edits to the examples themselves.

Frozen vectors, not a live embedder

The selector needs embeddings, and calling the embedding API in a unit test makes it slow, costly, network-dependent and non-reproducible across provider model updates. Precompute the vectors for the example bank and for a fixed set of test queries, commit them as a fixture, and have the test inject them.

Two consequences worth being explicit about. The test now verifies your ranking logic and not your embedding quality, which is correct — those are separate concerns with separate failure modes and separate fixes. And the fixture goes stale relative to the live embedding model, which is fine for this test and is exactly why a separate check on embedding drift exists as its own concern.

Keep the fixture small and hand-designed rather than sampled. A bank of twelve examples whose similarity relationships you chose deliberately — two near-duplicates, one exact match, one adversarial near-miss, four irrelevant — produces tests whose expected output you can justify. A bank of 2,000 real embeddings produces expected values you copied from the implementation, which asserts only that the code still does what it did.

import { describe, expect, it } from "vitest";
import { selectExamples } from "../src/fewshot";
import bank from "./fixtures/example-bank.json";      // {id, text, vector, label}
import queries from "./fixtures/queries.json";        // {id, text, vector}

const select = (queryId: string, opts = {}) =>
  selectExamples({ query: queries[queryId], bank, k: 3, ...opts }).map((e) => e.id);

describe("few-shot selection", () => {
  it("ranks the exact match first and the adversarial near-miss last", () => {
    expect(select("refund-over-limit")).toEqual(["ex-refund-exact", "ex-refund-similar", "ex-refund-partial"]);
  });
});

Six assertions on the selected set

• Cardinality. Exactly k examples come back when the bank has at least k candidates above the threshold, and fewer — not padding, not an exception — when it does not. Test the underfull case explicitly; it is the one that happens on a new tenant with an empty bank.
• Uniqueness. No id appears twice. Duplicates arrive from a bank containing near-identical entries and from a merge of two retrieval passes, and a duplicated example wastes budget while biasing the model toward one pattern.
• Deterministic order, including ties. Two examples with identical scores must come back in a defined order — by id as a secondary sort, say. Without an explicit tiebreak the order depends on the underlying sort’s stability and on insertion order, and you will eventually chase a heisenbug through a prompt cache that keys on the rendered prompt.
• Token budget. The combined examples fit the budget they were given. Assert against a counted budget, not a guessed one, and test the case where a single example exceeds the whole budget: the selector should skip it and take the next, not return an oversized prompt and not return nothing.
• Diversity, if you claim it. If the selector is supposed to avoid near-duplicates, construct a bank where the top three by similarity are near-identical and assert that the third slot went to something else. A diversity feature with no such test is a feature that quietly stopped working.
• Label balance, if you claim it. For classification few-shots, assert the returned set is not all one label when the bank can supply others. An all-positive example set biases the model toward the positive class, and this is a genuinely common bug.

The leak that makes your eval look brilliant

If your example bank and your evaluation set are drawn from the same historical data, the selector will eventually retrieve the eval case itself as a few-shot example. The model is then shown the answer and asked the question, your scores rise, and nothing in the pipeline is wrong — only the conclusion is.

Assert the exclusion directly: given a query that is byte-identical to a bank entry, that entry must not be selected. Then assert the harder version, which is near-duplicate exclusion — the same case with a different customer name or a reordered clause. Exact-match exclusion by id is easy and insufficient, because real duplication in support and ticket data is rarely byte-exact.

it("never returns the query's own case, exact or near-duplicate", () => {
  expect(select("ex-refund-exact-as-query")).not.toContain("ex-refund-exact");
  expect(select("ex-refund-exact-renamed")).not.toContain("ex-refund-exact");
});

Run that test against your eval harness’s configuration, not just the production one. The two often differ in exactly the way that matters, because the eval harness is the place somebody widened the bank to cover more cases.

Stability under a perturbed query

A selector that returns a completely different set when the query gains a comma will make your whole application feel non-deterministic for reasons the model gets blamed for. Test it: take a query, produce three trivial variants — added punctuation, a changed proper noun, a reordered clause with the same meaning — and assert the overlap with the original selection stays above a threshold you have written down.

const overlap = (a: string[], b: string[]) =>
  a.filter((id) => b.includes(id)).length / a.length;

it.each(["punctuation", "renamed-entity", "clause-order"])(
  "keeps at least two of three examples under a %s perturbation",
  (variant) => {
    expect(overlap(select("refund-over-limit"), select(`refund-over-limit--${variant}`)))
      .toBeGreaterThanOrEqual(2 / 3);
  },
);

The threshold is a product decision rather than a fact, and writing it in the test is how it becomes reviewable. Two of three is a reasonable default for a k of three; the value matters far less than the test existing, because what it really guards against is somebody swapping the similarity metric or normalising vectors differently and not noticing that selection now churns on every keystroke.