Unit Testing a Retriever With a Fixed Fake Corpus

Retrieval failures reach you as bad answers, which is the hardest possible place to diagnose them. A retriever tested on its own, against eight documents you wrote, turns “the answer was wrong” into “document four did not come back”.

Building a corpus you can argue about

The corpus needs to be small enough to hold in your head and adversarial enough to be worth running. Eight to fifteen documents is the range where both are true. Every query in the fixture has exactly one document that answers it, and at least one that looks like it should.

# tests/fixtures/corpus.py
CORPUS = [
    ("d1", "Refunds are issued to the original payment method within 5 "
           "business days of approval."),
    ("d2", "Returns must be posted within 30 days of delivery. Print the "
           "label from your account."),
    ("d3", "Refund approval requires a photo when the item arrived damaged."),
    ("d4", "Our warehouse in Porto handles all EU returns processing."),
    ("d5", "Gift cards are non-refundable and cannot be returned."),
    ("d6", "Delivery normally takes 2 to 4 business days within Portugal."),
    ("d7", "Payment methods accepted: card, SEPA direct debit, and PayPal."),
    ("d8", "Damaged items should be reported through the app within 48 hours."),
]

QUERIES = [
    ("how long does a refund take", "d1", "d3 shares 'refund'; d6 shares 'days'"),
    ("what if my order arrived broken", "d3", "no lexical overlap with 'broken'"),
    ("can I return a gift card",        "d5", "d2 is the generic returns doc"),
    ("where do returns get processed",  "d4", "d2 is about how, d4 about where"),
]

The third element of each query row is the trap that row exists to catch. “How long does a refund take” has two documents containing refund and one containing days, so a retriever that has quietly degraded to keyword overlap gets it wrong in a specific, readable way. “Arrived broken” shares no content word with d3 at all, so it fails the moment semantic matching stops working — which is exactly what happens when an embedding call silently returns zeros and nobody checks.

Write the corpus in the register of your real documents. A corpus of one-line sentences will not exercise chunking, will not exercise the length normalisation in your scoring, and will pass while the real index fails.

Assert on rank, never on score

Similarity scores are not stable across embedding model versions, index backends, or even index parameters. A test asserting that d1 scores above 0.82 fails on an upgrade that improved retrieval. Assert on ordering and membership instead: those are what the retriever is for.

import pytest
from tests.fixtures.corpus import CORPUS, QUERIES

@pytest.fixture(scope="module")
def retriever(fake_embedder):
    r = InMemoryRetriever(embedder=fake_embedder)
    r.index([(doc_ref, text) for doc_ref, text in CORPUS])
    return r

@pytest.mark.parametrize("query,expected,trap", QUERIES)
def test_expected_document_is_first(retriever, query, expected, trap):
    hits = retriever.search(query, k=3)
    refs = [h.ref for h in hits]
    assert refs[0] == expected, f"got {refs}; trap was: {trap}"

@pytest.mark.parametrize("query,expected,trap", QUERIES)
def test_expected_document_is_in_top_three(retriever, query, expected, trap):
    refs = [h.ref for h in retriever.search(query, k=3)]
    assert expected in refs, f"got {refs}; trap was: {trap}"

Two tests rather than one, because they fail for different reasons and you want to know which. Top-1 slipping to top-2 is a ranking wobble and often tolerable; falling out of the top three entirely means the document is not being found at all, which is a different bug with a different fix. Putting the trap description in the assertion message means a failure explains itself without anyone opening the fixture.

Keep k equal to what production uses. A test at k=10 against a system that passes 3 chunks to the model is asserting a property nobody depends on.

The embedder is the other variable

A retriever test that calls a real embedding API is slow, costs money per run, needs a key in CI, and changes behaviour when the provider updates the model. For the properties above, none of that is buying you anything, because you are asserting on ordering within a corpus you designed.

A deterministic fake embedder — a hashed bag-of-words projected into a fixed number of dimensions, seeded once — makes the whole suite instant and reproducible. Its vectors are meaningless as semantics, which is precisely the limitation to be honest about: it will not match “broken” to “damaged”. So split the suite. Structural tests — ordering is stable, k is respected, filters apply, an empty query does not crash, a duplicate document does not appear twice — run against the fake on every commit. Semantic tests, including the “arrived broken” row above, run against the real embedder on a slower schedule, marked so they can be deselected.

Whichever you use, cache embeddings for the fixed corpus rather than recomputing them per test. The corpus does not change during a run, so a module-scoped fixture that indexes once turns a suite of forty queries into forty comparisons rather than forty index builds.

The fake also enables a check that is otherwise awkward to write: an assertion that the embedder received the text you think it did. Text reaches an embedder through a surprising amount of preprocessing — whitespace collapsing, case folding, a query prefix that some models expect and others do not — and a spy on the fake shows you the final string. A prefix applied at index time but not at search time degrades every result, raises no error, and is invisible to any ranking assertion on a corpus this small.

What a tiny corpus can and cannot show

Be clear about the limits, because this is where these tests get oversold. Eight documents cannot tell you anything about recall at scale: with a corpus that small, almost any retriever returns the right answer in the top three, and an approximate index will behave exactly like an exact one because it never has to approximate. Nothing here predicts behaviour at a million vectors.

What it does show is that the pipeline is wired up and has not regressed in an obvious direction. Text reaches the embedder unmangled. The index stores and returns what it was given. The scoring function orders things sensibly. Metadata survives the round trip. Those are the things that break during a refactor, and they are the things a full-scale evaluation is far too slow and too noisy to pin down. Pair this with a proper labelled set at realistic scale for the recall question; the two suites answer different questions and neither replaces the other.

The part that is not fuzzy at all

Metadata filtering has no similarity in it and deserves exact assertions. If a document is tagged lang="pt" and the query filters for lang="en", it must not appear — ever, at any k, at any score. This is where the security-shaped bugs live: a tenant filter that is applied after the top-k cut rather than before does not merely return fewer results, it can return none while a correct implementation would have returned five, and in a multi-tenant index the mirror-image mistake returns another customer’s document.

Add a handful of documents to the corpus carrying a tenant tag, and assert that a query filtered to tenant A returns only tenant A documents and that the count is right. Assert the empty case too: a filter matching nothing returns an empty list rather than falling back to unfiltered results, which is a real and popular failure. These assertions are exact, fast, and worth more than every score-based test you could write.