Embedding Search Returns Nonsense

If your vector search returns results with no relationship to the query, the cause is structural — a model mismatch, a metric mismatch, or an index that does not contain what you think it does — and all three are settled by three quick checks. If the results are merely mediocre, that is a different and longer list.

Nonsense and merely-poor are different bugs

Look at the top ten results for a query whose correct answer you know, and classify what you see. Random-looking results, or the same documents returned for every query, mean something is broken. Plausible but unhelpful results mean the pipeline works and the quality is the problem. Chasing the second list while you have the first wastes days.

Three checks that take five minutes

1. Are the query and the documents from the same model? Vectors from two different embedding models occupy unrelated spaces, and similarity between them is noise. This survives a deploy unnoticed because nothing raises: dimensions sometimes even match. The test is to re-embed a document you have already indexed and compare it with its stored vector.

   import numpy as np
   
   stored = index.get_vector(doc_id)
   fresh  = embed(get_text(doc_id))          # today's model, today's code
   cos = np.dot(stored, fresh) / (np.linalg.norm(stored) * np.linalg.norm(fresh))
   print(f"self-similarity: {cos:.4f}")
   # ~1.000  -> same model, same settings. Move to check 2.
   # 0.3-0.8 -> a different version, or different preprocessing.
   # ~0.0    -> a completely different model. Re-embed everything.

   A value near zero means the index must be rebuilt; nothing else will help. Model identity should be stored alongside the vectors so this is a lookup rather than an experiment — re-embedding migrations covers doing it without downtime.
2. Does the metric match the vectors? Cosine similarity and dot product are identical for normalised vectors and entirely different otherwise. An index configured for inner product over unnormalised vectors ranks by magnitude, which usually means long documents win every query regardless of content — a distinctive symptom worth recognising. Check the norms:

   norms = [float(np.linalg.norm(v)) for v in sample_vectors]
   print(min(norms), max(norms))
   # all ~1.0 -> normalised; cosine and dot product agree
   # varied   -> unnormalised; the index metric must be cosine, or
   #             normalise at write time AND at query time

   Whichever you choose, apply it in both places. Normalising at index time and not at query time is a common half-migration.
3. Does the index contain what you think? Compare the row count in your source of truth with the vector count in the index, and spot-check that a document you know exists is retrievable by ID. Partial ingestion, a failed batch, an embedding job that errored on 30% of documents and logged nothing, or a filter applied at write time all present as “the search is bad”. Also check freshness: if documents changed and vectors did not, the index is answering last month’s question — see index freshness.

Six causes of merely-poor results

• Asymmetry, unhandled. A short question and a long passage are different kinds of text. Several embedding families are trained for this with required prefixes — a query prefix and a passage prefix — and omitting them, or applying the same one to both sides, measurably degrades retrieval. Check the model card’s usage instructions and apply them literally; asymmetric embeddings explains why it matters.
• Chunking. Chunks that are too large dilute the embedding across several topics, so nothing matches strongly. Chunks that are too small lose the context that made them meaningful — a fragment saying “it costs 40 dollars per seat” is unfindable if the product name was in the previous paragraph. Read your actual chunks; the fault is usually visible immediately. Chunking strategies covers the trade.
• The query is not a semantic query. Part numbers, error codes, SKUs, person names and exact quotations are what lexical search is for. Embeddings place ERR-4471 and ERR-4472 close together, which is exactly wrong. If your failing queries are identifiers, add BM25 and fuse the results — hybrid search is the standard answer and it is not a large change.
• Filtering applied after retrieval. Fetching the top 10 and then filtering by tenant, date or category leaves you with two results, or none. The filter must be applied inside the search so the top 10 are the top 10 matching documents — metadata filtering covers pre-filtering support, which not every index has.
• Approximate-index parameters. HNSW and its relatives trade recall for speed, and an aggressively tuned index genuinely misses documents that exist. Compare against an exact brute-force search over a sample: if exact search finds the document and the index does not, raise the search-effort parameter rather than blaming the embeddings. HNSW covers the parameters.
• Dimension truncation applied inconsistently. With Matryoshka-style models you may store 256 dimensions and query with 768, or truncate without renormalising. Both produce quietly wrong rankings rather than an error — Matryoshka embeddings.

A seventh, for multilingual corpora: a model that is not multilingual will not match a French query to an English document, and it will fail silently rather than complaining.

When retrieval was fine and the answer was not

A large fraction of tickets filed as “the search is bad” turn out to be generation failures on correctly retrieved context. The two are worth separating before you touch the index, because the work is entirely different and the retrieval work is the more expensive of the two.

The test is direct: print the retrieved chunks alongside the answer, and read them yourself.

hits = search(query, k=10)
for i, h in enumerate(hits):
    print(f"--- {i} score={h.score:.3f} id={h.id}")
    print(h.text[:400])
print("=== ANSWER ===")
print(generate(query, hits))

• The correct chunk is in the list and the answer ignored it. A generation problem. Common causes: the answer was at rank 8 and got lost in the middle of the context, the prompt does not tell the model to prefer the context over its own knowledge, or ten chunks of contradictory material gave it a choice it resolved badly. Passing three reranked chunks rather than ten usually helps more than any index change.
• The correct chunk is not in the list, but exists in the index. A ranking problem — go back to the six causes above, and try a reranker before anything structural.
• The correct chunk is not in the index at all. An ingestion problem. Check whether the source document was parsed: PDFs, tables and scanned pages routinely produce empty or garbled text that gets embedded anyway, and an embedding of whitespace is a valid vector that matches nothing.
• No chunk contains the answer because no chunk could. Questions requiring aggregation across many documents — how many, which is largest, what changed — are not retrieval questions, and no amount of tuning makes them work.

The third bullet deserves a standing check. Count the indexed chunks whose text is shorter than a few dozen characters; a cluster of them means a parser is failing silently on a document type, and those documents are effectively absent from your search. Generation failures over good context covers the first bullet in depth.

Measuring before changing anything

Every item above is a hypothesis, and hypotheses need a number. Twenty to fifty real queries with known correct documents is enough to make retrieval changes decidable, and it is an afternoon of work.

GOLDEN = [
    ("how do I rotate an API key",        {"doc_88", "doc_211"}),
    ("error ERR-4471 on upload",          {"doc_17"}),
    # ... 20-50 rows, taken from real queries, labelled by hand
]

def recall_at_k(k=10):
    hits = 0
    for q, expected in GOLDEN:
        got = {d.id for d in search(q, k=k)}
        hits += len(got & expected) > 0
    return hits / len(GOLDEN)

print(f"recall@10 = {recall_at_k():.1%}")

Record the number before you change anything. Without a baseline, every change feels like an improvement and the third change silently undoes the first. Two further notes: measure retrieval separately from generation, because a retrieval fix and a prompt fix are different work and mixing them means neither is attributable; and keep the golden set in version control next to the code, because it is the only artefact here that does not go stale. RAG evaluation covers the fuller measurement, and reranking is usually the cheapest quality win once retrieval is structurally correct.