Relevance Tuning: Making Search Feel Right

“Relevance tuning” sounds like taste. It is not. It is a loop with a measurable objective, and the reason it feels like taste is that most teams skip the step that makes it measurable.

Nothing works without judgements

A judgement list is a set of (query, document, grade) triples where a human has said how good that document is for that query. The methodology dates to the Cranfield experiments of the 1960s and it is still what TREC runs on: a fixed query set, graded judgements, and a metric computed over both. Everything else in this page assumes you have one.

Practical shape. Use graded relevance rather than binary — a four-point scale of 3 = perfect, 2 = good, 1 = marginal, 0 = irrelevant is the common choice and it is what nDCG expects. Sample queries by traffic band rather than uniformly: roughly a third from the head, a third from the torso and a third from the tail, because head queries are the ones you can least afford to regress and tail queries are where the failures live. Write the annotation guideline down before anybody judges anything, and check inter-annotator agreement with Cohen’s kappa; if two annotators cannot agree, the metric computed from their labels is measuring the guideline, not the search engine.

Store it as something diffable, one row per triple, with the annotator and the date. It is an asset that outlives every ranking model you will ship, and the same curation discipline that keeps an eval set alive applies here unchanged.

What k1 and b actually do

BM25 is the default lexical scorer in Lucene, Elasticsearch, OpenSearch and Postgres extensions that implement it, and it has exactly two knobs worth touching. Here it is in full:

score(q, d) = SUM over terms t in q of

                          f(t,d) * (k1 + 1)
      IDF(t)  *  ---------------------------------------
                 f(t,d) + k1 * (1 - b + b * |d| / avgdl)

  f(t,d)  times term t occurs in document d
  |d|     length of d in tokens
  avgdl   mean document length across the index
  IDF(t)  = ln(1 + (N - n_t + 0.5) / (n_t + 0.5))
  N       documents in the index
  n_t     documents containing t
  k1      term-frequency saturation, typically 1.2
  b       length normalisation strength, 0 to 1, typically 0.75

The saturation term

Set b = 0 for a moment so length drops out, and take k1 = 1.2. The term-frequency factor becomes f * 2.2 / (f + 1.2). Evaluate it:

The tenth occurrence of a word has already collected 89% of everything the hundredth will ever get. That is the whole point of k1: it stops a document that repeats a keyword forty times from beating a document that uses it twice in a sentence that means something. Lower k1 saturates faster; higher k1 lets repetition keep paying. On short fields — product titles, headlines — a lower value is usually right, because a repeated term in a twelve-word title is a spam signal rather than a relevance signal.

The length term

Now put b = 0.75 back and take a document twice the average length, so |d| / avgdl = 2. The bracket becomes 1 - 0.75 + 0.75 * 2 = 1.75. For f = 2 the factor is 4.4 / (2 + 1.2 * 1.75) = 4.4 / 4.1 = 1.073, against 1.375 for a document of average length. Twice the length costs about 22% of the term’s contribution. Set b = 0 and long documents win simply by containing more words; set b = 1 and long documents are punished in full proportion to their length, which is wrong for corpora where length correlates with thoroughness.

And IDF, which you do not tune

With N = 1,000,000: a term in half the corpus scores ln(1 + 500000.5 / 500000.5) = ln 2 = 0.693. A term in 100 documents scores ln(1 + 999900.5 / 100.5) = ln 9950.3 = 9.21. Rare beats common by better than thirteen to one, automatically. This is why BM25 remains hard to beat on queries containing a product code or a surname, and it is the failure mode dense retrieval has to be rescued from — the argument in semantic versus keyword search.

The tuning loop

• Baseline. Compute nDCG@10 over the whole judged query set, and record the per-query values, not just the mean.
• Change one thing. A field weight, k1, b, a synonym rule, an analyzer. One.
• Recompute, and diff per query. The mean moving is the least interesting output. The list of queries that got worse is the interesting one, and it is where every genuine insight in relevance work comes from.
• Keep or revert against a threshold you set in advance — see the next section for how to choose it.

The order in which you change things matters more than most tuning guides admit, because the changes are not independent. Work outward from the text pipeline: analyzer and tokenisation first, since everything downstream scores whatever tokens those produce and a stemming decision invalidates every field weight you tuned before it. Then field weights, which is where most of the available improvement actually lives in a system with titles, bodies and attributes. Then synonyms and query relaxation, which change recall rather than ordering. Only after all of that is it worth touching k1 and b, and by then you will usually find the defaults were fine.

How many queries before you believe it

Per-query nDCG varies enormously, so a mean over thirty queries moves by accident constantly. The judged query set is the same for both configurations, which makes this a paired comparison, and the sample size for a paired test is:

n = (z_alpha/2 + z_beta)^2 * sigma_d^2 / delta^2

  z_alpha/2 = 1.96   two-sided, 95% confidence
  z_beta    = 0.84   80% power
  sigma_d   standard deviation of the per-query nDCG DIFFERENCE
  delta     the smallest improvement worth detecting

Assume sigma_d = 0.15, which is a plausible order of magnitude for paired nDCG@10 differences and which you should replace with the value you compute from your own baseline. To detect delta = 0.02:

n = (1.96 + 0.84)^2 * 0.15^2 / 0.02^2
  = 7.84 * 0.0225 / 0.0004
  = 441 queries

for delta = 0.05:
  = 7.84 * 0.0225 / 0.0025
  = 71 queries

Two useful conclusions fall straight out. A judgement set of fifty queries can detect a large change and nothing else, which is fine for catching regressions and useless for tuning. And halving the effect you want to see quadruples the queries you must judge, so decide delta before you commission the judging rather than after. The same arithmetic applied to model evaluation works through the assumptions in more detail.

Four traps

• Tuning on the queries you looked at. If you added judgements because a query looked broken, and then tuned until it worked, you have fitted the training set. Hold out a query set you do not inspect.
• Boosting instead of fixing. A per-query boost rule fixes one query and leaves a permanent, untested interaction with every future change. Keep a count of them; when it grows, the ranking model is what needs work.
• Tuning k1 and b jointly with field weights. They interact, so a grid search over all of them will find a combination that is optimal on your judgements and fragile everywhere else. Fix the analyzer and the fields first; touch k1 and b last, and only if there is a reason.
• Judging from the results page. Annotators shown the current ranking anchor on it. Present documents in randomised order, stripped of position.