Personalisation vs Relevance: The Trade-off

Personalisation is usually argued about as a philosophy. It is easier to reason about as a weighted sum with one parameter, because then the question “when does this make results worse?” has an actual answer.

The knob, written down

Almost every personalised ranker, whatever it is called internally, reduces to this:

score(d | q, u) = (1 - lambda) * rel(d, q)  +  lambda * aff(d, u)

  rel(d, q)  query-document relevance, the same for everyone
  aff(d, u)  affinity between this document and this user
  lambda     how much of the ordering the personal term is allowed to own

Now the analysis. Let r be true relevance, the thing you would rank by if you were omniscient. The blended score correlates with r through the first term only, scaled by (1 - lambda). The second term contributes signal exactly to the extent that aff is correlated with r for this particular query — and contributes pure noise to the extent that it is not.

So the condition is sharp, and it is a property of the query rather than of the user. If a query has one correct answer that is the same for every person who could type it, then aff is uncorrelated with r by construction, and any lambda greater than zero strictly degrades expected ranking quality. Personalisation is not a feature you turn on. It is a bet that the query is ambiguous, and on the queries where it is not, the bet always loses.

It is worth separating two things that get lumped under the same word. Session context — the previous query, the category being browsed, the filters already applied — is cheap, unambiguous and almost always right, because the user supplied it deliberately and minutes ago. A long-run profile assembled from months of behaviour is the expensive, risky version: it needs storage, a retention policy and a privacy story, it goes stale, and it produces the failures people actually complain about. A good deal of what gets credited to “personalisation” is really session context, and it is worth exhausting that before building anything that remembers.

How much room a query leaves

This has been studied properly, and the studies are worth reading rather than summarising into a number. Teevan, Dumais and Horvitz (2007) approached it by having multiple people judge the same results for the same query and treating their disagreement as the headroom: where everyone agrees on what is relevant, there is nothing for personalisation to recover, and where they disagree, the gap between the best possible single ranking and the best possible per-person ranking is what a personalised system could in principle win. Their framing — the “potential for personalisation” — is the right mental model, and the important part of it is that the headroom varies enormously between queries and is zero for many of them.

Dou, Song and Wen (2007) came at the same question from logs rather than judges, and their contribution is the more operationally useful one: click entropy as a per-query estimate of ambiguity, computed from data you already have.

Click entropy as the gate

For a query q, take the historical distribution of clicks across results and compute its Shannon entropy:

H(q) = - SUM over results d of  p(d | q) * log2 p(d | q)

  p(d | q)  share of clicks on q that landed on d

Two examples, both computed here rather than observed anywhere. A query where 90% of clicks go to one result and 10% to another:

H = -(0.9 * log2 0.9 + 0.1 * log2 0.1)
  = -(0.9 * -0.1520 + 0.1 * -3.3219)
  =   0.1368 + 0.3322
  =   0.469 bits

And a query whose clicks are spread evenly over five results:

H = log2 5 = 2.322 bits

The first query has one answer and personalising it can only hurt. The second has five plausible answers and different people want different ones — that is where lambda should be non-zero. Gating on a threshold somewhere in between, and setting lambda as a function of entropy rather than as a constant, is the whole design. It also means personalisation degrades gracefully on new queries: no click history, no entropy estimate, no personalisation, which is the correct default.

The obvious caveat is that click entropy is computed from clicks, and clicks are position-biased. A query whose historical ranking was confidently wrong will show low entropy because everybody clicked the top result. Correct for propensity first — the arithmetic is in the metrics page — or at minimum treat the entropy estimate as a lower bound on ambiguity.

The loop that closes on itself

Personalisation has a structural problem that ordinary ranking does not. The model is trained on clicks; the clicks came from results the model chose; so the model is trained on its own output. Write the cycle out:

model promotes item i for user u
  -> i is examined more often            (higher e_i)
  -> i accrues more clicks                (even at constant relevance)
  -> training data shows i is good for u
  -> model promotes i harder

Nothing in that loop requires i to be good. The examination term alone drives it, which is the same position-bias mechanism that makes raw click-through rate uninterpretable, now compounding over training cycles. The result is a profile that narrows on whatever it happened to show early, and it is indistinguishable from the model having learned something.

Two things break the loop, and you need both. Propensity weighting removes the examination term from the training signal. Deliberate exploration — a small share of impressions given to items the model would not have chosen — creates the counterfactual data the model needs to ever change its mind. The bandit machinery for spending that exploration budget efficiently is in the cold-start page, and it is the same machinery for the same reason.

Guardrails

• Cap the personal term’s authority. A bound on lambda means personalisation can reorder within a band of comparable relevance but cannot promote an irrelevant document. This single constraint prevents most of the failures people attribute to personalisation.
• Decay the profile. Interests are non-stationary and some sessions are not about the user at all — buying a gift, doing research for somebody else. An exponential decay on the affinity vector, with a half-life in weeks, is the difference between a profile and a permanent record of one afternoon.
• Make it visible and reversible. “Because you looked at X”, with a way to say no. This is a relevance mechanism as much as a trust one: it is the cheapest source of explicit negative feedback you will ever get.
• Evaluate it separately. A personalised system cannot be judged by a single global nDCG, because its whole claim is that different users should see different orderings. Segment the metric by the entropy band of the query, and check that the low-entropy band has not regressed. That is where the damage shows up.