Embeddings for Recommendations: The Two-Tower Model

The two-tower model is usually introduced as a neural architecture. It is better understood as the only architecture that can score ten million items in twenty milliseconds, and everything odd about it follows from that constraint.

Why two towers and not one model

A model that reads the user and the item together — a cross-encoder — is more accurate, because it can compute features that depend on both. It is also unusable for retrieval: it must be run once per candidate, so scoring a ten-million-item catalogue means ten million forward passes per request.

The two-tower model gives that up deliberately. One encoder reads the user and their context, another reads the item, and the score is a dot product between the two outputs:

s(u, i) = dot( f(user features), g(item features) )

  f  user tower  -- runs once per request
  g  item tower  -- runs offline, once per item

Because g does not see the user, every item vector can be computed in a batch job and stored. At request time you run f once and turn the problem into nearest-neighbour search over a fixed set of vectors — which is a solved problem with a well-understood cost curve, covered in the HNSW page and the storage-cost page. That is the whole trade: accuracy given up at retrieval, bought back by a cross-encoder over the survivors. The cascade argument from retrieve, rank, rerank arrived here by a different route and reached the same shape.

Note what the item tower can read: text, images, categorical attributes, and an item id embedding. It is the attribute inputs that make the architecture handle new items — a brand new item with no id embedding worth anything still gets a reasonable vector from its text and category. A pure matrix factorisation cannot do that, and this is the practical reason two-tower models replaced it.

In-batch negatives and the logQ correction

Training needs negatives, and there are ten million of them per positive. Sampling them explicitly is expensive, so the standard trick is to reuse the batch: for each (user, positive item) pair, treat the other items in the batch as negatives. The loss is a softmax over the batch:

loss = -log(  exp(s(u, i+))  /  SUM over j in batch of exp(s(u, j))  )

This is free and it is biased, in a way that matters. Batches are drawn from the interaction stream, so an item appears in a batch roughly in proportion to how often it is interacted with. A popular item is therefore sampled as a negative far more often than a rare one, and the model learns to push its score down to compensate for a frequency that has nothing to do with whether it is a good recommendation. Left uncorrected, the system systematically under-recommends exactly the items most people want.

The correction is to subtract the log of the sampling probability from the logit before the softmax:

s_corrected(u, j) = s(u, j) - log p_j

  p_j  probability item j appears as an in-batch negative,
       estimated from a streaming frequency counter

The reasoning is short. Sampled softmax approximates the full softmax denominator by a weighted sample; the weights have to undo the sampling distribution for the approximation to be unbiased. Subtracting log p_j from the logit is exactly dividing exp(s) by p_j inside the sum, which is the importance weight the estimator requires. Yi and colleagues (2019) set this out for large-corpus item recommendation, including a streaming estimator for p_j that works when the item distribution moves; Covington, Adams and Sargin (2016) had described the same retrieval-then-ranking split with a sampled softmax for video recommendation a few years earlier.

If you take one operational thing from this page, take this: when a two-tower model behaves as if it dislikes your bestsellers, the first hypothesis is a missing sampling correction, not a modelling problem. The symptom is distinctive — quality looks fine on offline metrics computed over sampled negatives, because those metrics inherit the same bias.

What serving looks like

OFFLINE, nightly or hourly
  for each item:  vector = item_tower(features)
  build ANN index over all item vectors

ONLINE, per request
  user_vec  = user_tower(user features + context)
  candidates = ann_index.search(user_vec, k = 500)
  filtered   = apply business rules, stock, permissions
  ranked     = cross_encoder(user, filtered)      # or a GBDT
  final      = diversify(ranked)

Two operational details do most of the damage when they are wrong. The towers must be versioned together: an item index built by version 3 of the item tower is meaningless to version 4 of the user tower, and the failure is silent — you get plausible-looking nonsense rather than an error. And rebuilding the item index is the deployment, which makes the re-embedding migration problem part of your release process rather than a one-off.

What it cannot do

• Cross features. “This user has bought from this seller before” is a fact about the pair, and a dot product of two independently computed vectors cannot represent it. That is not a training failure, it is the architecture. Cross features belong in the ranking stage, which is one of the main reasons the ranking stage exists.
• Hard constraints. Stock, region, licensing, permissions. A dot product produces a score, not a guarantee, and a model trained to demote out-of-stock items will still occasionally rank one first. Constraints go in the index as filters or in a post-filter, never in the loss.
• Sequence, unless you build it in. A user tower reading a bag of past interactions loses the order, and order carries real signal — the item bought five minutes ago means something the item bought last year does not. Feeding a sequence encoder is standard now, and it is a change to the user tower, not to the architecture.
• Explaining itself. The output is a dot product in a learned space with no interpretable axes. If the product needs a reason string, it has to come from somewhere else — the nearest interacted item, a shared attribute — and that reason will sometimes not be why the model actually scored it highly.