How Cross-Lingual Embedding Alignment Works

Cross-lingual retrieval is not a property a model has or lacks. It is a geometric arrangement somebody trained into the model on purpose, using data, and it holds exactly as far as that data reached.

What alignment has to achieve

An embedding model maps text to a vector, and retrieval ranks by similarity between vectors. For a Spanish query to retrieve a Japanese document about the same thing, the Spanish sentence and the Japanese sentence have to land near each other — nearer than the Spanish sentence lands to Spanish text about something else.

That second clause is the hard part and it is usually left out. It is not enough that translations be close in absolute terms. They have to be closer than the competition, and the competition includes every same-language document in the index. Language identity is a strong, easily learned feature, so left to itself the model uses it, and same-language distractors win. That is a distinct problem from alignment and it has its own page: why embeddings cluster by language first. Alignment is the force pulling the other way.

The original trick: one rotation

The idea started at the word level. Mikolov, Le and Sutskever proposed in 2013 that word vector spaces trained separately on two languages have similar internal geometry — the relation between king and queen looks like the relation between rey and reina — so a single linear map should be able to take one space onto the other. You fit that map on a seed dictionary of a few thousand translation pairs and apply it to everything else. The paper is “Exploiting Similarities among Languages for Machine Translation”, published in 2013.

Later work constrained that map to a rotation, solved in closed form by orthogonal Procrustes, which preserves distances within each space rather than distorting one to fit the other. Facebook AI Research pushed this further in 2018 with “Word Translation Without Parallel Data”, which learned the mapping adversarially from no dictionary at all and refined it with Procrustes.

Notice the assumption that whole line of work rests on: that the two spaces are approximately isomorphic — same shape, different orientation. Everything that follows in this page is about what happens when that is not true.

How sentence encoders do it now

Current multilingual sentence encoders do not fit a mapping after the fact. They train one shared space directly, in one of two ways.

• Contrastive training on parallel sentences. Take translation pairs, pull each pair together and push everything else in the batch apart. The objective explicitly penalises the model for keeping the languages separate, because keeping them separate means the true pair is not the nearest neighbour.
• Distillation from a strong monolingual teacher. Train a multilingual student to output, for a sentence in any language, the vector the English teacher produces for its English translation. Reimers and Gurevych described this in 2020 in “Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation”. It has the practical advantage that you only need parallel data, not labelled task data, and it inherits whatever quality the English model had.

Both routes have the same dependency: parallel sentences. The alignment is as good as the parallel corpus for that language pair and as representative as that corpus’s domain. Parallel data is abundant for European language pairs, dominated by parliamentary, news and religious text, and thin to nonexistent for most pairs that do not involve English. Pairs that do not include English are often aligned only transitively, through English, which compounds error.

Where it fails: distance and hubness

Two distinct failure modes show up when the languages are typologically distant.

The isomorphism assumption stops holding

The rotation story requires the two spaces to have comparable structure. Two languages that differ in morphology, word order and lexical granularity do not carve meaning at the same joints: one has a single word where the other has a phrase, one marks a distinction grammatically that the other leaves to context, one packs into a single inflected form what the other spreads over four words. Søgaard, Ruder and Vulić documented in 2018, in “On the Limitations of Unsupervised Bilingual Dictionary Induction”, that unsupervised alignment degrades sharply when languages are typologically distant or the two corpora come from different domains — which is the case that matters, because that is the case you have.

Hubness

In high-dimensional spaces, a small number of points become the nearest neighbour of an unreasonable share of all other points. Radovanović and colleagues characterised this as a general property of high-dimensional data in 2010, and Dinu, Lazaridou and Baroni showed in “Improving Zero-Shot Learning by Mitigating the Hubness Problem” (2015) that it is a specific and severe failure in cross-lingual retrieval: a handful of hub vectors in the target language get returned for everything, so recall collapses for the rest.

The standard mitigation is to stop using raw cosine similarity for the ranking. Cross-domain similarity local scaling, introduced in the same 2018 word-translation work, penalises a candidate by how similar it is to its own local neighbourhood, which demotes hubs directly. It is a few lines of code at query time and needs no retraining. If your cross-lingual results contain the same few documents over and over regardless of query, hubness is what you are looking at.

Checking alignment on your own pairs

Alignment quality is a per-language-pair property, so a single headline benchmark number tells you almost nothing about the pair you care about. The measurement is small enough to do by hand.

# 200 known translation pairs, one language pair, one model.
# Metric: how often the true translation is the nearest neighbour.

src = [embed(s) for s in source_sentences]   # e.g. 200 sentences
tgt = [embed(t) for t in target_sentences]   # their translations, same order

sims = normalize(src) @ normalize(tgt).T     # 200 x 200
ranks = (sims > sims.diagonal()[:, None]).sum(axis=1)  # 0 = correct at rank 1

p_at_1 = (ranks == 0).mean()
mrr    = (1.0 / (ranks + 1)).mean()

Two hundred pairs from your own domain beat two thousand from a parliamentary corpus, because domain shift is one of the two things that breaks alignment. If precision at one is high but live retrieval still returns same-language results, alignment is fine and language clustering is your problem. If precision at one is low, alignment is genuinely weak for that pair and no amount of centering will rescue it — you need a different model, a translation step in the query path, or a per-language index.