word2vec and the Idea That Meaning Has Coordinates

word2vec, published by a Google team in 2013, made word vectors cheap enough to train on billions of words and gave the field its most repeated demonstration: king minus man plus woman lands near queen. The demonstration is real. It is also produced with the help of an evaluation rule that almost nobody who repeats it knows about.

The two papers, and what was in each

There are two, and the distinction matters because the famous engineering tricks are in the second one.

The design point of both is what was removed. Earlier neural language models had a hidden layer and a full softmax over a vocabulary of hundreds of thousands, and cost accordingly. word2vec is close to the minimum model that still learns useful vectors, which is why it could be trained on hundreds of billions of words on ordinary hardware, and why it spread within months.

What made it cheap, in numbers

The paper’s title says efficient estimation and it means it. The cost that had been blocking neural word representations was the output softmax: to turn the model’s prediction into a probability distribution over the vocabulary, you must compute a score for every word in the vocabulary, for every training example.

Why the full softmax was the bottleneck, and what negative
sampling replaces it with.

  vocabulary  V = 10⁶ words        (word2vec's published runs used
  embedding   d = 300 dimensions    vocabularies of this order)
  corpus      T = 10⁹ training positions

FULL SOFTMAX
  output-layer work per training position:
      V × d  =  10⁶ × 300  =  3 × 10⁸ multiply-accumulates
  over the corpus:
      T × V × d  =  10⁹ × 3 × 10⁸  =  3 × 10¹⁷ operations

NEGATIVE SAMPLING with k = 5 negatives
  score the true context word plus k sampled false ones:
      (k + 1) × d  =  6 × 300  =  1,800 multiply-accumulates
  over the corpus:
      T × (k+1) × d  =  10⁹ × 1,800  =  1.8 × 10¹² operations

  ratio  ≈  170,000×

ASSUMPTIONS: one training position per corpus token, ignoring the
input-side lookup, which is O(d) either way, and ignoring the
hierarchical softmax alternative the same papers also offer, which
reduces the V term to log₂V and is a smaller but still large win.

That factor is why word2vec could be trained on a corpus of billions of words on a single machine in hours, and why the two papers landed as an engineering event rather than a theoretical one. The contribution is that a representation which had previously required a research cluster now required an afternoon, and the field responded by embedding everything.

The sixty years before it

The idea that a word’s meaning can be characterised by the contexts it appears in is not from 2013. Zellig Harris stated the distributional hypothesis in 1954, and J. R. Firth gave it the line everyone quotes in 1957: you shall know a word by the company it keeps.

• 1990 — Latent semantic analysis. Build a term-document matrix, apply singular value decomposition, keep a few hundred dimensions. This is dense word vectors from co-occurrence counts, twenty-three years before word2vec.
• 2003 — Bengio and colleagues publish a neural probabilistic language model that learns word embeddings jointly with the language model. The embeddings are there; the cost is prohibitive at scale.
• 2008 — Collobert and Weston show that embeddings trained on a large unlabelled corpus transfer to multiple downstream NLP tasks — the pretraining idea, five years before it caught on.

So word2vec’s contribution is not the concept and not the embeddings. It is that it made them cheap, and the field responded to cheapness in the way it always does.

The analogy result, and the rule that makes it work

The 2013 paper introduced an analogy test set of roughly 19,500 questions, split between semantic analogies (Athens is to Greece as Oslo is to ?) and syntactic ones (walking is to walked as swimming is to ?). The method is vector arithmetic: compute b − a + a' and find the nearest word vector by cosine similarity.

Here is the detail that is almost never mentioned:

The standard analogy evaluation, as actually implemented

  question:  man : king  ::  woman : ?
  compute:   v = vec(king) − vec(man) + vec(woman)
  rank all vocabulary vectors by cosine similarity to v
  ── EXCLUDE vec(king), vec(man) and vec(woman) from the ranking ──
  answer:    the highest-ranked remaining word

Without that exclusion line, the nearest vector to v is very often
vec(king) itself — the input word — because subtracting "man" and
adding "woman" moves the point only a short distance in a space
where "king" already dominates the neighbourhood.

The exclusion is in the reference implementation and in the
evaluation scripts everyone used. It is not in the demonstration
as it is usually retold.

Tal Linzen documented this carefully in a 2016 analysis of analogy evaluation, and later work — including a 2020 Computational Linguistics paper whose title makes the point by turning the famous bias example into a tautology — pressed it further. The upshot is not that the analogy result is fake. It is that the result being demonstrated is narrower than stated: the offset between king and queen is similar to the offset between man and woman, and among words that are not the inputs, queen is the best match. That is a real and interesting property of the space. It is not the claim that the arithmetic returns the answer.

Two results that deflated it

Both came from the same group at Bar-Ilan University within two years, and together they substantially changed how the field understood what word2vec had achieved.

1. Skip-gram with negative sampling is implicitly factorising a matrix. Omer Levy and Yoav Goldberg showed in 2014 that the objective word2vec optimises corresponds to factorising a word-context matrix whose cells are pointwise mutual information shifted by a constant. In other words, the neural method and the count-based methods of the 1990s are much closer relatives than the presentation suggested. This did not make word2vec worse; it made the discontinuity smaller.
2. Most of the reported advantage was hyperparameters. Levy, Goldberg and Dagan, in 2015, systematically transferred word2vec’s design choices — context window handling, dynamic window sizing, subsampling, negative-sample distribution — onto traditional count-based methods, and found that much of the gap between the paradigms closed. Comparisons in the literature had been comparing tuned new systems against untuned old ones.

This is a pattern worth recognising because it recurs constantly: a new method arrives with a large reported margin, and a careful reimplementation of the baseline recovers a large part of it. It does not mean the new method is worthless. It means the size of the conceptual leap was overstated, which is a different claim from the one usually made in either direction.

GloVe and fastText: what each one fixed

Two successors arrived within three years, and each addressed a specific limitation rather than proposing a new idea about meaning. Both were still in production use long after word2vec itself had been displaced.

The fastText fix is the more consequential of the two, and for a reason that runs through the rest of this history: a fixed vocabulary of whole words is a bad fit for language. Subword units are how every modern model handles the same problem, and the tokenizers in use now are a direct descendant of that argument rather than of word2vec’s word-type table.

One evaluation detail worth adding to the analogy discussion above: Levy and Goldberg also proposed replacing the additive analogy arithmetic with a multiplicative combination of similarities, which performs measurably better on the same test sets. The fact that the scoring rule for the field’s most famous demonstration was still being improved a year after the demonstration is a reasonable summary of how settled the underlying result was.

What the vectors also encoded

In 2016 Bolukbasi and colleagues showed that embeddings trained on news text produced occupational analogies along gender lines that nobody had asked for, and that the effect was strong enough to be measured systematically rather than cherry-picked. In 2017 Caliskan, Bryson and Narayanan published a result in Science showing that a standard psychological measure of human implicit association could be reproduced from word co-occurrence statistics alone.

The correct reading of that second result is the interesting one. If the associations in a corpus reproduce the associations in human judgement, then the embedding is an accurate measurement of the text — and the text is a record of how people write. The bias is not introduced by the algorithm; it is preserved by it. Every debiasing method proposed since has had to confront the fact that the signal being removed is genuinely present in the data, and this is the same argument that runs through every subsequent discussion of what a model inherits from its training corpus.

Why one vector per word had to end

word2vec assigns one vector per word type. That is the assumption that killed it, and the counterexample is trivial:

  "I sat on the river bank."
  "I deposited the cheque at the bank."

One vector for "bank" must be a compromise between two unrelated
senses. It will sit somewhere between finance and geography and be
a good representation of neither.

Estimates of polysemy in English vary by dictionary, but the words
this affects are the frequent ones — "run", "set", "bank", "light" —
so the problem is not a tail case.

The fix was contextual embeddings: a vector computed for each occurrence of a word given its sentence, rather than looked up in a table. ELMo, in 2018, produced them from a bidirectional LSTM language model. BERT, later the same year, produced them from a transformer and displaced word2vec from serious NLP work within about a year.

Its intellectual legacy is nonetheless enormous, because the operation it made routine — represent a piece of text as a vector, compare by cosine similarity, retrieve nearest neighbours — is exactly what every modern embedding model and every vector database does. The models changed completely. The interface word2vec established did not.