Word2Vec and GloVe: The Ancestors of Embeddings

Word2vec is the moment dense representations became practical, and nearly everything about modern embeddings is a descendant of it. It is also the source of the field’s most-repeated demo, which turns out to be less impressive than it looks.

The idea in one sentence

You shall know a word by the company it keeps — the distributional hypothesis, usually credited to Firth in 1957. Words appearing in similar contexts have similar meanings, so if you can build a representation from context, similarity of representation approximates similarity of meaning.

People had been doing this by counting since the 1990s, building huge sparse co-occurrence matrices and factorising them. What Mikolov and colleagues contributed in Efficient Estimation of Word Representations in Vector Space (2013) was making it cheap: a shallow model trained by stochastic gradient descent that ran over billions of words on ordinary hardware and produced 300-dimensional vectors that were immediately, obviously useful. Not a better idea — a tractable one.

How skip-gram actually trains

The mechanism is worth knowing because it explains the properties. Skip-gram takes a centre word and tries to predict the words around it within a small window. Concretely, in the cat sat on the mat with a window of two, the pair (sat, cat) is a positive example, as are (sat, the) and (sat, on).

Predicting over the whole vocabulary at every step is far too expensive, so the second 2013 paper — Distributed Representations of Words and Phrases and their Compositionality (NeurIPS 2013) — introduced negative sampling: instead of a softmax over 100,000 words, draw a handful of random words as negatives and train a binary classifier to separate the real context word from them. That change is what made the method fast, and it is the direct ancestor of the contrastive objectives used to train modern sentence embedding models.

The same paper introduced subsampling of frequent words, which does automatically what a stop word list does by hand and does it better: very frequent words are dropped probabilistically in proportion to their frequency, so the stops dominating training without anyone maintaining a list.

GloVe, and the counting alternative

GloVe (Pennington, Socher and Manning, EMNLP 2014) argued that you do not need to predict at all: build the global word-word co-occurrence matrix, then fit vectors so that a dot product approximates the log co-occurrence count. It is a return to counting, with a weighted least-squares objective that stops rare pairs dominating.

The two families produce broadly interchangeable vectors, and the honest summary is that the difference between them is smaller than the difference made by corpus size and window width. The more consequential successor is fastText (Bojanowski, Grave, Joulin and Mikolov, TACL 2017), which represents a word as the sum of its character n-grams. That single change means a word never seen in training still gets a vector from its subwords, which fixes the out-of-vocabulary problem that made the original models awkward for morphologically rich languages, user-generated text and product catalogues.

The analogy demo, and what is wrong with it

The famous result is that vec(king) − vec(man) + vec(woman) lands near vec(queen), and it is genuinely striking that a model trained only to predict neighbours arranges its space so that a direction corresponds to a relation.

The demo is also weaker than it is usually presented. Nissim, van Noord and van der Goot, Fair is Better than Sensational: Man is to Doctor as Woman is to Doctor (Computational Linguistics, 2020), documented a detail that matters a great deal: the standard analogy evaluation excludes the three input words from the candidate answers. Without that exclusion, the nearest vector to the arithmetic result is frequently one of the inputs itself — most often king. The impressive answer partly depends on having removed the unimpressive one, and the same paper shows how this has distorted claims about bias derived from analogy tests.

None of this makes the vectors bad. It is a caution about the evaluation, and it generalises: a demo constructed to be legible is not a measurement, and this cluster tries to keep the two apart.

When static vectors are still right

The decisive limitation is in the name: one vector per word type, forever. bank gets a single vector averaging the financial and the riverside senses, and there is no mechanism to disambiguate, because the model never sees the sentence. Contextual models solved exactly this, and for anything requiring sentence meaning they are not in competition.

But there are cases where a static vector space is not a compromise:

• You need a vocabulary, not sentences. Building a synonym or query-expansion list for a lexical index means asking which words behave alike in your corpus. That is a type-level question and a type-level model answers it directly.
• Your domain vocabulary is not in any pre-trained model. Part numbers, drug names, internal jargon. Training fastText on a few hundred megabytes of your own text takes minutes on a laptop and produces vectors that actually cover your terms.
• The items are not text. The skip-gram objective applies to any sequence of discrete items — products viewed in a session, songs in a playlist, pages in a path. The item2vec family is this observation, and it is still the cheapest way to get useful item embeddings from behavioural logs.
• Lookup must be free and offline. A vector table is a dictionary. No GPU, no service, no per-call cost, microsecond lookup, and it never changes underneath you.

For sentence and document meaning, use modern embeddings — and note that they inherited the objective, the negative sampling and the cosine-similarity convention directly from this work.