word2vec, GloVe, and the analogy trick

01 · Concept

Concept

Around 2013 a demonstration circulated that convinced a great many people that machines had started to understand language. Take the vector for king, subtract the vector for man, add the vector for woman, and search the vocabulary for whatever lies nearest the result. Out comes queen. It is a genuinely striking result and it is worth understanding exactly, because the way it is usually retold omits the one detail that decides how impressive it is.

First the machinery that produced those vectors. Before contextual models, static word embeddings made a concrete claim: distributional patterns can become geometry. Rather than treating each word as an unrelated symbol, these methods learned a dense vector per word from how words occur around one another, and the resulting neighbourhoods supported similarity search, classification, and transfer into downstream systems.

word2vec names a family of predictive objectives and the tricks that made them fast. In continuous bag-of-words, surrounding context predicts a centre word. In skip-gram, the centre predicts nearby context. Computing a full softmax over the vocabulary for every training pair would be ruinous, so practical implementations approximate it with negative sampling or hierarchical softmax. Worth noticing for later: the method maintains two embedding tables, an input one and an output one, with genuinely different roles, and which table you export — or whether you average them — changes your results.

The context window defines what the geometry captures. A narrow window emphasizes syntactic and functional similarity; a wide one drifts toward topical association. Subsampling frequent words and the negative-sampling distribution both change which distinctions receive gradient. Corpus domain moves everything: legal, medical, news, and conversational text produce different neighbourhoods for the same word. There is no such thing as the word2vec embedding without the training configuration attached.

GloVe starts from the other end, with a global word-context co-occurrence matrix. Its central observation is that ratios of co-occurrence probabilities carry the informative signal: whether two probe words relate differently to a shared context word says more than raw counts, which are dominated by frequency. Training fits vectors whose dot products, plus bias terms, approximate a function of the logged counts. It bridges matrix-factorization intuitions and local-context learning, and in practice the two families produce broadly comparable geometry.

Now return to the demonstration, and run it as an experiment rather than a story. You compute the point $v_{king} - v_{man} + v_{woman}$ and search for its nearest neighbour by cosine similarity. Here is the omitted detail: the standard evaluation protocol excludes the three query words from the candidate set. That rule is not cosmetic. Linzen’s 2016 re-evaluation removed the exclusion and found drastic accuracy drops, including zero accuracy for several categories. An input word can remain highly competitive because the offset may move the point only modestly relative to the surrounding vocabulary. Excluding it can therefore change the winner. The important claim is the measured protocol effect, not that one particular query word must always be nearest.

This does not make the finding fake. A consistent offset across many pairs is a real and useful geometric property, and one linear operation transferring a relation across pairs is genuinely surprising given that nobody supervised it. What it does mean is that the demonstration measures a tendency under a specific search protocol, not the execution of a symbolic rule. The corrected claim reads: under cosine similarity, with query words excluded, the analogy target is frequently the nearest neighbour for certain relation types in certain corpora. Longer, duller, and defensible in the sense lesson 0.8 demanded.

The sensitivity does not stop there. Results shift with vector normalization, with the choice of the nearest-neighbour rule, with vocabulary frequency thresholds, and with which relation types the benchmark happens to contain. Some relations form clusters rather than a single reusable direction. Morphology and raw frequency explain part of the performance. A top-one miss may still place a sensible answer second. Hand-picked examples systematically overstate reliability.

Modern transformers use contextual states rather than one fixed vector per word, but the interpretive discipline transfers unchanged. Linear probes and vector operations can expose real regularities; what they mean depends on the layer, the metric, the corpus, and the task. For a reproducible comparison, publish the corpus snapshot, the window definition, the vocabulary threshold, the negative-sampling policy, which table was exported, the similarity metric, and the exact analogy-scoring rule including whether query words were excluded. Re-running one famous equation without that contract is a demonstration, and it should be labelled as one.