Embeddings: meaning as geometry

01 · Concept

Concept

The tokenizer has done its work and handed you a list of integers. Now a problem appears that is easy to miss because the obvious solution is so wrong it looks fine. You could simply feed those integers into the network as numbers. Token 742 goes in as 742. This fails immediately and instructively: it asserts that token 743 is nearer to 742 than token 20 is, that token 248,000 is enormous compared with token 5, and that the arbitrary order in which a merge algorithm happened to number its pieces is a semantic scale. None of that is true. Ids are labels, and labels have no arithmetic.

An embedding matrix fixes this by giving every vocabulary entry its own learned vector. Looking up a token means selecting one row, producing continuous features that later layers can combine, rotate, and mix. If the vocabulary has size $V$ and the hidden width is $d$, the matrix has shape $V \times d$. Multiplying a one-hot vector of length $V$ by that matrix returns the selected row, which is the tidy mathematical description; implementations index directly, because multiplying by a quarter-million zeros is a poor use of a GPU.

Put our specimen’s numbers in and the scale of this boundary becomes visible. Qwen3.8-27B has 248,320 rows and a hidden size of 5120. Work it out in two parts to keep it legible: $248{,}320 \times 5000 = 1{,}241{,}600{,}000$, and $248{,}320 \times 120 = 29{,}798{,}400$. Adding them:

More than a billion parameters, before the model has performed a single layer of computation. That is what it costs simply to have opinions about individual tokens.

Now the wrong turn, which almost everyone takes because almost every tutorial takes it. Many language models tie their embeddings: the same matrix that turns ids into vectors at the input is reused, transposed, to turn the final hidden state into vocabulary logits at the output. It is an elegant saving and it is extremely common, so the natural next step is to write down 1.271 billion and move on. Check the configuration instead, as lesson 0.7 taught. Qwen3.8-27B sets tie_word_embeddings to false. The language-model head is a separate matrix, with the same 248,320 by 5120 shape and therefore roughly another 1.271 billion parameters of its own.

Roughly a tenth of the entire model sits in those two matrices, and an estimate that assumed tying would have been short by more than a billion parameters — about 2.5 gigabytes of weights at two bytes each. This is the kind of error that survives review because the wrong number is plausible, which is precisely why lesson 0.8 insisted on reading the configuration rather than the convention.

Why does the geometry become useful at all? Because tokens appearing in contexts that demand similar predictions receive similar learning signals. If two words repeatedly show up in comparable grammatical and topical positions, the updates that help one tend to help the other, and their representations drift into related regions. Distributed coordinates let the network share statistical strength instead of learning a separate rule for every pair. Nothing inserts a definition into a coordinate; the structure is a residue of prediction.

Geometry offers several lenses and they answer different questions. Euclidean distance measures straight-line separation. Cosine similarity compares direction after dividing out length, which lesson 0.4 showed is what you usually want when magnitudes vary. A raw dot product mixes both. Nearest neighbours depend on which metric you chose and on any preprocessing, so a visualization must state both. Projecting 5120 dimensions down to two with PCA, t-SNE, or UMAP introduces real distortion, and an apparent cluster in such a plot is a hypothesis to test rather than a map of the space.

An embedding’s meaning is also relational and objective-dependent. A language-model embedding emphasizes substitutability useful for next-token prediction. A retrieval embedding trained on question and passage pairs places complementary texts together, which is nearly the opposite relation. A multimodal alignment embedding puts captions near images. Calling all three semantic similarity conceals three different contracts, and this matters concretely for our specimen, whose vision tower projects image patches into the same 5120-dimensional space that text tokens live in, as lesson 4.17 covers.

Static token embeddings face polysemy head-on. The row for bank is bit-identical in river bank and bank loan; only the layers above pull the two states apart. So a model contains many representation spaces, not one: the input table, each layer’s residual state, the attention projections, and the final state that meets the output matrix. When someone says the embedding, ask which layer and which pooling.

The durable idea is not that meaning lives on a perfect map. It is that learned vectors convert discrete symbols into a space where shared transformations and graded relationships become possible at all. Ids choose rows, training shapes the geometry, context reshapes the states, and metrics expose particular relationships. Two and a half billion parameters of Qwen3.8-27B exist to make that translation work in both directions.