Embeddings: meaning as geometry

1

Concept

A tokenizer emits integers, but their numerical values are arbitrary. Feeding ID 742 as the scalar number 742 would falsely imply ordered distance: token 743 would appear closer than token 20. An embedding matrix solves this by assigning every vocabulary item a learned vector. Looking up a token selects one row, producing continuous features that later layers can combine and transform.

If the vocabulary has size $V$ and embedding width $d$, the matrix has shape $V\times d$. A one-hot vector of length $V$ multiplied by that matrix returns the selected row, though implementations use indexing instead of wasteful multiplication. During training, gradients update rows involved in examples, along with all other weights. No dictionary definition is inserted into a coordinate.

Why does geometry become useful? Tokens occurring in contexts that demand similar predictions receive related learning signals. If “cat” and “dog” often appear in comparable grammatical and topical settings, parameter updates may make their representations useful in similar ways. Distributed coordinates let the network share statistical strength: a feature learned with one item can transfer to another without separate rules for every pair.

Geometry offers several lenses. Euclidean distance measures straight-line separation. Cosine similarity compares directions after normalizing lengths. A dot product mixes direction and magnitude. Nearest neighbors depend on the chosen metric and preprocessing, so a visualization must state both. Projecting hundreds of dimensions into two or three with PCA, t-SNE, or UMAP introduces distortion; apparent clusters in a plot are hypotheses to inspect, not exact maps of the original space.

An embedding’s meaning is relational and objective-dependent. A language-model embedding may emphasize substitutability useful for prediction. A retrieval embedding trained on paired questions and passages may place complementary texts together. A multimodal embedding may align captions with images. Calling all of these “semantic similarity” hides different contracts. Evaluate neighbors and retrieval behavior on the task the embedding was trained or selected to support.

Static token embeddings also face polysemy. The vocabulary row for “bank” is identical at input in “river bank” and “bank loan.” Contextual layers then produce different hidden vectors by mixing surrounding information. Modern language models therefore contain many representation spaces: the initial embedding table, each layer’s residual states, attention projections, and the final states used for prediction. “The embedding” can mean different objects; name the layer and pooling method.

Biases in data can appear in geometry. Associations among occupations, identities, sentiment, or dialect may reflect historical imbalance and harmful stereotypes. Removing one visible direction does not guarantee removal from the model, because information can be nonlinear and distributed. Privacy risks also remain: an embedding is numerical, not automatically anonymous, especially when linked to rare text or a reversible service.

The durable idea is not that meaning literally lives on a perfect map. It is that learned vectors turn discrete symbols into a space where shared transformations and graded relationships become possible. Token IDs choose rows; training shapes geometry; context reshapes states; metrics expose particular relationships. Embeddings are powerful interfaces between symbols and neural computation, but their interpretation must stay tied to data, objective, layer, and evaluation.