Tokenization I: characters, words, and subwords

1

Concept

Neural networks operate on numbers, so text must cross a boundary before a language model can process it. Tokenization is the rule for splitting text into units and mapping each unit to an integer ID. The model never receives the original sentence directly. It receives something like a list of IDs, and an embedding table converts each ID into a vector.

The split is a design choice, not a natural fact about language. Even “word” is ambiguous. Should don’t be one word or two? Is punctuation attached? Chinese normally has no spaces between every word. German can join several concepts into a long compound. Source code, emoji, URLs, and misspellings create more edge cases. A tokenizer needs a deterministic answer for every possible input, including text its designers never anticipated.

The simplest approach is character tokenization. Give every character an ID, so “cats” becomes four units. The vocabulary remains small, and an unseen word is no problem because its characters are already known. The cost is sequence length. A model must carry information across many more positions, and the useful semantic units are not handed to it. Unicode also complicates the apparent simplicity: what looks like one symbol on screen may contain multiple code points.

At the other extreme, word tokenization assigns one ID to every known word. Common prose becomes compact, and each unit often has an intuitive meaning. But the vocabulary explodes. Names, inflections, domain terms, typing errors, and newly coined words produce out-of-vocabulary cases. Replacing them all with one unknown token destroys precisely the distinctions a model may need. No finite word list can cover open-ended text.

Most modern LLMs therefore use subword tokenization. Frequent strings can remain whole tokens, while rarer strings are assembled from reusable pieces. A tokenizer might represent “walking” as “walk” plus “ing,” but keep “the” as a single unit. It may split an unusual surname into several fragments. The exact pieces are learned or constructed from a text corpus, commonly with families of algorithms such as byte-pair encoding, WordPiece, Unigram, or byte-level variants.

Subwords provide an engineering compromise. The vocabulary can be large enough to compress frequent patterns but small enough to store and train efficiently. Almost any text can be represented through smaller fallback pieces. The compromise is imperfect because token boundaries are statistical, not linguistic. A split may look strange to a person. Two related words may fragment differently. Whitespace may be attached to a following token. Capitalization can change the sequence.

Tokenization happens before the Transformer. The tokenizer is typically fixed during model training, so the model’s embedding matrix has one row per vocabulary item. Changing the tokenizer afterward would change the meaning of the IDs and invalidate those learned rows unless the model were adapted. This makes the tokenizer part of the model contract, not a cosmetic input function.

There are practical consequences. Context windows are measured in tokens, and API usage is commonly metered in tokens. Two texts with the same number of characters can occupy different numbers of positions. A language underrepresented in the tokenizer’s training corpus may be broken into more pieces than English, reducing how much text fits and raising compute per sentence. Code, numbers, and unusual formatting can also tokenize inefficiently.

Token boundaries can influence behavior. Arithmetic becomes harder when numbers split inconsistently. A prompt injection can exploit unusual Unicode or token sequences. A model may learn strong representations for frequent whole tokens and weaker composition for rare fragments. None of this means tokens carry fixed dictionary meanings: their meaning emerges from how the model uses their embeddings in context.

When you inspect a tokenizer, ask four questions: Can it represent arbitrary input? How large is its vocabulary? How many tokens does typical text require? Are important languages and domains fragmented fairly? Those questions reveal why tokenization is not mere preprocessing. It determines the alphabet of thought available to the model and the price of every sequence it processes.