Tokenization I: characters, words, and subwords

01 · Concept

Concept

Take the word unbelievability and ask a designer’s question: how many units should the model spend on it? Three answers have all been tried in production, and their consequences are worth working out before any terminology arrives.

Spell it one character at a time and it costs fifteen units. The vocabulary needed is tiny, a few hundred entries covers most Latin-script text, and no word is ever unrepresentable. But every sentence now stretches across five or six times as many positions, and as track 4 will show, the work a transformer does grows with positions rather than with characters. You have made the model cheap to store and expensive to run.

Store it as one whole word and it costs one unit — provided somebody put it in the vocabulary. If they did not, the model receives an unknown symbol and the distinction between unbelievability and incomprehensibility is destroyed at the door. Names, inflections, domain jargon, typing errors, and words invented last Tuesday all land in the same hole. No finite word list closes over open-ended text, and enlarging it has its own price we will come to.

Split it as un, believ, abil, ity and it costs four units, with pieces that are reusable across thousands of other words. That is subword tokenization, and it is what essentially every modern model does, Qwen3.8-27B included. Frequent strings stay whole; rare ones are composed. The pieces are learned from a corpus by an algorithm, and lesson 1.3 traces one of those algorithms merge by merge.

Qwen’s variant goes one step further down. Its tokenizer applies NFC normalization and a ByteLevel pre-tokenizer before BPE. The byte-derived base can spell every UTF-8 byte produced by a valid Unicode string, so unfamiliar scripts and emoji need not collapse to an unknown token. This is not a fallback path—the published BPE configuration says byte_fallback: false and declares no unknown token—and it does not make the public text API accept arbitrary binary. Coverage of normalized Unicode text is broad; efficiency is not, because unusual strings may expand into long chains of byte-derived pieces.

Notice that none of these splits is a fact about language. Even word is ambiguous. Is don’t one word or two? Does the punctuation attach? Chinese does not put spaces between words at all. German cheerfully welds four concepts into one compound. A tokenizer must nevertheless return a deterministic answer for every valid Unicode string, including strings its designers never imagined, so the boundaries it draws are statistical rather than linguistic. A split may look absurd to a person and still be the right engineering choice.

Now the number in the configuration file, which is where this lesson earns its place. Qwen3.8-27B declares 248,320 vocabulary rows. The obvious wrong turn is to read that as the model knows 248,320 words, and the more sophisticated wrong turn — the one that catches people who already know better — is to read it as the tokenizer can emit 248,320 distinct tokens. Neither is guaranteed by that field. What vocab_size governs is a tensor shape: the number of rows in the embedding matrix, and the number of columns in the output projection that produces logits.

The tokenizer artifacts make the gap measurable. The BPE model contains 248,044 entries, and 33 added tokens occupy ids through 248,076, for 248,077 mapped ids in total. The model configuration sets $\text{vocab\_size}=248{,}320$, so the embedding and output tensors reserve 243 additional rows. The shape is divisible by 512—$248{,}320 = 512 \times 485$—which is convenient for sharding, but the published artifacts do not state why this exact size was chosen, what training signal the unmapped rows received, or how every server handles their logits. The pinned generation configuration declares no generic suppress_tokens list. Inspect the actual runtime before making a masking claim.

The vocabulary size is not free, and the cost is easy to underestimate. Every row is a vector of 5120 numbers, and in this model the input embedding and the output projection are separate matrices rather than a shared one, so each additional row costs storage twice over. Lesson 1.5 does the full accounting and the total is startling. The design tension is therefore genuine: more vocabulary entries mean shorter sequences and cheaper attention, but a larger table to store, load, and multiply against at every single decoding step. There is no universally correct answer, only a point on a curve chosen for a particular set of languages, hardware, and workloads.

Because the tokenizer is fixed when training begins, it is part of the model’s contract rather than a preprocessing convenience. The embedding matrix has one row per vocabulary entry, and those rows learned their values under one specific mapping from text to ids. Swap the tokenizer afterwards and every id points at a row that learned something else. This is why you cannot mix a model’s weights with another model’s tokenizer, however similar the two look.

When you inspect any tokenizer, four questions settle most of what matters. Can it represent the valid text your interface accepts? How many tensor rows does it have, and how many ids do its tokenizer artifacts map? How many tokens does typical text in your languages require? And are the languages and domains you care about fragmented fairly against the ones you do not? Those questions show why tokenization is not mere preprocessing. It fixes the alphabet available to the model and sets the price of every sequence that model will ever process.