Training your own tokenizer

01 · Concept

Concept

Open the pinned Qwen tokenizer and config and two related numbers appear. The tokenizer’s model vocabulary plus added tokens use IDs from 0 through 248,076, while the embedding and output matrices allocate 248,320 rows. That leaves 243 rows the pinned tokenizer never emits. The artifacts establish those counts; they do not publish why the team chose the larger allocation. This distinction is useful because tokenizer design is linguistic, while matrix alignment is an engineering constraint, and evidence should not turn a plausible explanation into history.

The linguistic half comes first. A language model never receives raw text; the tokenizer maps text to integer IDs and the embedding table maps those IDs to 5120-dimensional vectors. Training that tokenizer means choosing a representative sample from the governed corpus of lesson 5.2, with the mixture caps already applied, and then learning a fixed-size inventory of subword pieces from it. Byte-level schemes start from bytes so any input is representable; BPE repeatedly merges the most frequent adjacent pair, producing an ordered merge list; unigram starts from many candidate pieces and prunes those contributing least to a probabilistic objective. SentencePiece can train on raw sentences and treats whitespace explicitly. All of them are compromises between tiny universal units and an impossible full-word dictionary.

Size is the real decision, and it is a direct trade. A larger vocabulary means fewer tokens per sentence, which means shorter sequences, less attention work, and — this matters more than it sounds — fairer treatment of languages that a small English-centric vocabulary would shred into fragments. Lesson 1.4 measured that effect: the same meaning costs more tokens, and therefore more money at USD 0.45 per million input tokens, in Portuguese than in English under a badly balanced tokenizer.

But rows are not free. In this model the embedding and the output head are untied, which the config states plainly as tie_word_embeddings: false. So every row is paid for twice. Compute the marginal cost:

Ten thousand extra vocabulary entries cost about 205 MB of weights. Scale that to the full table: $248{,}320\times 5120\approx 1.271$ billion parameters in the embedding, another 1.271 billion in the head, roughly 2.54 billion combined — around 9% of a 27-billion-parameter model spent on the mapping between text and vectors, before a single transformer layer exists.

Now inspect the engineering property of the allocated row count without assigning intent. Tensor parallelism can split a vocabulary dimension evenly only when the row count is divisible by the parallel degree. The shipped value factors as:

It divides cleanly by 2, 4, 8, 16, 32, 64, 128, 256, and 512. Any power-of-two tensor-parallel degree through 512 therefore gets an equal slice: at degree 8, each rank owns 31,040 rows; at degree 32, 7,760; at degree 64, 3,880. By contrast, 250,000 divides only through degree 16 among those powers of two. Frameworks can pad incompatible dimensions, so divisibility is a real implementation benefit. The public artifacts do not say that this benefit caused Qwen to select 248,320; that causal account remains an inference.

which divides by 16 and then stops. At tensor-parallel degree 32 you would get 7,812.5 rows per rank — impossible — and the framework would pad it silently or fail loudly at launch. So the tokenizer’s learned inventory is padded up to a highly composite number, and the surplus rows are simply never emitted. They cost a few million parameters and buy freedom in every future parallel layout.

The classic wrong turn follows immediately: treating unused tail rows as safely removable. Trimming the matrix would change checkpoint shapes and may remove useful divisibility, while saving only 243 × 5120 × 2 matrices × 2 bf16 bytes, about 5 MB. The token mapping itself must also remain unchanged. The safe conclusion is contractual: load the matrix and tokenizer shapes the checkpoint publishes; do not invent or delete rows from a trained artifact.

Everything else about tokenizer training follows from the fact that the artifact is permanent. Reserve special tokens — padding, sequence boundaries, chat-role markers, thinking-mode markers, FIM markers — before freezing IDs, because appending later means resizing embeddings and training a cold row, and inserting in the middle corrupts every association after it. Decide normalization deliberately: Unicode can spell visually identical text several ways, and case folding or accent stripping shrinks a vocabulary at the cost of distinctions your product may need. Evaluate on held-out samples, reporting fertility by language and domain rather than one global mean, plus round-trip correctness, adversarial maximum expansion, and encoding throughput. Inspect real examples by hand, because Portuguese morphology, combining accents, emoji sequences, CJK text, right-to-left scripts, indentation, and long digit strings each fail differently.

Then freeze and hash the files, store the training-code version, normalization rules, sample manifest, vocabulary, merges, special-token table, and evaluation report, and test that every data worker produces byte-identical IDs for a golden set. A silent tokenizer-version mismatch will let a run train happily on meaningless input while the loss stays finite and plausible — the most expensive silent failure in this track.