Training your own tokenizer

1

Concept

A language model never receives raw text. Its tokenizer maps text to integer IDs, and the embedding table maps those IDs to vectors. Training a tokenizer therefore sets the model’s alphabet, compression scheme, and several boundary conventions before the first gradient step.

Choose a representative sample from the governed training corpus. Preserve the languages, code, numbers, whitespace, scripts, and domain formats that matter, while capping giant sources so they do not monopolize vocabulary. Tokenizer training is much cheaper than model training, but a biased sample can impose inefficiency for the lifetime of every checkpoint.

Decide normalization deliberately. Unicode can represent visually similar text in multiple ways. Case folding, accent removal, whitespace cleanup, or compatibility normalization may reduce vocabulary but can destroy distinctions. The tokenizer must be reversible enough for the product: decoding token IDs should recover the intended text bytes or documented normalized form. Security-sensitive systems also need to consider confusable characters.

Byte-level schemes begin from bytes, guaranteeing that any input can be represented, then learn common byte sequences. BPE repeatedly merges frequent adjacent units, producing a deterministic merge list. Unigram tokenization begins with many candidate pieces and optimizes a probabilistic vocabulary, pruning pieces that contribute least. SentencePiece can train directly on raw sentences and treats whitespace explicitly. The algorithms differ, but all create subword compromises between tiny universal units and a huge word dictionary.

Reserve special tokens before freezing IDs. Padding, beginning or end markers, unknown tokens when applicable, FIM markers, chat-role markers, and span-corruption sentinels have operational semantics. Their strings, IDs, and allowed contexts must be versioned. Adding a token later requires resizing embeddings and training the new row; inserting it in the middle and shifting IDs corrupts every existing association.

A simple BPE training loop is conceptually runnable: count adjacent pairs over tokenized words, merge the most frequent pair, update the corpus representation, and repeat until the vocabulary budget is reached. Production implementations optimize counting and encode edge rules, but the learned artifact remains a base vocabulary plus ordered merges.

Evaluate on held-out samples. Fertility is the number of tokens per word or character span; lower is not automatically better, but extreme fragmentation consumes context and compute. Report fertility by language and domain, not only one global average. Measure unknown or fallback behavior, round-trip correctness, maximum expansion for adversarial strings, encoding speed, and the treatment of whitespace, numbers, URLs, and source code.

Inspect examples manually. Portuguese morphology, combining accents, emoji sequences, CJK text, right-to-left scripts, indentation, and long numbers expose different failures. A tokenizer trained mostly on English may spend several tokens on common words in another language, effectively charging those users more context and computation for the same amount of meaning.

After training, freeze and hash the tokenizer files. Store training-code version, normalization rules, sample manifest, vocabulary, merge model, special-token table, and evaluation report. Test that every data worker produces identical IDs for a golden set. A silent tokenizer-version mismatch can make a run learn from meaningless input while losses still remain finite.

The tokenizer is not neutral preprocessing. It decides the units the model predicts and the length at which attention operates. Train it with the same care as a public API: representative inputs, explicit semantics, adversarial tests, and immutable versioning.