Tokenization III: SentencePiece, byte-level, and why token counts bite

1

Concept

“How many tokens is this text?” has no answer until a tokenizer is named. A token is not a stable linguistic unit; it is an entry in one model’s fixed vocabulary under one normalization and encoding procedure. The same paragraph can occupy different sequence lengths in two models, even when both use subwords and display identical decoded text.

Many pipelines first split on whitespace or punctuation and then learn subwords inside those pieces. That can bake language-specific assumptions into preprocessing. SentencePiece was designed to train from raw sentences, representing whitespace with an explicit symbol and treating tokenization and detokenization as a self-contained process. It supports model families such as BPE and Unigram; SentencePiece is a toolkit and representation approach, not one unique segmentation algorithm.

The Unigram model starts with a candidate vocabulary and assigns probabilities to pieces. A string may have several possible segmentations. Training iteratively removes pieces while seeking a vocabulary that gives the corpus high likelihood, and encoding can choose a best segmentation or sample alternatives for regularization. This contrasts with the ordered greedy merges commonly associated with BPE, though real implementations add their own details.

Byte-level approaches solve coverage differently. Text is encoded to bytes, and the base vocabulary can represent every byte value. Frequent byte sequences are merged into larger tokens. Unseen scripts, emoji, corrupted text, or arbitrary data never require a generic unknown character, because they can fall back to bytes. Coverage does not imply efficiency: a familiar English fragment might be one token while an uncommon grapheme expands into several.

Normalization changes counts before subword segmentation. Unicode-equivalent strings can differ in code points. Lowercasing, compatibility normalization, whitespace cleanup, or accent handling may merge distinctions that a task needs. Special tokens then add another layer: beginning-of-sequence, end-of-sequence, role separators, tool delimiters, or chat-template markers may consume context even though they are absent from visible user text.

This is why common conversion rules are estimates, not contracts. A character-to-token heuristic derived from English prose may fail for Portuguese inflections, CJK text, code, tables, long numbers, URLs, or emoji. Even two English phrasings with equal character counts can segment differently because one contains frequent vocabulary pieces. The only reliable count is produced by the exact tokenizer and template used by the deployed model version.

Token counts have direct engineering consequences. Context limits apply to input plus generated output and often to hidden formatting added by an API. More positions increase prefill work; generated tokens add sequential decode steps. Billing schemes commonly count tokens, but prices and definitions are provider-specific and time-varying, so a course should not hardcode a universal rate. Measure the request assembled on the wire, not only the text box.

Tokenizer comparisons should use representative corpora by language and domain. Report distributions rather than one anecdote: median and tail sequence lengths, unknown or byte-fallback frequency, and fragmentation of important terms. Inspect reversibility and special-token safety. The practical conclusion is firm: tokens are model-specific packaging. SentencePiece, BPE, Unigram, and byte-level fallbacks are design choices that trade vocabulary size, coverage, sequence length, and inductive bias; none supplies a language-neutral natural unit.