Tokenization III: SentencePiece, byte-level, and why token counts bite

01 · Concept

Concept

You ship a support assistant in two languages. Same product, same system prompt, same model, translated faithfully. A month later the finance report shows the Portuguese deployment costing noticeably more per conversation than the English one. Nobody wrote different code. The difference was decided years earlier, by whichever corpus the tokenizer’s merges were counted over.

The question how many tokens is this text has no answer until a tokenizer is named. A token is not a stable linguistic unit; it is an entry in one model’s fixed vocabulary, produced under one normalization policy and one ordered merge list, as lesson 1.3 established. Two models can display identical decoded text and occupy different numbers of positions doing it.

Work the example concretely, with the measurement step made explicit because it is the step people skip. Take an English sentence and its Portuguese translation, run both through the tokenizer that ships with Qwen3.8-27B, and record the counts. Suppose your measurement returns 18 tokens for the English and 26 for the Portuguese — an illustrative pair of numbers, not a published fact, and precisely the kind of thing you must measure rather than quote. The ratio is about 1.44, meaning the Portuguese text consumes forty-four percent more billable input tokens and context positions in this example. It does not prove exactly forty-four percent more prefill compute or latency: those depend on architecture, sequence length, batching, kernels, and hardware, so measure the deployed runtime.

Now convert positions into money. As of August 2026, Cloudflare Workers AI listed Qwen3.8-27B at USD 0.45 per million input tokens and USD 3.20 per million output tokens. One million requests at 18 input tokens each is 18 million tokens:

A difference of USD 3.60 per million requests. Per request it is invisible, three ten-thousandths of a cent, and that is exactly why it never gets caught in review. At volume it is a structural tax levied on one of your two user populations, and it grows with every system prompt, every retrieved document, and every conversation turn that gets replayed into the context.

Here is the wrong turn, and it is nearly universal. Somebody estimates tokens as characters divided by four, or words divided by 0.75. Those ratios were calibrated on English prose and they collapse elsewhere. Portuguese inflection and accented characters segment differently. Chinese and Japanese have no spaces to anchor pieces. Source code, long numbers, URLs, tables, and emoji all fragment in ways no prose-derived constant anticipates. Worse, the rule fails asymmetrically, so it under-counts precisely the languages that are already paying more, which turns a budgeting error into a fairness problem. The correction is not a better constant. It is to stop estimating: the tokenizer is a few lines away and returns the truth in milliseconds.

The tokenizer families differ in how they reach that truth. Many pipelines first split on whitespace or punctuation and then learn subwords inside the pieces, which quietly bakes language-specific assumptions into preprocessing. SentencePiece was designed to train directly from raw sentences, representing whitespace as an explicit symbol so that tokenization and detokenization form a closed, reversible process. It is a toolkit rather than a single algorithm, supporting BPE-style merges as well as the Unigram model, which starts from a candidate vocabulary, assigns probabilities to pieces, and iteratively prunes toward a vocabulary that gives the corpus high likelihood. Unigram admits several segmentations of a string and can sample among them, which BPE’s ordered merges do not.

Byte-level approaches, which is the family Qwen3.8-27B belongs to, solve coverage from below. Qwen’s ByteLevel pre-tokenizer provides byte-derived units for every UTF-8 byte of valid Unicode text after NFC normalization, so unseen scripts need not require an unknown symbol. Its public tokenizer is still a text interface, not an arbitrary-binary decoder, and the published BPE model sets byte fallback to false. Efficiency is not implied by it: a familiar English fragment may be one token while an uncommon grapheme expands into several byte pieces, and that asymmetry is the mechanism behind the invoice above.

Normalization moves counts before any of this happens. Canonically equivalent Unicode strings can differ in code points, while Qwen’s NFC normalizer collapses the two café spellings examined in lesson 1.1 before BPE. Lowercasing, compatibility normalization, whitespace cleanup, and accent handling each merge distinctions that some task somewhere needs. Then special tokens add a layer that is entirely invisible in the text box: beginning and end markers, role separators, tool-call delimiters, and chat-template scaffolding all occupy positions and all are billed. For a model with thinking and non-thinking modes, the template itself differs between them, which lesson 6.2 develops.

When comparing tokenizers seriously, use representative corpora per language and domain and report distributions rather than an anecdote: median and tail sequence lengths, the share of byte-derived fragments, and how badly your important domain terms fragment. Check decoding against the tokenizer’s declared normalization contract and that special tokens cannot be forged from user text.

The conclusion is firm and slightly uncomfortable. Tokens are model-specific packaging. SentencePiece, BPE, Unigram, and byte-level tokenization are engineering choices trading vocabulary size against coverage, sequence length, and inductive bias, and none of them delivers a language-neutral natural unit. The consequences land unevenly on real users, and the only defensible practice is to measure the exact request against the exact deployed tokenizer, then price it at a rate you can date.