Why computers can't read

01 · Concept

Concept

Type the word café into a chat box and send it to Qwen3.8-27B. A perfectly ordinary thing to do, and it hides a problem that has broken production systems. There are two standard ways to represent that word in Unicode. One uses four code points, ending in a single precomposed character for the accented e. The other uses five, ending in a plain e followed by a separate combining acute accent. On screen they are indistinguishable. Serialized to UTF-8 the first occupies five bytes and the second occupies six, because the accented character costs two bytes in the first form and the bare e plus combining mark costs three in the second.

The model never sees your screen. Its public tokenizer receives a Unicode string and, for this pinned release, normalizes it to NFC before byte-level processing. The two spellings of café therefore converge to the same normalized string and the same token ids. Other distinctions—such as an ordinary space versus a non-breaking space—can survive normalization and produce different sequences. The lesson is not that every visual twin differs, but that the tokenizer’s declared normalization policy decides which distinctions reach the model.

So follow the whole path, naming what each stage promises and what it does not. The first boundary is encoding. Unicode assigns code points to a vast repertoire of text elements, and UTF-8 defines how each code point becomes one to four bytes. A byte value on its own is meaningless without knowing the encoding that produced it. Even the notion of a visible character is slippery: an emoji with a skin-tone modifier and a zero-width joiner can occupy several code points while drawing as one glyph, and the café example shows that even a Latin word with one accent has more than one legal spelling in bytes.

Normalization can force canonically equivalent forms into agreement, and Qwen3.8-27B’s pinned tokenizer explicitly applies NFC. Identifiers, passwords, and source code often must preserve distinctions that natural-language text is happy to lose. Whitespace carries the same hazard: an ordinary space, a non-breaking space, and a narrow no-break space look nearly identical and are entirely different bytes. Every one of these choices is made before any model is involved.

The second boundary is tokenization. A tokenizer maps normalized Unicode text through UTF-8 and byte-level BPE to vocabulary entries and integer ids. Qwen3.8-27B’s artifacts expose 248,077 mapped ids, while those ids index tensors with 248,320 rows; the remaining 243 rows have no tokenizer mapping in the published artifacts. The ids themselves are labels, not quantities. Token 900 is not larger, later, or more important than token 12. The tokenizer for this model is byte-level, which has one enormously useful consequence: because its byte-level alphabet can represent every UTF-8 byte produced from a valid normalized Unicode string, the tokenizer needs no unknown token for such text. That statement does not make the public text API an arbitrary-binary interface. Lesson 1.2 examines what that vocabulary contains and lesson 1.3 shows how it was built.

Run the round trip in your head. After NFC normalization, either spelling of café becomes the same UTF-8 byte sequence and then the same short list of ids. Each id selects a row of the embedding table, producing a 5120-dimensional vector as lesson 0.4 described. Those vectors pass through 64 layers, each rewriting the state. The final layer emits 248,320 scores for the next position, a token is chosen, and the decoder maps chosen ids back to bytes and then to text you can read. For ids produced by encoding this normalized input, decoding reconstructs that normalized text, including distinctions that normalization retained, but it cannot recover a canonically equivalent code-point spelling that NFC replaced. Arbitrary ids sampled from the model have no corresponding encoded input and therefore carry no NFC guarantee; malformed byte fragments may also decode with replacement handling.

An embedding table maps ids to vectors, and at initialization those vectors contain no definitions. Training reshapes them, along with everything downstream, so that the representations become useful for prediction. Notice that the same row serves every occurrence of a token. The word bat in a cave and bat in baseball begin at one identical row; it is the surrounding tokens, mixed in by the layers above, that pull the two states apart. Meaning, insofar as the word applies at all, is a property of a contextual state rather than of a vocabulary entry.

Syntax and semantics are therefore delivered by neither Unicode nor the tokenizer. They emerge, imperfectly, from patterns in the training data, from architectural bias, and from the training objective. A model can learn that an opening quotation mark predicts a later closing one, that a subject constrains a verb, or that certain names co-occur with certain professions. Those regularities can support genuinely useful behavior without amounting to an explicit grammar or a checked knowledge base.

The layered view explains a catalogue of real failures. Corrupted bytes break decoding before the model is reached. Visually confusable characters can hide a substituted identifier in what looks like ordinary text. An unlucky tokenizer can shatter one language into many more positions than another, which lesson 1.4 turns into money. Rare spellings receive weakly trained representations. And a fluent model can still attach the wrong real-world referent, because its objective rewarded probable continuation rather than grounded verification.

Computers process text with extraordinary sophistication, and every stage of that processing has a contract narrow enough to write on one line. UTF-8 maps normalized symbols to bytes. Tokenizers map that representation to ids. Embeddings map ids to vectors. Layers transform vectors. Decoding maps scores back to tokens. Keeping those six boundaries distinct is what stops the vague verb reading from concealing where information entered, changed, or was quietly lost.