What is a language model, really?

01 · Concept

Concept

You type The capital of France is into a chat box and a word comes back: Paris. Before any theory, ask a blunt engineering question about that word. Where, physically, did it come from? Two answers feel obvious and both are wrong. It was not looked up in a table of questions and answers, because nobody wrote that table and the same system answers questions nobody anticipated. It was not decided by a small person living in the datacenter. What actually happened is narrower, stranger, and completely inspectable, and this course spends the next hundred-plus lessons opening it up.

Throughout, we will not talk about language models in general when we can talk about one. Our specimen is Qwen3.8-27B, released in August 2026 under the Apache 2.0 licence, with published weights anyone can download and open. Choosing a real model means every claim has a place to be checked, and it means the numbers are specific rather than round. Here is the first and most important one. Qwen3.8-27B is a function that takes a context and returns 248,320 next-position logits. Its tokenizer maps 248,077 ids, and a serving runtime turns the relevant logits into a probability distribution over selectable next tokens. That is the language-model contract instantiated, with the tensor shape kept distinct from the tokenizer mapping.

Follow the request through, one step at a time. First the text becomes tokens: not letters and not quite words, but units from a fixed vocabulary, which lesson 1.2 unpacks in detail. Those tokens become integer identifiers. The identifiers become vectors, and those vectors pass through the network. At the end, one final matrix produces 248,320 raw scores, called logits, for the position we are predicting. A function called softmax converts those scores into probabilities that are non-negative and sum to exactly one. That is the output. Perhaps the token for Paris receives most of the mass, a punctuation token receives a little, and other mapped candidates split what is left. The published artifacts do not specify how every runtime handles the 243 output rows with no tokenizer mapping. Then, and only then, a separate rule picks one.

That last sentence hides the classic wrong turn, so let us make it explicit. Beginners routinely say the model outputs a word. It does not. It outputs a quarter of a million numbers, and the choice among them is made afterwards by a decoding rule you control. This is not pedantry: it is why the same model, given the same prompt twice, can answer differently. Qwen3.8-27B ships with two official sampling presets, one at temperature 1.0 for its thinking mode and one at temperature 0.7 for its instruct mode, and swapping between them changes the output without changing a single parameter. Lesson 0.3 explains what temperature does to a distribution, and lesson 7.4 makes the presets concrete.

One framing choice governs the whole course, and it is easier to state now than to discover later. Almost every textbook, diagram, and tutorial teaches the transformer as a stack of identical blocks: attention, then a feed-forward network, repeated N times. That uniform block is the pedagogical baseline, and it is genuinely the right thing to learn first, because everything else is described as a departure from it. Qwen3.8-27B is the shipped instance, and shipped instances modify the baseline for reasons that are always about cost, memory, or hardware. Its 64 layers are not identical; only one layer in four is the classic attention block you will study in track 4. The Signal Observatory on the explore page animates the baseline, not the instance. When the two disagree, the Observatory is showing you the idea and the model is showing you the engineering, and each lesson will say which one it is talking about.

Training and inference are different phases and it is worth keeping them apart from the very beginning. During training, the parameters were adjusted so that text actually observed in the training data received higher probability than text that did not. During inference, which is everything you experience as a user, those parameters are frozen. Nothing you type teaches the model anything. The apparent learning inside a conversation is entirely an effect of the growing context being fed back in, which is why the size of that context, 262,144 tokens natively for this model, turns out to be an architectural headline rather than a footnote.

Because the knowledge lives distributed across weights rather than in retrievable documents, the question which page did this answer come from? usually has no answer. Verbatim memorization does happen, especially for distinctive or heavily repeated passages, and it matters for privacy and for honest evaluation. But a useful model also continues combinations of ideas that never appeared together anywhere, and no index of sources could produce that.

Whether such a system understands is a question this course will not settle, and you should be suspicious of anyone who settles it quickly in either direction. What we can say precisely is that it builds context-sensitive internal representations, that those representations support a startling range of tasks, and that its failures have a characteristic shape unlike a confident human speaker’s. Holding both halves at once is a better foundation than mysticism or dismissal.

So: a learned probability engine over token sequences, with 248,077 mapped token ids, 248,320 output rows, and a context of a quarter-million positions. Conversation, translation, summarization, and code generation are behaviors assembled out of that one operation, repeated.