What actually happens when you hit send

01 · Concept

Concept

Type a question for Qwen3.8-27B, hit send, and about a second later words begin streaming back. Between those two moments a specific, inspectable pipeline runs — and because this course fixes one specimen model, we can trace it concretely. The same weights run in two typical homes: a local vLLM process serving Qwen/Qwen3.8-27B on your own GPU, or Cloudflare Workers AI hosting it as @cf/qwen/qwen3.8-27b. The operator changes; the lifecycle does not.

First, the edge. A gateway authenticates the caller, checks quota, validates the request schema, and assigns a request ID. Limits on bytes, images, tools, and maximum generation are enforced before any expensive work. Rate limiting belongs here, not inside a prompt, and user text is untrusted data from the first byte.

Next, prompt construction. The application combines system instructions, history, and your new message using the checkpoint’s chat template. Qwen3.8-27B’s template includes the role boundaries and thinking-mode markup learned during post-training; a request can switch reasoning off by passing enable_thinking set to false. Two templates that render identically to a human can produce different token sequences and different behavior, so the template is part of the model contract, not decoration.

The tokenizer then maps the assembled text into integer ids drawn from a 248,320-entry vocabulary (padded for parallelism; lesson 1.2 covers why). The model never sees characters. Character count is not token count, so input limits must be checked again after tokenization.

The request now enters the scheduler. Weights occupy roughly fixed memory, but every admitted sequence also claims growing attention-cache space — lesson 7.2 derives exactly how much per token for this model — so a request that fits alone may not fit beside current work. Modern servers build continuous batches, merging requests at different generation positions instead of waiting for a fixed batch to drain.

Then the model finally runs. During prefill, all prompt positions flow through the 64 layers in parallel. In Qwen3.8-27B those layers are not uniform: the 16 full-attention layers project and store keys and values for every prompt position, while the 48 Gated DeltaNet layers fold the prompt into fixed-size recurrent states (lesson 4.16 covers this hybrid layout). The final prompt position produces logits — one score for each of the 248,320 vocabulary entries.

Here is the classic wrong turn: assuming that this one big pass produces the whole answer, so response time should be roughly one forward pass. It is not. Under ordinary autoregressive decoding, the sampler selects one token from the logits, appends it, and runs the model again; temperature 1.0, top-p 0.95, and top-k 20 are three of the model card’s recommended thinking-mode settings. A 500-token answer therefore means roughly 500 target-model decode passes in this baseline. Speculative decoding or the checkpoint’s MTP module can verify several proposals in one target pass, as lesson 7.5 explains, so one pass per emitted token is not a universal runtime invariant.

The decode loop is also where money is counted. On Workers AI, Qwen3.8-27B costs, as of Aug 2026, USD 0.45 per million input tokens and USD 3.20 per million output tokens. An illustrative request with a 300-token prompt and a 500-token reply costs

about a fifth of a cent — with output tokens contributing more than ten times the input cost. Pricing asymmetry mirrors the work asymmetry you just traced: each output token is a full pass through the model.

Streaming happens in parallel with the loop. The server detokenizes incrementally, and token boundaries do not align with characters or valid UTF-8 chunks, so buffering must follow the tokenizer’s decoder. Stop sequences, tool-call parsers, and structured-output constraints may delay or suppress bytes. A slow client must not pin unbounded server memory.

Cancellation travels the other direction. If you stop generation or the connection drops, decode should halt promptly, queued work should disappear, and cache blocks should return to the allocator with the actual token counts recorded for billing. Cleanup leaks become capacity incidents under load.

The durable mental model is a pipeline with a loop inside it. Tokenize once, prefill once, then decode one token per pass until done, detokenizing as you go. The next two lessons zoom into the two halves of that loop: the memory that decode reuses (7.2) and why prefill and decode behave like two different machines (7.3).