What actually happens when you hit send

1

Concept

Pressing “send” starts a distributed systems workflow, not a single model call. The exact components vary, but a useful lifecycle has seven boundaries: request handling, prompt construction, tokenization, admission and scheduling, model execution, decoding and streaming, then cleanup. Each boundary has its own correctness and latency failures.

At the edge or API gateway, the service authenticates the caller, checks authorization and quota, validates the schema, and assigns a request ID. Limits on bytes, messages, images, tools, and maximum generation should be enforced before expensive work. Rate limiting belongs here, not inside a prompt. User text is untrusted data; it must never become authority over credentials or tenant access.

The application builds the model input. It combines system instructions, conversation history, tool results, and the new message using the checkpoint’s chat template. This template inserts role and boundary tokens learned during post-training. Two templates that render similarly to a person can map to different token sequences and behavior. Context management then truncates or summarizes according to an explicit policy; silently dropping the system message is a correctness bug.

The tokenizer maps text into integer IDs. Input validation must occur both before and after tokenization because character count is not token count. The server may reuse a cached prefix when an identical leading token sequence was processed earlier. Otherwise the request enters a queue with its prompt length, output budget, priority, and cache requirements.

The scheduler decides whether to admit it. Model weights occupy mostly fixed memory; the KV cache grows with active sequences. A request that fits alone may not fit beside current work. Modern servers build dynamic batches, combining requests at different generation positions rather than waiting for a fixed batch to finish. Fairness, deadlines, and preemption determine whether a long generation starves short interactive work.

During prefill, the model processes prompt tokens and writes key/value states for every layer. The final prompt position produces logits for the first new token. A decoder applies temperature, truncation, constraints, and a random or deterministic selection rule. The chosen token is appended, and decode runs another forward step using cached states rather than recomputing the prompt.

The server detokenizes output incrementally and sends stream events. Token boundaries do not necessarily match characters, words, or valid UTF-8 chunks, so buffering must follow the tokenizer’s decoder. Stop sequences, tool-call parsers, safety filters, and structured-output constraints may delay or suppress bytes. Backpressure matters: a slow client should not retain unlimited server memory.

Cancellation travels the other direction. If the user stops, the connection closes, or a deadline expires, generation should halt promptly, queued work should disappear, and cache blocks should return to the allocator. Billing and logs need the actual token counts and terminal reason. Cleanup leaks become capacity incidents under load.

Production observability follows request IDs across these stages without storing sensitive prompts by default. Record model and tokenizer versions, template, queue duration, prompt and generated token counts, cache allocation, batch state, first-token and per-token timing, finish reason, and errors. The durable mental model is a pipeline: the neural network is the computational center, but scheduling, boundaries, and resource ownership decide whether it becomes a dependable service.