The roofline model

01 · Concept

Concept

A profiling report lands on your desk. The decode kernel for Qwen3.8-27B achieves three tenths of one percent of the GPU’s peak floating-point rate. The obvious reading is that somebody wrote a terrible kernel and a rewrite will recover two orders of magnitude. The roofline model exists to tell you, in about thirty seconds and without touching the code, that this reading is almost certainly wrong and that the kernel may already be perfect.

Williams, Waterman, and Patterson published the model in 2009 to make exactly this judgement cheap. It compresses a machine into two numbers and a kernel into one. The machine contributes its peak arithmetic rate and its peak memory bandwidth. The kernel contributes its arithmetic intensity, the operations it performs per byte it must move from memory. Attainable performance is then the smaller of what the arithmetic units can deliver and what the memory system can feed:

where $\beta$ is bandwidth in bytes per second and $I$ is intensity in operations per byte. Plotted with intensity on the horizontal axis and performance on the vertical, both on logarithmic scales, this draws a slanted line of slope one that stops abruptly at a horizontal ceiling. The bend is the ridge point, the intensity at which the two limits are equal:

Using the vendor-reported H100-class figures from lesson 9.1, roughly one thousand teraflops of dense bf16 against 3.35 TB/s of HBM3, the ridge point lands near 300 operations per byte, as of Aug 2026 and subject to every hedge those numbers carry. Left of the ridge, the memory system is the constraint. Right of it, arithmetic is. That is the entire model, and it is enough to price most optimization proposals before anyone writes code.

Place Qwen3.8-27B on it. Prefill of an 8,192-token prompt carries the intensity established in lesson 7.3, about 8,192 operations per weight byte, because each weight fetched serves all prompt positions simultaneously. That is roughly 27 times past the ridge point, so prefill rides the flat compute roof. Its attainable performance is $P_{\text{peak}}$, and the honest engineering questions there are about tensor-core occupancy, tiling, and numeric format, because those are what move you along a ceiling you are actually pressed against. Time to first token responds to arithmetic. Buying FLOPs for prefill is a defensible purchase.

Batch-1 decode carries an intensity near one. Its attainable performance is therefore

which against a thousand-teraflop ceiling is about 0.34 percent of peak. So the profiling report describes a kernel sitting exactly on its roofline. It is not inefficient. It is on a different roof. Nothing in the instruction stream can be reordered to fix it, because the arithmetic units were never the scarce resource, and this is the correction that saves teams from months of kernel micro-optimization that recovers single-digit percentages. Batching to 32 sequences moves the point to intensity 32 and allows about 107 teraflops, roughly a thirty-twofold gain, still an order of magnitude short of the ridge, still on the slanted roof, and obtained entirely by changing the workload rather than the kernel.

The hybrid layout puts two distinct dots on the chart for one model, which is what makes Qwen3.8-27B interesting to profile. The 16 full-attention layers read a key-value history that grows at 64 KiB per token, the figure derived in lesson 7.2, while the arithmetic each new query performs against that history grows at the same rate. Their intensity is roughly constant and low, but the absolute bytes per step climb, so as context lengthens these layers consume a rising share of the step time on the bandwidth roof. The 48 Gated DeltaNet layers behave differently. They read and update a fixed recurrent state of roughly 3 MiB per layer, about 144 MiB in total in the reference float32 path, no matter how long the conversation has run. Their dot does not move. Three quarters of the model’s depth therefore contributes a constant, bounded cost to every decode step while a quarter contributes a growing one, which is the architectural bargain restated in roofline terms.

Turning this from a sketch into evidence requires real counters, not estimates. Kernel profilers report achieved bandwidth and achieved arithmetic separately and can place a measured kernel on a measured roofline, including secondary roofs for L2 and shared memory. That distinction matters because the estimated intensity you derive on paper and the intensity a kernel actually exhibits diverge whenever caching, padding, or memory layout differ from the idealization.

The payoff is a habit. Before optimizing, estimate intensity, compare it to the ridge point, and let the chart tell you whether you are shopping for bytes or for operations. Lesson 9.4 applies that habit to specific silicon, where a newer part changes both roofs at once and the capacity underneath them.