Memory bandwidth vs FLOPs

01 · Concept

Concept

Rent a single H100-class GPU, load Qwen3.8-27B in bf16, open one chat window, and time the output. Tokens arrive at a few dozen per second while the card’s specification sheet advertises arithmetic throughput in the petaflop range. Nothing is broken and nothing is misconfigured. The machine is simply being paid in the wrong currency.

Every accelerator sells two things, and they are priced independently. The first is arithmetic: floating-point operations per second, delivered by tensor cores across the streaming multiprocessors. The second is memory traffic: bytes per second moved between high-bandwidth memory and those compute units. An H100 SXM module carries 80 GB of HBM3 at a vendor-reported peak near 3.35 TB/s, and its dense bf16 tensor-core rate is quoted on the order of a thousand teraflops. Treat both as vendor figures as of Aug 2026, measured under vendor conditions and essentially unreachable in a real kernel. Even so, their ratio is the number that matters, because it tells you how much arithmetic the machine wants for every byte it fetches:

Feed this machine fewer than a few hundred operations per byte and the tensor cores stall, waiting. That single ratio, the machine balance point, decides which currency any given kernel spends.

Now the worked example. Lesson 7.3 established that decoding advances one position per sequence per step through every layer, and that at batch 1 the arithmetic intensity of a weight matrix is about one operation per weight byte. The bytes are not abstract. Qwen3.8-27B holds roughly 27 billion parameters in bfloat16, two bytes each, which is about 54 GB of weights, but 54 GB is the full resident artifact, not exact active bytes for a text decode step. Every decoder projection, gated feed-forward matrix, and the untied language-model head run, while the input embedding is a row lookup and the vision tower does not run for text-only input. The full-checkpoint planning quotient is:

Call it about 60 tokens per second. This is not a benchmark and not a measurement anyone here made. It is a conservative checkpoint-size estimate derived from a vendor bandwidth claim, and it ignores attention work, the growing key-value reads of lesson 7.2, DeltaNet state updates, kernel launch overhead, and every inefficiency that stands between peak and attained bandwidth. A measured rate can land on either side of this checkpoint quotient: inactive checkpoint tensors lower active bytes, while cache, state, kernels, and unattained peak bandwidth add cost. Its value is order of magnitude; an exact roofline uses measured active bytes per token.

Here is the wrong turn, and it is expensive enough that people make it with purchase orders. Decode feels slow, so the team concludes the GPU is out of arithmetic and specifies a part with more FLOPs. Run the arithmetic on the proposal instead. Under that checkpoint-size heuristic the card performs roughly two operations per parameter for the multiply-accumulate, about 54 billion operations in total, against a peak of a thousand teraflops. That works out to something like three tenths of one percent of the arithmetic the machine can deliver. The tensor cores are idle more than 99 percent of the step. Doubling them doubles the idleness. The correction is to attack bytes, not operations, and every effective decode remedy in this course is exactly that: validated lower-bit weights reduce active bytes, batching amortizes one weight read across many sequences, and speculative decoding verifies several candidate tokens per read.

Batching deserves its own line of arithmetic because it is the cheapest lever and the most misunderstood. Decoding 32 independent sequences together still reads the weights once per step, so the intensity rises from about one operation per byte to about 32. Total throughput rises roughly 32-fold while per-sequence latency barely moves. But 32 is still far below the 300 balance point, so the step remains bandwidth-limited, just usefully so. This is why serving economics and chat latency are different problems with different answers, and why a system tuned for one can look broken measured by the other.

The training side of the ledger runs the other way, which is worth holding in mind alongside lesson 5.13. Pretraining cost is estimated as roughly six operations per parameter per token, a formula that counts FLOPs because pretraining processes enormous token batches and therefore lives far to the arithmetic-rich side of the balance point. The same weights that made training a FLOP-purchasing exercise make single-stream serving a bandwidth-purchasing one. One model, two hardware regimes, and a procurement decision that inverts depending on which phase you are paying for.

The durable model is a two-currency machine. Ask of any kernel how many operations it performs per byte it must move, compare that to the machine’s balance point, and you know before writing a line of optimization which resource you are short of. Lesson 9.2 turns that comparison into a single chart you can place every kernel on, and lesson 9.3 spends the other half of the hardware budget, the capacity rather than the bandwidth, on the key-value cache.