Interconnects and clusters

01 · Concept

Concept

Lesson 9.4 left you with a sizing problem. Qwen3.8-27B in bf16 is 54 GB of weights, and lesson 7.2 adds up to 16 GiB of KV cache for one full-length sequence, so a single 80 GB accelerator serves the model with very little room for users. The obvious fix is more GPUs. The question this lesson answers is what that actually costs, because the wire between the chips has a price that does not appear on any spec sheet you were shopping from.

Lesson 5.8 established the mechanism. Under tensor parallelism each weight matrix is split across ranks — attention and DeltaNet heads distributed one way, the FFN’s up and down projections another — so every rank computes a partial result over its slice, and the slices must be summed before the residual stream continues. In the standard arrangement each layer has two such reconciliation points: one after the output projection that follows the sequence mixer, and one after the FFN down projection. Every reconciliation is an all-reduce over a single hidden vector for the token being decoded.

That is the whole worked example, and it is small enough to do in your head. Hidden size 5120, bf16:

A ring all-reduce puts $\tfrac{2(N-1)}{N}$ times the payload on each rank’s wire, so two GPUs move 1.25 MiB per token and four move about 1.875 MiB. At an interactive 50 tokens per second that is under 100 MiB per second per rank — a rounding error next to NVLink’s vendor-reported hundreds of gigabytes per second, and comfortable even for PCIe.

So the bytes are not the problem, and this is exactly where the classic wrong turn happens. An engineer computes that payload, sees it is trivial, concludes the interconnect is irrelevant, and buys four GPUs on a PCIe-only board expecting four times the throughput of one. The error is measuring the wrong quantity: 128 all-reduces per token are not one transfer of 1.25 MiB, they are 128 serial blocking round trips, and what dominates a small collective is latency, not bandwidth.

Put stated assumptions on the latency — these are assumptions to make the shape visible, not vendor figures — and redo the arithmetic. Say a fast intra-node fabric costs about 10 microseconds per small all-reduce and a slow one about 50. For a stated checkpoint-size scenario, divide the per-GPU file share by bandwidth; this is a planning heuristic, not measured active traffic:

Communication adds a fixed 128 times the per-collective latency: 1.28 ms on the fast fabric, 6.4 ms on the slow one. Adding those stated heuristics and latency assumptions:

Read those three lines carefully. Doubling from two to four GPUs on the fast fabric returned 1.76 times, not 2 — the checkpoint-size term halved while the assumed communication stayed fixed. And four GPUs on the slow fabric are slower than two on the fast one. The interconnect, not the accelerator count, decided the outcome. This is why NVLink exists as a distinct product from PCIe, why tensor parallelism is kept inside a node, and why cross-node fabrics such as InfiniBand or RoCE are reserved for the parallelism forms that reconcile rarely — data parallelism, which synchronizes once per training step, or pipeline parallelism over the model’s 16 super-blocks, which passes an activation between stages rather than reconciling every layer.

Prefill behaves oppositely and it is worth seeing why. A 4,096-token prompt runs all positions at once, so the same 128 collectives carry 4,096 hidden vectors each:

Now the bytes genuinely matter and bandwidth governs, which is the arithmetic-intensity split of lesson 7.3 reappearing on the network instead of on memory. Prefill is bandwidth-bound over the fabric; decode is latency-bound over it.

The durable rule for cluster design is that you should place the parallelism that reconciles most often on the fastest wire you own, and never split a model across a boundary its own dimensions do not divide. For this specimen that means tensor parallelism inside one node, at a degree of 2 or 4, on NVLink; everything else across the network. Lesson 9.10 takes those configurations and finally prices them.