Apple silicon and unified memory

01 · Concept

Concept

Lesson 7.12 left a loose end. It sized a 4-bit Qwen3.8-27B build at roughly 17.1 GB and observed that unified-memory machines “punch above their apparent class” — but did not say why, or how far. That is this lesson, and the answer splits cleanly into two questions that beginners routinely fuse into one: can the model load, and how fast will it run.

The loading question is architectural. On a conventional workstation the model file is read into host RAM, then copied across a PCIe boundary into the discrete GPU’s own VRAM, and it is that VRAM — not the machine’s memory — that must hold the resident weights. A machine with 128 GB of RAM and a 24 GB card is, for this purpose, a 24 GB machine. Apple silicon removes the boundary: the CPU, the GPU and the neural engine address one physical pool, and a tensor produced by one is visible to the others without a copy. Apple documents this as the unified memory architecture underlying Metal, and MLX is built directly on the property — arrays live in shared memory, and an operation is dispatched to a device rather than a buffer being shipped to one. So a 17.1 GB artifact on a machine with 64 GB of unified memory is simply resident. Nothing is staged, nothing is duplicated, and the KV cache from lesson 7.2 draws on the same pool rather than competing for a smaller private one.

The speed question is the roofline of lesson 9.1, unchanged. Batch-one decode is memory-bound: producing one token streams the active language-model tensors, so a hard roofline is bandwidth divided by actual active bytes per token. Apple reports a memory bandwidth of roughly 800 GB/s for its Ultra-class desktop parts. Take that vendor figure and the pinned artifact size as a planning proxy:

About 47 tokens per second as an artifact-size heuristic. It assumes every file byte is active weight traffic and perfect bandwidth use, yet the GGUF also contains metadata and mixed tensors while cache, state, kernels, and thermals add other traffic. Because those effects move in opposite directions, this quotient is not a hard upper bound; profile active bytes and measured throughput.

Here is the classic wrong turn, and it is a purchase-sized mistake. A buyer reads that an Ultra-class machine can be configured with 192 GB of unified memory, notes that lesson 7.12 put bf16 weights at 54 GB, and concludes: excellent, I will skip quantization entirely and run the model at full precision on my desk. The capacity claim is correct — 54 GB fits in 192 GB with room for a long context. The speed conclusion does not follow, but the checkpoint-size quotient is only a planning comparison:

About 14.8 tok/s as a checkpoint-size heuristic. Capacity bought the ability to load bf16; it did not establish speed. Quantization can reduce active bytes, but the gain depends on the actual runtime representation rather than scaling mechanically with file size.

Two second-order effects deserve naming. First, the cache is not free of the same pool: at lesson 7.2’s 64 KiB per token, a 32,768-token conversation adds 2 GiB of resident KV state, and every byte of it is also read during decode, so long contexts erode the ceiling as well as the capacity. Second, the roughly 144 MiB of reference float32 Gated DeltaNet recurrent state is constant in sequence length, which makes the hybrid architecture unusually well-suited to a fixed-size memory budget — the cache still grows linearly with context — 8,000 tokens to 32,000 still quadruples it — but it grows from a base four times smaller than a same-size pure-attention model’s, because only 16 of the 64 layers cache at all, and the recurrent state beside it never grows.

The durable point is that Apple silicon did not repeal the bandwidth wall of lesson 9.1; it removed a different wall. The PCIe staging boundary that makes discrete-GPU capacity a hard cliff is gone, so the question “does it fit” gets a generous answer on consumer hardware. The question “how fast” is answered by exactly the same division as every other chip in this track, and lesson 9.10 will turn both answers into a monthly bill.