MLX

01 · Concept

Concept

Here is a purchasing question that a spec sheet cannot answer. You have a desktop with a discrete 24 GB GPU and a Mac with 32 GB of unified memory, and you want to run quantized Qwen3.8-27B with long documents. The discrete GPU may offer much higher bandwidth; the Mac exposes a different memory topology. Which deployment works better depends on the realized working set and measured throughput, and understanding why is the whole point of MLX.

MLX is an array framework built for Apple silicon, with an API deliberately close to NumPy and a neural-network layer close to PyTorch, so most model code reads as you expect. Two design decisions make it different, and both follow from the hardware.

The first is lazy evaluation. Writing an expression in MLX does not compute it. It records a node in a computation graph, and the graph is executed when some value is genuinely required — when you print it, convert it, or force evaluation explicitly. This is not a performance trick bolted on; it is the default execution model. The payoff is that chains of operations can be fused into fewer kernels and intermediate arrays that nothing ever reads need never be materialised at all. The cost is that timing a line of code measures nothing: the work you think you just did has probably not happened yet, which trips up every newcomer benchmarking a forward pass.

The second is unified memory, and this is the one that answers the purchasing question. On a conventional accelerator, host memory and device memory are separate pools connected by a bus; a tensor exists in one place, and using it elsewhere means an explicit copy. MLX arrays live in memory that both the CPU and the GPU address directly. An operation is scheduled onto a device; the array does not move. There is no .to(device) in the mental model, no pinned-memory staging, and — critically — no ceiling defined by a separate VRAM budget. The model’s working set is bounded by system RAM and by how much of it the GPU is permitted to hold, not by a soldered 24 GB.

Do not turn unified-memory capacity into a paper fit calculation. Lesson 7.2 owns Qwen3.8-27B’s KV-cache and recurrent-state derivation, and lesson 9.3 owns the complete memory budget. The reference implementation’s Gated DeltaNet state is about 144 MiB per sequence when stored in fp32, constant in context length; MLX may use a different dtype or layout, while Metal allocations, the operating system, and graph-evaluation peaks draw from the same physical pool. Load the exact converted artifact, force evaluation, and measure peak working set at the intended context and concurrency before deciding whether a 32 GB or 64 GB machine has usable headroom.

mlx_lm.convert --hf-path Qwen/Qwen3.8-27B -q --q-bits 4 --mlx-path qwen3.8-27b-4bit
mlx_lm.generate --model qwen3.8-27b-4bit --temp 0.7 --top-p 0.80 \
  --top-k 20 --min-p 0 --chat-template-config '{"enable_thinking":false}' \
  --prompt "..."

The mlx-lm package supplies the model implementations, the conversion and quantization path from a Hugging Face checkpoint, generation, and a local server. The second command explicitly disables thinking through the template and passes every part of the card’s instruct sampling preset that this pinned CLI exposes: temperature 0.7, top_p 0.80, top_k 20, and min_p 0. Omitting --top-k 20 would not preserve the preset, because mlx-lm defaults top-k to 0, which disables that filter. This CLI revision has no presence-penalty or repetition-penalty flags, so the command is a disclosed partial implementation of the card’s full preset, not an exact reproduction. The same discipline from lesson 8.8 applies here: state the effective configuration and its unsupported fields rather than silently inherit defaults.

Two things follow from that conversion command being one line long. The first is that you rarely need to run it: most people download somebody else’s output instead, and for MLX that somebody is usually the mlx-community organisation on the Hub, which hosts conversions of a very large number of models. Membership of that org means a conversion was uploaded, not that it was evaluated — lesson 7.13 is about telling those apart. The second is that --q-bits 4 is the blunt version of the tool. mlx-lm also ships mixed recipes that assign different widths to different layers (mixed_2_6, mixed_3_4, mixed_3_6, mixed_4_6), and a learned-quantization path whose --target-bpw takes an average as the goal and works out the per-layer assignment itself. That is where a checkpoint advertising a fractional bits-per-weight comes from, and why a number like 3.8 is a setting rather than a mistake.

The classic wrong turn is to choose from brochure capacity alone. Artifact size is not the full working set on either machine: weights, cache, recurrent state, runtime buffers, and concurrency share the available pool, and their realized dtypes and layouts belong to the exact runtime. A discrete GPU generally offers higher decode bandwidth while its measured working set fits; if it requires CPU spill, PCIe can become the bottleneck. Unified memory removes the explicit host-to-device copy boundary and can expose a larger shared pool, but the OS and Metal compete for it and swap is not usable serving capacity. Benchmark the exact artifact and workload on both paths; do not infer a winner from 24 GB versus 32 GB.

The durable idea is that memory topology is an architectural choice with consequences a spec sheet hides. Lazy graphs buy fusion; unified memory buys a ceiling. Lesson 9.6 takes the bandwidth side of this bargain seriously and works out what tokens per second the arrangement can actually sustain.