llama.cpp and GGUF

01 · Concept

Concept

You downloaded one file. It is exactly 17,106,773,984 bytes in the pinned release, its name ends in Q4_K_M.gguf, and it claims to be Qwen3.8-27B. Lesson 7.12 told you which rung of the quantization ladder that is; this lesson opens the box. What is inside a GGUF, why is this pinned file 17.1 GB rather than the naive 13.5 GB estimate, and what has to be true about your machine before the file is worth the download?

Start with the engine, because the format exists to serve it. llama.cpp is an inference runtime written in C and C++ with no Python runtime dependency, built on the ggml tensor library. Its defining constraint is that a model must be one artifact and one binary, runnable on a laptop CPU, an Apple GPU through Metal, an NVIDIA GPU through CUDA, or a cross-vendor path through Vulkan, with the same file in every case. That constraint produced GGUF.

A GGUF file is a container, and the single most common misunderstanding is to treat it as a compression method. It is not. The layout is a header of typed key-value metadata followed by a tensor directory followed by the tensor bytes. The metadata names the architecture, the hyperparameters a runtime needs to build the compute graph, the full tokenizer including merges and special tokens, and the chat template. The tensor directory gives every tensor a name, a shape, an offset, and — this is the important part — its own quantization type. A GGUF is therefore self-describing and mmap-able: the runtime maps the file, reads the manifest, decides before allocating anything whether it can build this graph, and pages weights in from disk rather than copying them through a loader.

The compression lives in those per-tensor types, and they form a ladder. Q8_0 is a straightforward 8-bit block scheme, close to lossless in practice and about half the bf16 size. The k-quants — Q6_K, Q5_K_M, Q4_K_M, Q4_K_S, Q3_K_M, down to Q2_K — use a super-block structure in which small blocks of weights share a scale and the scales themselves are quantized against a block-level scale, which is how they buy precision at low bit rates. The trailing letter is a mix policy rather than a bit width: an M variant keeps tensors the converter considers sensitive at a higher type than the S variant does. That is why the arithmetic from lesson 7.10 works out. Nominally 4 bits over 27 billion parameters gives $27\text{e}9 \times 0.5 = 13.5$ GB; the scales, super-block metadata, and deliberately preserved tensors add bytes, and the pinned artifact lands at 17.1 GB (15.93 GiB). The rounded 27B label is not a sufficient denominator for an exact effective-bits claim. The gap is not waste. It is the reason the file is still usable.

Below the k-quants sits a second family, the i-quants — IQ1_S, IQ2_XXS through IQ4_XS and IQ4_NL. These reconstruct weights from a codebook rather than a plain scaled integer, which is what lets them stay coherent at bit rates where a k-quant falls apart. Two things they are routinely confused with, both worth getting straight before you download one. The importance matrix is separable from the format: plenty of well-regarded Q4_K_M files are imatrix-calibrated, and not every IQ type needs one — llama.cpp refuses at its lowest rungs (the IQ1 and IQ2 families, the smaller IQ3 types, and Q2_K_S), warning that the result would otherwise be garbage, while IQ4_XS and IQ4_NL build fine without. The check is per tensor and keyed on the target type, so the precise set shifts between releases. And lower is not automatically better value, because codebook lookups cost decode time that a k-quant does not pay. Reach for an i-quant when memory is the binding constraint, not as a general upgrade. Lesson 7.13 compares these artifacts properly.

Do not repeat the model’s memory budget from artifact size alone. Lesson 7.2 owns the model-specific KV-cache and recurrent-state derivation, and lesson 9.3 owns the complete accelerator budget after runtime reserve. The reference implementation’s Gated DeltaNet state is about 144 MiB per sequence when stored in fp32, constant in context length; llama.cpp can use a different dtype or layout, and its compute buffers and graph workspace depend on the exact build and backend. On either a 24 GB or 16 GiB-class device, load the exact GGUF, inspect the realized allocation and offload reported by llama-server, and validate the intended context and concurrency instead of declaring a fit from the nominal VRAM ceiling.

llama-server -m Qwen3.8-27B-Q4_K_M.gguf -ngl 99 -c 8192 \
  --temp 0.7 --top-p 0.80 --top-k 20

If the measured allocation requires partial offload, the performance warning is still real. Decode is bandwidth-bound — each token reads essentially all resident weight bytes — so layers left on the host run at system-memory speed and their activations cross PCIe during every token. A configuration with ninety percent of layers on the GPU does not imply ninety percent of full-offload throughput; benchmark the exact split. The honest decision is between a smaller artifact with measured quality loss and the measured CPU-offload throughput, not a fit inferred from the filename.

The durable idea is separation of concerns. The quantizer decided how many bits each tensor deserves; the container recorded those decisions in a manifest a runtime can read before committing memory; the engine builds a graph if it recognises the architecture and refuses if it does not. Everything downstream in this track — the convenience runners in lesson 8.8, and the comparison with Apple’s own path in lesson 8.9 — is built on that one file and that one manifest.