Quantization II: GPTQ, AWQ, GGUF, QAT

01 · Concept

Concept

Search a model hub for quantized builds of Qwen3.8-27B and you meet a wall of suffixes. Community quantizations of this model circulate across the llama.cpp/GGUF, Ollama, LM Studio, and MLX ecosystems — maintained by those communities rather than by the model’s authors. The names on those artifacts mix three different layers: an algorithm that chooses approximate values, a format that stores tensors and metadata, and a runtime whose kernels consume the result. GPTQ, AWQ, GGUF, and QAT each live at a different layer, and choosing well begins by separating their jobs.

GPTQ is a post-training weight-quantization method. Starting from the trained bf16 checkpoint and calibration inputs, it quantizes weights column by column while using approximate second-order information to compensate reconstruction error across the layer — later weights adjust to absorb damage done by earlier rounding, preserving layer outputs far better than independent nearest-value rounding. Implementations expose bit width, group size, and ordering choices.

AWQ, activation-aware weight quantization, observes activations on calibration data to find the small fraction of weight channels that matter most to observed outputs, then protects them via scaling before low-bit conversion. The activation data informs how weights are treated; the shipped artifact is still weight-only low-bit. Both GPTQ and AWQ are post-training quantization (PTQ): no retraining, so quality depends heavily on calibration coverage — a set of short English prompts may not preserve behavior on code, other languages, or Qwen’s very long contexts.

A third member of the same family is the one you will actually see named on the Hub. An importance matrix — imatrix — is produced by running calibration data through the model and accumulating, per tensor column, the sum of squared activations passing through it; the quantizer then weights its error by that statistic, spending scarce precision on the columns the corpus exercised hardest. It measures activations rather than per-weight sensitivity, and it is a measurement rather than a format, so it can be applied underneath several of them. Lesson 7.13 works through the artifacts it produces and the distinction it is most often confused with.

QAT, quantization-aware training, moves the effort earlier: fake-quantization ops round and clip inside the training forward pass while higher-precision master weights receive approximate gradients, so the optimizer steers values away from fragile rounding boundaries. It costs training compute and needs a recipe matched to the deployment format, but can recover quality where one-shot PTQ struggles at aggressive bit widths.

GGUF is not an algorithm at all. It is the container ecosystem of llama.cpp-family runtimes: one file carrying model metadata, tokenizer information, and tensors in any of a ladder of quantized types from 8-bit down past 4-bit. Saying a GGUF of Qwen3.8-27B says nothing about quality until you name the tensor type, the tool that produced it, and the source checkpoint.

The file sizes themselves teach the layering. Pure 4-bit codes for a rounded 27 billion parameters would occupy $27\text{e}9 \times 4 / 8 = 13.5$ GB. At the pinned community revision, the Q4_K_M text-model file is exactly 17,106,773,984 bytes: 17.1 GB, or 15.93 GiB. Group metadata and tensors retained at higher precision account for the gap. Do not reverse that file size through the marketed 27B count and call the result exact bpw: the marketed count is rounded and includes the vision tower, while the GGUF is the text model; the multimodal projector is a separate roughly 0.93 GB file.

A second classic wrong turn is treating conversion as free. Repackaging bf16 weights into a 4-bit format hands the algorithm the best available source. Requantizing an already 4-bit artifact into another format starts from values that were rounded and clipped once; the discarded precision is gone, and errors compound. A container can change; information cannot return. Keep provenance and hashes for the original checkpoint and every derived artifact.

How much quality do the rungs cost on this model? Hedge honestly: no dedicated technical report exists for Qwen3.8-27B, and community quantizations ship without standardized evaluations, so treat any specific perplexity or benchmark delta you find as unverified. The qualitative shape is stable across models: 8-bit weight-only is usually hard to distinguish from bf16; mainstream 4-bit recipes give up measurable but often acceptable quality; below 4 bits degradation grows quickly and unevenly, hitting low-margin decisions — rare tokens, structured output, long-context retrieval — before averages move. Evaluate the artifact on your tasks and languages before trusting it.

The durable map: GPTQ reconstructs with error awareness, AWQ protects what activations prove salient, QAT lets training adapt to deployment error, and GGUF packages the result for compatible runtimes. Algorithms create approximations, formats transport them, kernels turn them into speed — and lessons 7.12 and 8.7 put these artifacts onto real machines.