Quantization II: GPTQ, AWQ, GGUF, QAT

1

Concept

Quantization discussions often mix three layers: an algorithm chooses approximate values, a format stores tensors and metadata, and a runtime implements kernels for those representations. GPTQ, AWQ, GGUF, and QAT live at different layers. A useful comparison begins by separating their jobs.

GPTQ is a post-training weight-quantization method. Starting from a trained higher-precision model and calibration inputs, it quantizes weights while using approximate second-order information to compensate reconstruction error across a layer. Processing order and updates to remaining weights aim to preserve the layer’s outputs better than independent nearest-value rounding. Implementations expose group size, bit width, ordering, and kernel-specific packing choices.

AWQ, or Activation-aware Weight Quantization, observes activations on calibration data to identify salient weight channels. Its central premise is that not every weight contributes equally to observed outputs. Protecting a small important subset through scaling can reduce error while retaining weight-only low-bit execution. The activation data informs weight treatment; AWQ is not simply dynamic activation quantization.

Both methods are post-training quantization (PTQ): they start from a trained checkpoint and do not repeat full model training. Calibration quality and implementation matter. A quantizer tuned on short English prompts may not preserve the same behavior for code, other languages, or long context. Evaluate the resulting artifact, not only layer reconstruction.

Quantization-aware training (QAT) puts simulated quantization into the training forward pass. Fake-quantization rounds and clips as deployment would, while trainable higher-precision parameters receive approximate gradients. The optimizer can move values away from fragile boundaries and compensate across layers. QAT costs training compute and needs a recipe that matches the intended runtime format, but it can recover quality where one-shot PTQ struggles.

GGUF is a container format and tensor representation ecosystem associated with GGML and llama.cpp runtimes. It packages model metadata, tokenizer information, and tensors that may use different quantized types. Saying “a GGUF model” does not specify quality or bits. The tensor type, quantization tool, source checkpoint, conversion path, and runtime compatibility are essential.

Conversion is not magic. Converting higher-precision weights directly into a target quantization gives the algorithm the best available source. Converting one lossy quantization into another starts from already rounded values and usually compounds error. A container change can repack bytes but cannot infer discarded precision. Preserve provenance and hashes for the original checkpoint and every derived artifact.

Kernel coupling is decisive. GPTQ and AWQ artifacts may use different packing orders, scale layouts, zero points, and supported group sizes. A server can load a file yet fall back to dequantized matrix multiplication. GGUF tensor types likewise vary by backend. Measure actual memory, selected kernels, TTFT, output throughput, and quality on the target hardware.

Comparisons should hold source checkpoint, evaluation prompts, context lengths, decoding, and precision of non-quantized tensors constant. Report disk size separately from resident memory and KV-cache use. Include factual, code, multilingual, structured, safety, and long-context slices; quantization error can change low-margin choices without noticeably changing average perplexity.

The durable map is: GPTQ uses error-aware post-training reconstruction, AWQ uses activation evidence to protect salient weights, QAT adapts weights while simulating deployment error, and GGUF packages model tensors for compatible runtimes. Algorithms create approximations; formats transport them; kernels turn them into actual speed.