Quantization I: int8, int4, the basics

1

Concept

Model parameters are real-valued numbers, commonly trained and stored in floating-point formats. Quantization maps them—or runtime activations—onto a smaller discrete representation. The goal may be reduced model size, lower memory bandwidth, larger batches, or faster arithmetic. The cost is approximation error plus metadata and kernel complexity.

In symmetric uniform quantization, a real value $x$ is approximated using an integer code $q$ and scale $s$:

Affine quantization adds a zero point, allowing an asymmetric represented range. Rounding maps many real values to the same code; clipping maps extremes to the boundary. Their difference from the original produces quantization error.

“Int8” or “int4” is incomplete. Ask what is quantized: weights, activations, or KV cache? At what granularity: whole tensor, output channel, input group, or token? Are scales floating point, and how often are they stored? Is the representation symmetric? What compute dtype accumulates products? A four-bit weight format can still use higher-precision activations and accumulators.

Granularity controls a trade-off. One scale for an entire tensor has little metadata but must cover the widest magnitude. A single outlier can enlarge the step size, making common small values poorly resolved. Per-channel scales adapt to rows or columns. Groupwise quantization divides a dimension into smaller chunks. Smaller groups usually track local distributions better but store more scales and complicate kernels.

Weights and activations behave differently. Weights are fixed after training and can be calibrated offline. Activations depend on the prompt and can contain channel-wise outliers. SmoothQuant, published in 2022, moves some quantization difficulty from activations into weights through a mathematically equivalent scaling transformation, making weight-and-activation quantization easier for supported models.

Weight-only quantization reduces bytes read for weights, especially valuable during memory-bound decode. The kernel may unpack and dequantize tiles into a compute dtype while multiplying. If the runtime first expands the entire model into floating point, storage shrinks but execution memory and bandwidth do not. Native packed kernels are the difference between a compact file and efficient inference.

Accuracy loss is uneven. Some layers, channels, or tensors are more sensitive. Embeddings and output heads may use different precision. Mixed schemes retain selected weights at higher precision. Perplexity is one signal, but task accuracy, structured-output validity, multilingual behavior, rare-token predictions, and long-context stability can reveal failures averages miss.

Calibration chooses ranges or scales from representative data in many post-training methods. A narrow calibration set can preserve familiar activation patterns and fail elsewhere. Record dataset, sequence lengths, languages, preprocessing, algorithm, and software version. Quantized artifacts are derived models and need independent hashes and evaluations.

The simplest memory estimate divides raw element bits by eight, then adds scales, zero points, indexes, alignment, and any higher-precision tensors. Runtime memory also includes KV cache, activations, workspace, and allocator overhead. A model file that fits in RAM may still fail to run.

The durable model is a measured approximation. Discrete codes save movement and capacity; scales reconstruct useful magnitudes; grouping adapts to local statistics; accumulators preserve arithmetic range. Quantization succeeds when the error stays inside the task’s tolerance and the hardware truly consumes the compact form.