Quantization I: int8, int4, the basics

01 · Concept

Concept

You want to run Qwen3.8-27B, and the first number that greets you is 54 GB — the bf16 weight footprint the course’s budget math establishes (lessons 7.2 and 9.3). No consumer GPU holds that, and even an 80 GB data-center card surrenders most of itself before a single token of cache is allocated. Quantization is the standard answer: store each parameter in fewer bits plus a little scale metadata, and accept some approximation error in exchange.

The ladder is easiest to see as bytes per parameter. bf16 spends two bytes on each of the model’s roughly 27 billion parameters — the 54 GB starting point. Int8 spends one byte: halve it, about 27 GB. Int4 spends half a byte: halve again, a 13.5 GB floor. At the pinned community revision, near-4-bit text-model GGUF files span 15.7–17.9 GB (14.6–16.7 GiB), and Q4_K_M is 17.1 GB (15.93 GiB), because group metadata and selected tensors kept at higher precision ride alongside the codes. That real rate is often described as bits per weight, or bpw, but an exact bpw needs the artifact’s exact tensor denominator; the marketed 27B count is rounded and includes the separate vision tower. A 24 GB consumer GPU is therefore a course deployment scenario for these text weights plus cache and runtime headroom, not a minimum claimed by the model card. Lesson 7.12 works through that scenario.

What the rungs actually do is map real values onto a discrete codebook. In symmetric uniform quantization, a real value $x$ becomes an integer code $q$ through a scale $s$:

Affine variants add a zero point so the represented range need not straddle zero symmetrically. Rounding maps many nearby reals to one code; clipping maps extremes to the boundary; the difference between $x$ and $\hat{x}$ is quantization error.

Saying int8 or int4 alone is an incomplete sentence. Ask what is quantized — weights, activations, or the KV cache; at what granularity — whole tensor, output channel, group, or token; whether the representation is symmetric; and what dtype accumulates products. A 4-bit weight format routinely multiplies against 16-bit activations and accumulates in higher precision.

Granularity is the central trade. One scale per tensor is nearly free in metadata but must span the tensor’s widest magnitude, so a single outlier weight coarsens the step size for every ordinary value. Per-channel scales adapt to rows or columns; groupwise quantization slices a dimension into chunks of, say, 32–128 values, each with its own scale — better local fit, more metadata, more complex kernels. Weights and activations also behave differently: weights are fixed and can be calibrated offline, while activations depend on the prompt and can carry violent channel-wise outliers. SmoothQuant’s contribution was a mathematically equivalent rescaling that shifts quantization difficulty from activations into weights, easing joint weight-and-activation schemes.

Why does this help speed and not just disk space? Recall lesson 7.3: decode is memory-bound, moving roughly the full weight set per generated token. Weight-only quantization cuts precisely those bytes. Here is the classic wrong turn: a runtime loads a 4-bit file, then expands every tensor to floating point in memory before computing. The download was small; resident memory and bandwidth are unchanged; tokens per second do not move. The correction is native packed kernels that dequantize tile-by-tile inside the multiply, so the compact form is what actually crosses the memory bus. Always verify which path your runtime took by measuring resident memory after load, not by reading the filename.

Accuracy loss is uneven across the network: some layers and channels are far more sensitive, and mixed schemes keep embeddings, output heads, or fragile tensors at higher precision. Perplexity is one signal, but structured-output validity, multilingual behavior, and long-context stability can reveal failures that averages hide. Calibration sets matter too — a quantizer tuned on short English prompts may not preserve behavior on code or on 100,000-token documents.

The durable model is a measured approximation. Discrete codes save capacity and, with the right kernels, bandwidth; scales reconstruct useful magnitudes; grouping adapts to local statistics; accumulators preserve range. The ladder from 54 through 27 toward the pinned 15.7–17.9 GB near-4-bit text artifacts is what moves Qwen3.8-27B from the data center toward your desk — and lesson 7.10 owns the exact artifact sizes and sources.