Choosing and judging a community quant

01 · Concept

Concept

Lesson 7.12 ended with a decision already made: you know which rung of the quantization ladder your machine affords. Now you search the Hub for that model and get back forty repositories. Some are published by an organisation, most by individuals. Their names disagree about almost everything. One says 4bit, one says Q4_K_M, one says iQ-3.8bpw, and one is somebody’s experiment uploaded last night. They are not the same file, they will not behave the same way, and nothing on the page ranks them.

This lesson is about reading those names, and then about not trusting them.

What a name can tell you

None of that is a standard, and the disagreement you just saw is the proof: a repository id is a namespace and a free-text name, so these are clues that may appear, in roughly this order, not fields that every artifact carries. 4bit names a recipe and no bit rate; Q4_K_M names a recipe whose bit rate you have to look up; iQ-3.8bpw names both. Plenty of repositories omit the format because the whole namespace implies it, and plenty attach the bit rate to individual files rather than the repository.

Of the clues that do appear, three are dependable. The base model names the weights the artifact was derived from. The container format — GGUF, MLX, safetensors — decides which runtimes can load the file at all, which is the check lesson 7.12 told you to make before downloading sixteen gigabytes. Bits per weight, when it is present, is a measurement.

The other two are not dependable, and confusing them for standards is the common mistake. A publisher name is not a quality signal: mlx-community is where a great many MLX conversions live, and being there means somebody uploaded a conversion, not that anybody evaluated it. The recipe label is worse. Outside llama.cpp’s Q and IQ families, a label like iQ or oQ4e is the publisher’s private shorthand for a recipe they chose. It may encode something careful. You cannot tell from the string.

Bits per weight is the only honest number

Lesson 7.9 established that a 4-bit artifact does not consist only of half-byte codes: group metadata and deliberately protected tensors ride alongside them. Lesson 7.10 pins Qwen3.8-27B’s Q4_K_M text file at 17.1 GB (15.93 GiB) against a 13.5 GB pure-code floor. Bits per weight — bpw — names an artifact’s measured storage rate, but computing it exactly requires that artifact’s exact tensor denominator, not a rounded marketed parameter count.

Once you can see it, the labels look different. Measured on Llama-3.1-8B, Q4_K_M stores 4.8944 bits per weight and Q8_0 stores 8.5008. Neither is the round number in its name. So a repository advertising 3.8bpw is not being eccentric — it is being more precise than the one advertising 4bit, and it is telling you something the other name conceals.

There is a second reason the number is fractional, and it is the more interesting one. A recipe need not use the same width everywhere. Mixed quantization assigns different bit widths to different layers — protecting the projections that hurt most when coarsened and spending less on the rest — and the average lands wherever that mix puts it. In mlx-lm these ship as named recipes (mixed_2_6, mixed_3_4, mixed_3_6, mixed_4_6) and, in the learned-quantization path, as a target you set directly: --target-bpw asks for an average and lets the tool choose the per-layer assignment that reaches it. A 3.8 is a dial position, not a typo.

Importance is measurable

The question a mixed recipe has to answer is which layers deserve the bits. Lesson 7.10 gave one family of answers: GPTQ compensates for error layer by layer, AWQ protects the channels that activations show to be significant. The Hub label you have not met yet belongs to the same family.

An importance matrix, usually written imatrix, is built by running calibration data through the model and accumulating, per tensor column, the sum of squared activations that flow through it. Columns the calibration data exercises hard score high; columns it barely touches score low. The quantizer then weights its error accordingly, spending precision where the statistic says it matters. Note what that is and is not: it is a measurement of the activations a corpus produces, not of how much each individual weight moves the output — and llama.cpp’s own documentation flags that using squared rather than raw activations makes the statistic less reliable than it first sounds.

Two corrections most readers need. First, an importance matrix is not the same thing as an IQ format: K-quants are routinely built with one, and many well-regarded Q4_K_M files on the Hub are imatrix-calibrated. Nor does every IQ type require one. llama.cpp refuses outright at its lowest rungs — the IQ1 and IQ2 families, the smaller IQ3 types and Q2_K_S — where it warns the result would be garbage without calibration data; IQ4_XS and IQ4_NL build without one and merely benefit. The check is per tensor and keyed on the target type, so the exact set moves between releases: read the error, do not memorise a list. Second, it matters most where precision is scarcest, which is the same statement from the other side: above roughly 4 bits the benefit is modest; at the rungs that refuse to build without it, it is the difference between a working model and garbage. MLX’s analogue is sensitivity-based dynamic quantization, which estimates each layer’s influence and saves the per-layer sensitivities so the same measurement can be reused for other targets.

Why two 4-bit files differ

Lesson 7.9 warned that bit width alone predicts neither speed nor quality, and lesson 8.7 observed that two converters produce measurably different files. Now the mechanism is available. Two artifacts at the same average can differ in group size — how many weights share one scale, and therefore how closely scales track local statistics; in which tensors were kept at higher precision; and in the calibration corpus, since a quantizer tuned on short English prompts has been shown nothing about code or about a hundred-thousand-token document.

That is why a practitioner’s report that one 4-bit build beats another is not surprising and not mystical. Same nominal width, different recipes, different measurements underneath.

What fewer bits actually buy

The first half of that figure is lesson 7.3 made visible. Decode re-reads roughly the whole weight set for every token it generates, so it is bandwidth-bound and shrinking the weights attacks its binding constraint directly. In this Llama-3.1-8B benchmark, the quantized rows generate 1.75 to 3.09 times as fast as F16: Q8_0 is the low end at 50.93 versus 29.17 tokens per second, while Q2_K_S is the high end at 90.01. Prefill is compute-bound, and quantization does not multiply arithmetic units — worse, dequantizing is extra arithmetic — so prompt processing does not improve and in fact slips, from 923 tokens per second at F16 down to 709 at the slowest quantized rung.

The second half is the part worth slowing down for, because it refuses to be a rule. Below about 8 bits, size stops predicting speed. In this table Q2_K_S at 2.97 bpw is the fastest artifact measured, at 90 tokens per second — faster than every 4-bit build. Q4_K_S at 4.67 bpw decodes at 76.7 while the smaller Q3_K_S at 3.64 manages only 69.8. And Q4_K_M, the most-downloaded rung on the Hub, is slower than both at 71.9 despite sitting between them in size.

The mechanism is kernel efficiency, not bytes. Once a format is small enough that the memory bus is no longer the only constraint, how well its unpacking kernel maps onto the hardware decides the rest — which is why i-quants, whose codebook lookups cost decode time a k-quant never pays, can lose to larger k-quants, the caveat lesson 8.7 raises from the llama.cpp side.

So when someone reports that 6-bit is the sweet spot for them, they are making a claim about fidelity against speed that is entirely plausible and not derivable from bit width. Nothing in this table would let you predict it. That is the honest reason the last section of this lesson exists.

Judging one yourself

The measurement is not elaborate. Fix a prompt set drawn from work you actually do, not from a leaderboard. Fix the sampling parameters and state them — the model card’s instruct preset, per lesson 8.9’s discipline, rather than whatever a runner defaults to. Run both artifacts on the same machine, same runtime, same context length, and record load time, time to first token, tokens per second and peak resident memory beside your judgement of the output, as lesson 7.12’s benchmarking note sets out. Then run it again, because a laptop ten minutes into a session is a different machine.

And apply lesson 0.8’s reflex to what you read. When someone reports that one quant won their overnight comparison, or names a private benchmark of their own, they have given you a hypothesis formed on their hardware, their prompts and their thermal envelope. That is genuinely useful — it is more evidence than the filename carries. It is not a result until you know the harness, the number of samples and the variance, and those are usually the parts that go unstated.

The durable idea is a hierarchy of trust. The container format is a fact. Bits per weight is a measurement. The recipe label is a claim. The ranking is a stranger’s experiment. Only the last two are about quality, and they are the two you cannot check without running the thing yourself.