Once a model is in GGUF, the next question is which quant to ship. The format supports a whole ladder of tiers, and the "k-quants" add _m (medium) and _s (small) variants on top of that. They all trade the same three things against each other: file size, answer quality, and speed. This is how we pick a rung for theLAB's ADI line.
The short version: q4_k_m is the default sweet spot, and you only climb the ladder when your own eval set tells you to. Sizes below are approximate for a 7B model — they scale with parameter count, so a 4B lands proportionally smaller.
The tiers
| Quant | ~Size (7B) | Quality | Use it when |
|---|---|---|---|
q4_k_m | approximately 4.4 GB | Strong — the practical floor for "feels like the full model" | Default for almost all local use |
q5_k_m | approximately 5.1 GB | A touch better than q4 | Good middle ground with spare VRAM |
q6_k | approximately 5.5 GB | Near-lossless vs f16 | Quality-critical work |
q8_0 | approximately 7.2 GB | Effectively lossless | Rarely worth it locally |
f16 | approximately 14 GB | Unquantized reference | Conversion intermediate / baseline |
Quality: where the floor is
Perplexity rises gently as you step down from q8 → q6 → q5 → q4. The whole top of the ladder is close enough that most users can't tell q8 from q4_k_m on general chat. The cliff is below q4: q3 and q2 degrade hard and fast. That's why we treat q4_k_m as the practical floor — the last rung that still feels like the full model.
q4_k_m, but a coder model may need q5_k_m or q6_k to keep generated code correct — a single flipped token breaks a function in a way it never breaks a paragraph. Score it on a held-out set, don't eyeball it.
Speed and footprint
On the memory-bandwidth-bound GPUs we run, smaller quants are faster, not just lighter. Fewer bytes per weight means fewer bytes to move per token, so tokens/sec goes up, and more of the model (plus more KV cache) fits in VRAM. So the trade is clean and one-directional: smaller is faster and cheaper on memory, paid for in quality. That's the whole reason the ladder exists.
What we ship
theLAB defaults the ADI line to q4_k_m. For example, adi-qwen3.5-4b-glm5.2-general is a 2.7 GB q4_k_m for a 4B base. We bump to q5_k_m or q6_k only when there's spare VRAM and we measure real degradation on a model's eval set; q8_0 and f16 stay reserved for quality-critical models or as a conversion intermediate.
# quantize an f16 GGUF down to q4_k_m
./llama.cpp/build/bin/llama-quantize in-f16.gguf out-q4_k_m.gguf q4_k_m
# swap the last arg for a different rung: q5_k_m, q6_k, q8_0 ...f16 (or bf16) GGUF, never from a lower tier. Re-quantizing a q4_k_m into q5 can't recover lost precision — it just bakes the q4 loss in and adds more on top. Keep the f16 around as the source of truth.
The rule of thumb
Default to q4_k_m. Climb to q5_k_m or q6_k if you have the VRAM and your eval set shows degradation. Reach for q8_0 or f16 only when quality is non-negotiable or you need the unquantized intermediate for a conversion.