Quantization (LLM)
Quantization stores a model's weights at lower numeric precision — 8, 4, or even 2 bits instead of 16 — trading a small, measurable quality loss for a large cut in memory and bandwidth. It is the technique that moved 70B-class models from data-center GPUs onto desktops.
The precision ladder
| Format | Bits/weight | 70B model size | Quality |
|---|---|---|---|
| FP16 | 16 | ~140 GB | Reference |
| Q8 / INT8 | ~8.5 | ~74 GB | Effectively lossless |
| Q6_K | ~6.6 | ~57 GB | Very close |
| Q5_K_M | ~5.7 | ~49 GB | Minor degradation |
| Q4_K_M | ~4.85 | ~42 GB | Standard local default |
| Q3 / Q2 | <4 | ~30 GB | Noticeable degradation |
How it works
Weights within a block share a scale factor; each weight is stored as a small integer offset from that scale. K-quants (the _K suffix in GGUF) use per-sub-block scales and keep more precision for the layers that matter most. AWQ and GPTQ instead calibrate on sample data to protect the most salient weights.
The trade-off is not linear: Q8 is nearly free, Q4 is the accepted sweet spot for most local use, and below Q4 the loss compounds — especially on reasoning and code. Quantizing the KV cache is a separate decision from quantizing the weights.