Quantization

How it works

Quantization shrinks a model by storing its weights (and sometimes activations) at lower numeric precision, for example 8-bit or 4-bit integers instead of 16-bit or 32-bit floating point. Fewer bits per weight means a smaller model that uses less memory and runs faster, at the cost of some precision. Done carefully, the accuracy loss is small relative to the large gains in memory footprint and inference speed, which is why quantized models are common for running LLMs on constrained hardware or at lower cost.

When it matters

Quantization matters when you need to run a model cheaply, on smaller GPUs, or on-device (edge and mobile), and when inference cost or latency is the constraint. It is one of the main levers for making a capable model affordable to serve at scale. The tradeoff is quality: aggressive quantization can degrade reasoning or accuracy, so you evaluate the quantized model on your own task rather than assuming parity.

Common misconceptions

• "Quantization is free accuracy." There is a quality tradeoff; measure it on your workload.
• "Quantization equals distillation." Quantization lowers weight precision; distillation trains a smaller model to mimic a larger one. Different techniques.
• "4-bit is always fine." How far you can push precision down depends on the model and task.