Quantization reduces the numerical precision of a model's weights, for example from 16-bit to 4-bit numbers, so the model uses less memory and runs faster with only a small loss in quality. It is like compressing a high-resolution photo: the file shrinks but the image stays recognizable. Quantization lets businesses run capable language models on ordinary laptops or local servers. A Moroccan clinic worried about data privacy could run a quantized model on-site instead of sending patient notes to the cloud.
AI Infrastructure
Quantization
Related terms
Learn to use these concepts in practice.
Join the 212AY Academy →