Rounding the numbers
Quantization reduces the numeric precision of a model’s parameters in order to shrink its size and its computational cost. The name sounds technical; what it does is round off digits that are finer than necessary.
How it works
Parameters are normally stored at high precision. Replacing 16-bit values with 4-bit ones cuts file size to a fraction and substantially reduces both memory and computation.
It resembles lowering image quality to make a file smaller. There is some degradation; the craft lies in compressing while keeping the effect on capability minimal.
What it buys you
- LLMs that run on ordinary computers and phones — the reason local LLMs became practical
- Lower inference cost: bigger models or more requests on the same hardware
- Less energy, since there is less computation
When a model download page offers a 4-bit or 8-bit version, those are compression levels of the same model. Smaller numbers are lighter and slightly less capable.
How it differs from distillation
Both reduce weight, but distillation has a smaller model learn the capability, while quantization compresses the numeric representation of the same model. In practice they are combined to get capable models down to sizes that run on your own hardware.
Quantization has nothing to do with quantum computing — it is an older term from signal processing. Think of it as putting a model on a diet.
You will meet the word as soon as you try local LLMs. Knowing that 4-bit is light but coarser and 8-bit is the balanced choice is enough to choose a download.