Learn AI · Level 4 · Frontier view · step 2 of 15 · word 4 of 4
What is quantization in AI?
Quantization shrinks an AI model by storing its numbers with less precision, for example 4 bits instead of 16, so it uses less memory and runs faster with only a small loss in quality.
In plain English
A model is billions of numbers, its parameters. Normally each is stored very precisely. Quantization rounds them off, like saving a photo at lower quality. The file gets much smaller and the model runs faster, and most answers barely change.
Going from 16 bits to 4 bits per number makes a model about four times smaller.
An example
A 70-billion-parameter model needs about 140 GB of memory at 16 bits, far more than any laptop. At 4 bits it needs about 35 GB, which some powerful laptops and desktop computers can handle. That’s how people run open models at home (see run AI on your computer).
Watch out
Squeeze too hard and quality drops, especially for maths, code and less common languages. When a model is offered in several sizes, test the smaller versions on your own tasks.
Learn AI path · Level 4 · Frontier view · step 2 of 15
Done? Next stepThe AI ladder: how close are we to really smart AI? →See all of level 4