Learn AI · Level 4 · Frontier view · step 2 of 15 · word 3 of 4
What is distillation in AI?
Distillation trains a small "student" model to copy the answers of a big "teacher" model, so you get much of the big model's skill in a model that is cheaper and faster to run.
In plain English
A big model is expensive to run. So you ask it thousands or millions of questions, keep its answers, and train a smaller model on them. The student learns to give similar answers without the teacher’s size. The idea comes from a 2015 paper by Geoffrey Hinton and colleagues.
An example
When DeepSeek released its R1 reasoning model in January 2025, it also released smaller versions made by distilling R1 into open models from Alibaba (Qwen) and Meta (Llama). Some of them run on a home computer.
Watch out
The student is rarely as good as the teacher, especially on unusual questions. Distilling from another company’s model without permission usually breaks its terms of use, and labs have accused rivals of doing exactly that.