Transferring capability to a smaller model
Knowledge distillation transfers the capability of a large, capable model — the teacher — into a small, light one — the student. As distilling a spirit extracts its essence, the essence of a huge model’s knowledge is concentrated into something compact, which is where the name comes from.
How the transfer works
The student trains not only on correct answers but on how the teacher answers. The teacher’s output carries nuance — this answer at 90%, the next at 8% — and learning that whole distribution lets a small model acquire judgment close to the teacher’s.
Think of a student who watches not just what the teacher writes on the board but how they think through the problem.
Why it matters
- Cost: small models are far cheaper to run
- Speed: responses are faster and more comfortable to use
- Local execution: sizes that fit on a phone or laptop
Alongside building the most capable model, delivering that capability at a usable size has become just as important, and distillation is the main technique for it. Much of the AI people use daily runs on distilled models.
Why it makes the news
Distillation is standard practice, but disputes have arisen over whether using another company’s model output to distill your own breaches its terms. Sitting at the intersection of the scale race and the cost race, it appears in industry news regularly.
Coverage uses phrases like distilled version. Read it as a lighter model that inherited a larger one’s capability. Combining capability with low cost is central to making AI practical.
Major labs now routinely ship distilled, faster, cheaper versions of their top models as part of the official lineup.