Last reviewed:
What is AI model distillation? Definition and business implications
Distillation is a technique that transfers the knowledge of a large AI model (teacher model) to a smaller model (student model), while preserving most of the performance. It enables the deployment of lightweight models with reduced inference cost, viable on more modest infrastructures.
The concept was formalised by Hinton, Vinyals, and Dean in 2015 in the paper Distilling the Knowledge in a Neural Network. The principle: instead of training the student model directly on the original data, it is trained to imitate the outputs (probabilities, logits) of the teacher model. The student learns not only the right answers, but also the relative confidence the teacher model assigns to each alternative. This so-called soft information carries far more learning signal than binary labels alone. DistilBERT (Sanh et al., 2019) is the historical example: it achieves 97% of BERT's performance while being 40% smaller and 60% faster at inference. Several families of small open models are officially produced by distillation: Llama 3.2 1B and 3B (Meta, from the outputs of Llama 3.1 8B and 70B), Gemma 2 2B and 9B (Google), or the six DeepSeek-R1-Distill models (from 1.5 to 70 billion parameters, built on Qwen 2.5 and Llama 3). Distillation has become a common process for producing cost-efficient inference models.
Concrete example
Illustrative case. A 50-employee customer-service SME handles 8,000 conversations per month with an AI assistant. With a flagship model (Claude Sonnet 5.5 or equivalent), monthly inference cost amounts to about 320 euros. By migrating to a distilled model of close quality (for example DeepSeek-R1-Distill-Llama-70B), the cost falls to 35 euros per month, that is, a saving of 3,400 euros per year for a strictly equivalent use case. The assumed quality loss is 3 points on the first-contact resolution rate, to be checked on a sample of real annotated conversations.
FAQ
What is distillation of an AI model?
It is a technique that transfers the knowledge of a large model, called the teacher, to a smaller model, called the student, while preserving most of the performance. It lets you deploy lightweight models that are cheaper to run at inference and viable on more modest infrastructure.
How does the student learn from the teacher?
Instead of being trained directly on the original data, the student model learns to imitate the teacher's outputs, that is, the probabilities it assigns to each possible answer. This so-called soft information carries far more learning signal than the right answers alone. The concept was formalised by Hinton, Vinyals and Dean in 2015.
What does distillation change on the bill?
It can sharply cut inference cost. In an illustrative case, for a customer-service SME handling 8,000 conversations per month, moving from a flagship model (about 320 euros per month) to a distilled model of close quality (about 35 euros per month) represents a saving of 3,400 euros per year. The assumed quality loss is 3 points on the first-contact resolution rate, to be checked on your own conversations.
Are models we know produced by distillation?
Yes. The historical example is DistilBERT, which reaches 97% of BERT's performance while being 40% smaller and 60% faster at inference. Several families of small open models are produced by distillation, such as Llama 3.2 (1B and 3B), Gemma 2 (2B and 9B) and the DeepSeek-R1-Distill models (from 1.5 to 70 billion parameters).
What question should we ask our IT director about this?
"Is the flagship model we use justified, or would a distilled model suffice?" To answer it, compare a distilled model's outputs objectively against your current model on 100 real cases, check whether it can run locally or on a sovereign cloud (which reduces dependence on a foreign provider), and re-evaluate the choice regularly, for example with a quarterly benchmark.
See also
Further reading
Distilling the Knowledge in a Neural Network, Hinton et al., 2015
Sources
- Distilling the Knowledge in a Neural Network, Hinton, Vinyals & Dean, arXiv:1503.02531, 2015. https://arxiv.org/abs/1503.02531
- DistilBERT, a distilled version of BERT, Sanh et al., arXiv:1910.01108, 2019. https://arxiv.org/abs/1910.01108
- Llama 3.2: Revolutionizing edge AI and vision, Meta AI, September 2024 (1B and 3B trained with logits from Llama 3.1 8B and 70B, then distillation). https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/
- Gemma 2: Improving Open Language Models at a Practical Size, Google DeepMind, arXiv:2408.00118, 2024 (2B and 9B trained with distillation). https://arxiv.org/abs/2408.00118
- DeepSeek-R1, model card, DeepSeek-AI, Hugging Face (six dense distilled models from 1.5B to 70B, based on Qwen 2.5 and Llama 3). https://huggingface.co/deepseek-ai/DeepSeek-R1