# What is AI model distillation? Definition and business implications

> Distillation is a technique that transfers the knowledge of a large AI model (teacher model) to a smaller model (student model), while preserving most of the performance. It enables the deployment of lightweight models with reduced inference cost, viable on more modest infrastructures.

## Definition
The concept was formalised by Hinton, Vinyals, and Dean in 2015 in the paper *Distilling the Knowledge in a Neural Network*. The principle: instead of training the student model directly on the original data, it is trained to imitate the outputs (probabilities, logits) of the teacher model. The student learns not only the right answers, but also the relative confidence the teacher model assigns to each alternative. This so-called soft information carries far more learning signal than binary labels alone. DistilBERT (Sanh et al., 2019) is the historical example: it achieves 97% of BERT's performance while being 40% smaller and 60% faster at inference. Several families of small open models are officially produced by distillation: Llama 3.2 1B and 3B (Meta, from the outputs of Llama 3.1 8B and 70B), Gemma 2 2B and 9B (Google), or the six DeepSeek-R1-Distill models (from 1.5 to 70 billion parameters, built on Qwen 2.5 and Llama 3). Distillation has become a common process for producing cost-efficient inference models.

## Example
Illustrative case. A 50-employee customer-service SME handles 8,000 conversations per month with an AI assistant. With a flagship model (Claude Sonnet 5.5 or equivalent), monthly inference cost amounts to about 320 euros. By migrating to a distilled model of close quality (for example DeepSeek-R1-Distill-Llama-70B), the cost falls to 35 euros per month, that is, a saving of 3,400 euros per year for a strictly equivalent use case. The assumed quality loss is 3 points on the first-contact resolution rate, to be checked on a sample of real annotated conversations.

## For a leader
“Is the flagship model we are using really justified, or would a distilled model suffice?” This question, asked of your IT director or AI provider, can cut your bill significantly with no perceptible degradation. Three sub-questions to chain. First, have we objectively compared the outputs of a recent distilled model (for example a small open model from the Gemma or Llama families) with our current model on 100 real cases? If quality remains comparable, the flagship surcharge is no longer justified. Second, can this distilled model run locally or on a sovereign cloud? Many can, which also reduces dependence on a foreign provider. Third, how often do we re-evaluate this choice? Distilled models evolve fast; a quarterly benchmark should be systematic.

## FAQ

**What is distillation of an AI model?**
It is a technique that transfers the knowledge of a large model, called the teacher, to a smaller model, called the student, while preserving most of the performance. It lets you deploy lightweight models that are cheaper to run at inference and viable on more modest infrastructure.

**How does the student learn from the teacher?**
Instead of being trained directly on the original data, the student model learns to imitate the teacher's outputs, that is, the probabilities it assigns to each possible answer. This so-called soft information carries far more learning signal than the right answers alone. The concept was formalised by Hinton, Vinyals and Dean in 2015.

**What does distillation change on the bill?**
It can sharply cut inference cost. In an illustrative case, for a customer-service SME handling 8,000 conversations per month, moving from a flagship model (about 320 euros per month) to a distilled model of close quality (about 35 euros per month) represents a saving of 3,400 euros per year. The assumed quality loss is 3 points on the first-contact resolution rate, to be checked on your own conversations.

**Are models we know produced by distillation?**
Yes. The historical example is DistilBERT, which reaches 97% of BERT's performance while being 40% smaller and 60% faster at inference. Several families of small open models are produced by distillation, such as Llama 3.2 (1B and 3B), Gemma 2 (2B and 9B) and the DeepSeek-R1-Distill models (from 1.5 to 70 billion parameters).

**What question should we ask our IT director about this?**
"Is the flagship model we use justified, or would a distilled model suffice?" To answer it, compare a distilled model's outputs objectively against your current model on 100 real cases, check whether it can run locally or on a sovereign cloud (which reduces dependence on a foreign provider), and re-evaluate the choice regularly, for example with a quarterly benchmark.

## See also

- [Training](https://wyp.agency/en/glossary/training/)
- [Parameter](https://wyp.agency/en/glossary/parameter/)
- [Inference](https://wyp.agency/en/glossary/inference/)
- [LLM (Large Language Model)](https://wyp.agency/en/glossary/llm/)
- [Fine-tuning](https://wyp.agency/en/glossary/fine-tuning/)
- [SLM (Small Language Model)](https://wyp.agency/en/glossary/small-language-model/)
- [Frugal AI](https://wyp.agency/en/glossary/frugal-ai/)

## Sources

- Distilling the Knowledge in a Neural Network, Hinton, Vinyals & Dean, arXiv:1503.02531, 2015 (https://arxiv.org/abs/1503.02531)
- DistilBERT, a distilled version of BERT, Sanh et al., arXiv:1910.01108, 2019 (https://arxiv.org/abs/1910.01108)
- Llama 3.2: Revolutionizing edge AI and vision, Meta AI, September 2024 (1B and 3B trained with logits from Llama 3.1 8B and 70B, then distillation) (https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/)
- Gemma 2: Improving Open Language Models at a Practical Size, Google DeepMind, arXiv:2408.00118, 2024 (2B and 9B trained with distillation) (https://arxiv.org/abs/2408.00118)
- DeepSeek-R1, model card, DeepSeek-AI, Hugging Face (six dense distilled models from 1.5B to 70B, based on Qwen 2.5 and Llama 3) (https://huggingface.co/deepseek-ai/DeepSeek-R1)

---
Entry from the WYP.agency AI glossary · https://wyp.agency/en/glossary/distillation/ · Last updated : 2026-10-14