Distillation: how DeepSeek packed a big model's smarts into small ones

One of the more consequential ideas in modern AI isn't a bigger model — it's a way to shrink one. Knowledge distillation trains a small model to imitate a large one, transferring much of the capability at a fraction of the size and cost. DeepSeek's reasoning models put it in the spotlight.

The idea

A large "teacher" model is expensive to run but very capable. Instead of training a small model from scratch on raw data, you train it — the "student" — to match the teacher's outputs. The teacher's rich outputs carry more signal than a bare label, so the student learns faster and ends up punching well above its size.

What DeepSeek did

DeepSeek trained a strong reasoning model, then used it as a teacher: it generated high-quality reasoning traces, and those traces were used to fine-tune smaller open models (in the Qwen and Llama families). The result was a set of compact models that inherited a large share of the big model's reasoning ability — and could run on far more modest hardware.

• Cheaper to run — a small distilled model serves at a fraction of the cost and latency.
• More accessible — capability that needed a data centre starts fitting on commodity GPUs.
• A quality ceiling — the student rarely beats the teacher; distillation transfers ability, it doesn't create new frontiers.

Distillation is the closest thing AI has to teaching: the expensive expert trains the affordable apprentice, and the apprentice is who shows up to work.