Blog/What is model distillation?
Definition

What is model distillation?

By Samuel Seidel · Published September 9, 2026

Model distillation trains a smaller "student" model to reproduce the behavior of a larger "teacher" model, instead of learning from raw data alone. The goal is a compact model that keeps as much of the teacher's capability as possible while needing far less memory and compute, which pays off at inference time, when the model is deployed and queried over and over.

How the teacher-student setup works

In the classic form, the teacher processes training examples and produces a full probability distribution over possible next tokens, which carries more information than a single correct label. The student is trained to match that distribution, alongside or instead of the raw ground-truth labels.

The teacher's distribution encodes relationships the raw data does not state, such as which wrong answers are "less wrong" than others. That lets the student learn a compressed version of the teacher's behavior more efficiently than a small model trained from scratch on the same data.

A more common version in current practice

A simpler variant, sometimes called data distillation, skips the probability distributions. The teacher generates large volumes of example outputs (question-answer pairs, reasoning traces, completions), and those become the student's training data. Most publicly discussed small open-weight models are built this way. It is less precise than matching full distributions but far simpler, because it needs only the teacher's outputs, not its internal probabilities.

What distillation trades away

A distilled student is not a compressed copy of the teacher. It is a separate model that approximates the teacher's behavior on the data it saw, and its capability depends on how much of that training signal there was and how good it was. Distilled models do well on tasks close to what they imitated and can fall short where the distillation data was thin. Test a distilled model on your own workload before trusting its parameter count or its teacher's reputation. See evaluating an LLM before deploying it for how to run that check.

Distillation vs quantization vs fine-tuning

The three come up together but do different things. Quantization keeps a model's architecture and lowers the numeric precision of its existing weights. Fine-tuning continues training an existing model on new data to specialize it. Distillation produces a new, usually smaller model that approximates a larger one's general behavior. They combine: a distilled model can be quantized afterward to shrink it further, and fine-tuned for a specific task.

Why it matters for self-hosting

A distilled model with a fraction of its teacher's parameters is often the difference between fitting comfortably in one node's memory and not fitting at all. A GPUwerk Spark has 128 GB of unified memory, enough to run many distilled and mid-sized open-weight models at higher precision than a much larger model would allow on the same machine. See what is a parameter count for how size maps to memory needs, and the hardware page for full specifications.

Related pages

First top-up: pay $20, get $30 in credit

Run a right-sized model on a dedicated node.

One dedicated Spark, $0.79/hour, deployed in minutes in Europe, North America or East Asia.

Create account