with how fast AI models are getting bigger and more expensive, I think knowledge distillation is one of the most important ideas to understand.
the basic idea is simple.
a large model, usually called the teacher, is used to train a smaller model, called the student.
the goal is to compress part of the teacher’s capability into a model that runs faster, costs less and uses less memory.
this matters because modern AI models usually improve when you increase training compute, data quality and number of parameters.
larger models tend to be more capable, but they are also expensive to run.
in real products, this becomes a problem very quickly.
a company might be able to train or use a huge model internally, but serving that same model to millions of users can be too slow or too expensive.
knowledge distillation gives companies a way to separate the learning stage from the deployment stage.
the expensive model helps during training.
the smaller model is used later in production.
this way, the student can inherit part of the teacher’s behavior without carrying the full inference cost.
the idea started before modern LLMs.
early machine-learning systems often used ensembles.
an ensemble is a group of many models working together.
for example, to classify an email as spam, you could train hundreds or thousands of small classifiers and average their predictions.
this usually worked better than relying on a single model because each model made slightly different mistakes.
the average prediction became more stable.
the problem was deployment.
running thousands of models on a small device or in a low-latency product was impractical.
so researchers started training one smaller model to imitate the output of the whole ensemble.
after that, the ensemble was no longer needed at inference time.
you kept much of the performance, but reduced the cost.
that was one of the early versions of model compression.
later, Geoffrey Hinton, Oriol Vinyals and Jeff Dean made the modern idea of knowledge distillation much more popular.
the key point was that the student should learn from more than the correct answer.
it should learn from the teacher’s probability distribution.
imagine a model looking at a handwritten digit.
the official label says the image is a 5.
a normal hard label gives the student one answer: 5.
the teacher gives a richer signal.
it may say the image is probably a 5, but it also looks a little like a 3 and maybe slightly like an 8.
that matters because the teacher is showing the student which alternatives are close and which ones are far.
this extra information is often called dark knowledge.
the useful knowledge is hidden in the probabilities assigned to the wrong answers.
a hard label only tells the student the final answer.
a soft label shows uncertainty, similarity and relative confidence.
this makes training easier for the student.
instead of discovering every subtle relationship directly from raw data, the student receives those relationships from a stronger model that already learned them.
the same idea applies to language models.
an LLM predicts the next token.
in normal training, if the correct next token is “Matt”, the target basically says “Matt” is correct.
in distillation, the teacher might assign probability to “Matt”, but also to “John”, “Mark” or another plausible token.
the student learns a broader view of what could make sense in that context.
this is why proper distillation can be more informative than ordinary next-token training.
the student learns from the teacher’s judgment, not just from the one token that appeared in the dataset.
temperature is also important here.
before a model outputs probabilities, it produces raw scores called logits.
softmax turns those logits into probabilities.
temperature changes how sharp or smooth that probability distribution becomes.
with higher temperature, the distribution becomes softer.
less likely alternatives receive more probability.
that helps in distillation because the wrong answers may otherwise have probabilities so tiny that they provide almost no useful signal.
a smoother distribution exposes more of the teacher’s preferences among the alternatives.
in today’s model interfaces, temperature is usually discussed as a creativity setting.
higher temperature gives more variation.
lower temperature gives more predictable outputs.
the math is related, but in distillation the purpose is training signal, not creative generation.
this distinction matters because people now use the word distillation for many different things.
classical knowledge distillation means the student learns from the teacher’s probability distribution.
the teacher gives soft targets over many possible outputs.
that transfers uncertainty, alternatives and relationships between classes or tokens.
another common method is behavioural cloning.
in behavioural cloning, the teacher generates answers and those answers become training data for the student.
the student sees the selected output from the teacher.
it does not receive the full probability distribution behind every token.
this can still work very well.
if a strong teacher writes high-quality reasoning traces, explanations or solutions, a smaller model can learn useful patterns from them.
but the information transferred is different.
behavioural cloning transfers the teacher’s visible output.
proper distillation transfers more of the teacher’s statistical judgment.
this difference becomes important when companies say that a model was distilled from a larger one.
sometimes that means full or partial probability distributions were used.
sometimes it means the larger model generated synthetic data and the smaller model was fine-tuned on those examples.
both methods can produce strong students, but technically they are not the same process.
proper distillation is also harder to do with modern LLMs.
to train from the teacher’s distribution, you usually need access to logits or token probabilities.
a public API often gives you the generated text, not the full distribution across the vocabulary.
so if you only have API access, you can generate synthetic data, imitate outputs and fine-tune on demonstrations.
full distillation usually requires deeper access to the teacher model.
the scale also becomes a problem.
modern vocabularies can have more than 100,000 tokens.
for every token position in a training dataset, the teacher could produce a probability for every token in the vocabulary.
with trillions of training tokens, storing all of those distributions becomes unrealistic.
the compute cost is also large because the teacher has to process the training sequences to produce targets for the student.
so at LLM scale, distillation becomes an infrastructure problem too.
you need teacher inference, storage, data movement, training compute and a pipeline that makes the whole process worth the cost.
because of this, people often use approximate forms of distillation.
instead of keeping the full distribution, they keep only the most relevant part.
for example, the top 100 or top 256 tokens predicted by the teacher.
this creates a sparse version of the teacher’s output.
some information is lost, but the student still gets much more than a single hard label.
another option is computing the teacher probabilities during training and discarding them right after they are used.
that saves storage, but it still needs expensive teacher forward passes.
there is also co-distillation.
in normal distillation, the teacher is trained first, then the student learns from it.
in co-distillation, the teacher and student are trained at the same time.
the teacher learns from the original data, and the student learns from a mix of the original targets and the teacher’s current outputs.
this can be more efficient because the teacher’s output is already being computed during training.
the risk is that early in training the teacher is still immature.
so the student usually needs a mix of hard labels and soft labels, instead of depending only on the teacher’s early predictions.
another related topic is the difference between ensembles and mixture-of-experts models.
an ensemble is made of separately trained models.
each model produces its own prediction, and the predictions are combined.
a mixture-of-experts model is one architecture with multiple expert subnetworks inside it.
a router decides which experts process each token or input.
the experts are trained together as part of the same system.
both approaches combine multiple components, but the structure and training process are very different.
in modern AI, people usually bring up examples like Gemma, Llama and DeepSeek-R1 when talking about distillation.
Google’s Gemma models are often described as smaller models trained with supervision from stronger Gemini models.
Meta has also talked about larger Llama models teaching smaller ones.
DeepSeek-R1 is an interesting case because the smaller models were trained from outputs generated by R1.
that kind of training can be extremely useful, especially when the teacher produces reasoning traces.
the technical question is what exactly the student received.
if it learned from generated answers, that is closer to behavioural cloning or synthetic-data fine-tuning.
if it learned from probability distributions, that is closer to classical distillation.
the result can be good either way, but the mechanism matters.
a smaller model can learn from a larger one because the teacher has already organized the learning problem.
raw data is noisy.
a hard label only gives the final answer.
the teacher gives a richer training signal by showing confidence, plausible alternatives and relationships between outputs.
the student has less capacity than the teacher, so it cannot absorb everything.
but it can still learn more from a strong teacher than from hard labels alone.
that is the real value of distillation.
it can reduce inference latency, lower operating cost and make deployment easier.
a smaller model can run on cheaper servers, edge devices or local environments.
companies can use expensive infrastructure during training and a much cheaper model during production.
this is especially important because not every task needs the largest possible model.
some tasks only need a smaller model that learned enough from a stronger one.
there are also limitations.
a student usually loses some capability compared to the teacher.
teacher mistakes, biases and weaknesses can pass into the student.
sparse distributions reduce cost, but discard part of the dark knowledge.
closed models make full distillation harder because they usually do not expose logits.
and the term distillation is now used very loosely, which creates confusion.
the simplest way I think about it is this.
a model can be large and expensive while learning, then smaller and cheaper when serving users.
the teacher represents the expensive learning stage.
the student represents the practical deployment stage.
that is why distillation will probably keep mattering.
as AI models get larger, companies will keep needing smaller versions that are easier to run.
the large model teaches.
the small model serves.
and the important technical detail is what gets transferred.
synthetic-data training transfers the teacher’s selected answers.
classical distillation transfers a richer probability distribution.
that distribution contains uncertainty, preferences and relationships between possible answers.
and this is why knowledge distillation is more than just making a small model copy a big one.
it is a way to compress useful behavior from a larger system into a smaller one that is cheaper to use in the real world.
Antonioni Nascimento
Software, cybersecurity, data & AI. Building advanced systems at KRX. Writing about technology, venture capital, and how software shapes markets.