LLM model distillation explained: teacher-student training and model compression

What distillation is (and is not)

At its core, distillation trains a compact model to reproduce the behavior of a more capable one. Geoffrey Hinton's original formulation used soft labels: instead of training only on hard class targets (0 or 1), the student learns from the teacher's full probability distribution over the vocabulary. A teacher that assigns 40% probability to a plausible but wrong token and 55% to the correct one conveys dark knowledge — relative similarities between outputs that a one-hot label discards.

For modern LLMs, distillation rarely means matching logits token-by-token on every forward pass (teacher inference is too expensive at scale). More often it means:

• Response distillation — the teacher generates completions; the student is fine-tuned on those text outputs (supervised fine-tuning on synthetic data).
• Logit / sequence distillation — the student minimizes KL divergence against teacher softmax outputs on a curated corpus (common in research and some on-prem pipelines).
• Task-specific distillation — distill only one capability (tool calling, JSON extraction, code completion) rather than general chat.

Distillation is not the same as pruning (removing weights), quantization (lowering numeric precision), or simply training a small model from scratch on public data. Each technique addresses a different bottleneck; the best results often stack them: distill first, then quantize for inference.

Why teams distill LLMs

The economic case is straightforward. API calls to GPT-4-class models at volume dominate inference budgets. A 7B or 8B student that handles 80% of routine queries locally — routing only hard cases to the teacher — can cut cost by an order of magnitude. Latency drops too: fewer parameters mean less memory bandwidth per token, and a smaller KV cache footprint enables longer contexts on the same GPU.

Privacy and compliance push distillation toward the edge. Hospitals, banks, and device manufacturers often cannot send user prompts to third-party APIs. A distilled on-prem model inherits teacher-like behavior without exposing data upstream. See edge AI and on-device inference for how distilled plus quantized models land on phones and laptops.

Product teams also distill to specialize. A general teacher fine-tuned on your support docs, then distilled into a 3B classifier-plus-responder, beats a generic small model trained only on open web text — because the teacher already encoded domain reasoning the student copies.

The synthetic data pipeline (most common in practice)

The dominant production pattern today is a three-stage loop:

1. Prompt design and teacher generation

Curate prompts that mirror real user traffic: support tickets, coding tasks, multi-turn dialogues, edge cases. Run them through the teacher with consistent system prompts and decoding settings (temperature, top-p). Diversity matters — if every synthetic example looks alike, the student overfits to teacher stylistic tics instead of underlying reasoning.

2. Filtering and quality control

Not every teacher output is gold. Filter with heuristics (length, format checks, refusal detection), secondary model graders, human spot audits, and deduplication (MinHash or embedding similarity). Bad synthetic data is worse than no data: the student learns confident nonsense. Reject truncated outputs, hallucinated citations, and answers that fail executable tests for code tasks.

3. Student fine-tuning

Fine-tune the student with standard causal LM loss on (prompt, teacher response) pairs — often with LoRA or full fine-tune depending on budget. Some pipelines mix synthetic teacher data with a slice of original human-labeled data to preserve alignment and reduce reward-hacking on benchmarks. Learning rate, epoch count, and catastrophic forgetting of base capabilities need careful tuning; too many epochs on narrow synthetic sets collapses general chat quality.

Soft labels, temperature, and KL divergence

When you have budget to run teacher forward passes during student training, logits distillation adds a KL-divergence term between teacher and student output distributions. Temperature scaling controls how "soft" the teacher distribution is: dividing logits by T > 1 spreads probability mass, exposing secondary plausible tokens the student should consider.

A typical combined loss looks like:

L = α · KL(σ(zt/T) || σ(zs/T)) · T² + (1 − α) · CE(y, zs)

where zt and zs are teacher and student logits, σ is softmax, CE is cross-entropy against hard labels if present, and α balances the two terms. The T² factor compensates for gradient shrinkage at high temperature. In practice α is tuned empirically — too much KL and the student mimics teacher uncertainty on tasks where crisp answers are needed; too little and you lose the dark-knowledge benefit.

Full-vocabulary KL on every token is memory-heavy for large vocabs (128k+ tokens). Teams sometimes distill on truncated top-k teacher logits or sequence-level losses on final answer spans only.

Instruction distillation and chain-of-thought

Instruction-following models add another layer: the teacher must produce not just answers but well-formatted, safe, tool-aware responses. Instruction distillation feeds the student multi-turn chats where the teacher demonstrates reasoning traces (chain-of-thought), JSON tool calls, or structured markdown. The student learns format and reasoning style jointly.

Chain-of-thought distillation is double-edged. Including teacher reasoning steps in training data can boost student performance on math and logic benchmarks — but if reasoning is stripped at inference ("think step by step" removed), quality may collapse. Decide explicitly whether the student should show its work or internalize reasoning silently. Some teams train two heads: one verbose teacher trace for training, one concise output mode for production.

Alignment behaviors (refusals, tone, policy compliance) transfer imperfectly. Students sometimes over-refuse (copying teacher caution on benign prompts) or under-refuse (missing subtle jailbreak patterns the teacher handled). Plan a red-team pass on the student independent of teacher scores.

Distillation vs fine-tuning vs quantization

Order of operations usually runs: pick student architecture (often smaller transformer with fewer layers and narrower hidden size) → distill or SFT on teacher data → evaluate → quantize for deployment. Quantizing before distillation can degrade teacher signal if you are doing logit matching; quantizing after is standard for serving.

Evaluation traps and failure modes

Distilled models look great on benchmarks until real traffic arrives. Watch for:

Benchmark inflation from teacher leakage

If synthetic training prompts overlap with public eval sets (MMLU, GSM8K, HumanEval), scores overstate generalization. Hold out task families and use private eval suites built from production logs.

Style mimicry without reasoning

Students learn fluent teacher phrasing ("Certainly! Let's break this down...") while reasoning depth regresses. Test with perturbed questions, adversarial paraphrases, and multi-hop problems absent from training.

Capability cliffs by size

Below a certain parameter count, some skills do not compress — long-context retrieval, rare-language fluency, nuanced tool selection. Map which queries must still escalate to the teacher and instrument routing accuracy.

Distribution drift

User behavior shifts; teacher data goes stale. Schedule periodic re-distillation or continuous small-batch synthetic refresh. Version student models in production the same way you version embedding indexes.

Production checklist

• Define the student architecture and non-negotiable latency / cost targets before generating data.
• Sample prompts from real traffic (anonymized); cover long tail and failure cases, not just happy paths.
• Fix teacher decoding settings and document them; changing temperature mid-pipeline shifts the label distribution.
• Filter synthetic outputs with automated graders plus human audit on a random slice.
• Hold out entire task categories for evaluation; never train and test on the same prompt templates.
• Compare student vs teacher on private evals, not just public leaderboards.
• Red-team refusals, jailbreaks, and PII leakage on the student independently.
• Quantize after distillation; re-benchmark quality at target precision (INT8 / INT4).
• Implement teacher fallback routing for queries below confidence thresholds.
• Log student–teacher disagreement rates in production to trigger re-distillation.

Key takeaways

• Distillation transfers teacher capability to a smaller student via soft logits, synthetic responses, or task-specific pipelines.
• The dominant production path is teacher-generated synthetic data plus filtered student fine-tuning — not live logit matching on every token.
• Temperature-scaled KL divergence preserves dark knowledge when logits distillation is feasible.
• Stack distillation with quantization for cost and edge deployment; they solve different bottlenecks.
• Evaluate on held-out private tasks and monitor routing — benchmark scores alone lie.

Related reading

• LLM fine-tuning explained — SFT, LoRA, and when to fine-tune vs distill
• LLM quantization and inference explained — INT8/INT4, GPTQ, and serving after distillation
• LLM KV cache explained — why smaller students use less memory at decode time
• Edge AI and on-device inference explained — deploying distilled models on devices