The premise is clean: take a large frontier model, use its outputs to train a smaller one, and serve the smaller one at a fraction of the compute cost. In practice, the arithmetic holds — but only inside a well-defined problem envelope. Outside that envelope, the savings evaporate and the quality gap becomes visible at the worst possible moment.

Where the Cost Compression Actually Comes From

Inference cost scales roughly with parameter count and sequence length. A 7B-parameter model running on a single A100 handles roughly 10–20x more requests per second than a 70B model on the same hardware, and the memory footprint difference is substantial enough to change your deployment topology entirely — multi-GPU orchestration versus a single-node setup. Distillation compresses the teacher’s learned distributions into fewer parameters, which means the student can hit comparable accuracy on narrow, well-scoped tasks without the full parameter budget.

The key structural word is narrow. Distillation transfers task-specific competence efficiently. It does not transfer generalized reasoning at the same fidelity. A student model trained on a customer support classification corpus will perform competitively on that corpus. Ask it to handle edge cases outside the distribution and the brittleness surfaces quickly.

What Actually Works in Production

The use cases where distilled models demonstrate durable production viability share a common trait: the output space is constrained. Structured extraction, intent classification, sentiment scoring, code completion within a narrow syntax domain — these tasks allow the student to learn a distribution that is genuinely approximable at lower capacity. Teams running these workloads at scale are observing meaningful reductions in per-query cost, often in the range of 60–80% versus the equivalent frontier model call.

• Synthetic data quality is the ceiling. The student learns from the teacher’s outputs, which means any systematic error or bias in the teacher’s generations propagates. Distillation does not clean noisy labels — it amplifies them at scale.
• Task boundary definition is upstream of everything. Operators who invest in precise task scoping before distillation consistently see better retention of the teacher’s quality. Vague task boundaries produce vague students.
• Evaluation infrastructure matters more than model selection. Without a benchmark suite that stress-tests distribution edges, production failures look like random model degradation rather than the predictable capability boundary they actually represent.

Where the Limits Are Structural, Not Temporary

Multi-step reasoning chains remain genuinely difficult to distill at high fidelity. Chain-of-thought behavior in large models emerges from depth and breadth of parameter interaction; compressing that into a 7B architecture does not preserve the reasoning structure, it approximates the final outputs. The distinction matters in domains where the reasoning process — not just the conclusion — needs to be auditable or reliable under adversarial input.

There is also a maintenance cost that is frequently underweighted in initial economics: student models require re-distillation when the task distribution shifts. A frontier model accessed via API absorbs provider-side updates passively. A distilled model deployed on-premises does not. For companies with rapidly evolving product surfaces, that retraining cycle has real engineering overhead that partially offsets the inference savings.

The Operator Read

The structural setup favors distillation most clearly for operators running high-volume, narrow-scope inference workloads where the task definition is stable. The cost compression is real and repeatable in that zone. The discipline required is resisting the temptation to expand the student model’s responsibilities beyond what the training distribution supports — that is where the economics quietly reverse. Operators who draw that boundary deliberately, and maintain rigorous evaluation coverage of it, are finding distillation to be a durable infrastructure position rather than a temporary arbitrage.