The economics of running large language models in production look very different from the economics of training them. Distillation sits at that fault line. A well-executed distillation pipeline can compress a frontier model’s capability into a fraction of the parameter count, cutting per-token inference costs by an order of magnitude. The catch is that “well-executed” carries more weight than most infrastructure discussions acknowledge.

What Distillation Actually Does to the Cost Stack

Inference cost scales roughly with parameter count and sequence length, not with the original training bill. When a 70B teacher model is distilled into a 7B student, the operator is trading peak capability headroom for a predictable reduction in GPU-hours per query. At high request volumes, that compression changes the unit economics materially. A deployment running 50 million tokens per day on a 70B model and shifting to a well-tuned 7B distillate can move from GPU-bound infrastructure to a configuration that fits within reserved cloud capacity at significantly lower effective cost-per-token.

The mechanism matters here. Knowledge distillation transfers soft probability distributions from teacher to student during training, not just hard labels. This is why distilled models often outperform models of identical size trained from scratch on the same task distribution. The student learns the teacher’s uncertainty structure, which generalizes better than pure supervised signal on a narrow dataset.

Where Production Performance Diverges from Benchmark Claims

The gap between distillation benchmarks and production behavior opens in three specific places. First, out-of-distribution prompts. A distilled model trained on a curated task distribution degrades faster than its teacher when user inputs drift outside that distribution. Second, multi-step reasoning chains. Chain-of-thought capability compresses poorly relative to single-turn factual recall. Operators running agentic workflows or complex document synthesis find the student model’s reasoning paths collapse on problems requiring five or more logical dependencies. Third, instruction-following consistency at the edges. Subtle formatting requirements, conditional logic in system prompts, and structured output fidelity all show higher failure rates in compressed models under real traffic.

This is not an argument against distillation. It is an argument for honest capability mapping before committing a distillate to a production path where degradation is expensive to catch after deployment.

The Practical Limits and Where Investment Is Concentrated

The current research frontier on distillation is focused on speculative decoding, layer-wise transfer, and task-specific distillation over general-purpose compression. Task-specific distillation, in particular, is showing durable production results because it narrows the capability surface intentionally. An operator distilling a 70B model specifically for medical coding classification is not asking the student to replicate general intelligence. They are asking it to replicate one slice of the teacher’s behavior reliably and cheaply, which is a solvable problem with current tooling.

• Task-specific distillates with narrow scope outperform generalist compressions in production reliability metrics.
• Speculative decoding architectures, where a small draft model proposes tokens and a larger model verifies them, offer a hybrid path that avoids the capability ceiling of pure distillation.
• Quantization applied post-distillation compounds the cost reduction but compounds the edge-case degradation risk in equal measure.

The Operator Read

The structural observation for capital allocators and infrastructure operators is this: distillation is not a general solution to AI inference cost. It is a scoped solution. The organizations extracting durable efficiency gains are the ones running tightly defined task distributions against distillates built specifically for those tasks, with monitoring in place to catch distributional drift before it becomes a quality problem. The market for managed distillation tooling and task-specific fine-tuning services is structurally early relative to the scale of the inference cost problem operators are trying to solve.

Retrieval-Augmented Generation entered the enterprise conversation as the pragmatic alternative to full fine-tuning: cheaper, updatable, auditable. Two years of production deployments later, the picture is more complicated. The architecture works, under specific conditions, and fails in ways that are predictable enough to inform where capital and engineering attention is now concentrating.

Where RAG Is Actually Holding

The clearest wins are narrow-domain, high-document-density applications. Legal contract review, internal knowledge bases with structured metadata, regulated-industry compliance lookups. In these environments, the retrieval layer operates against a bounded, well-maintained corpus, and the generation step is constrained enough that hallucination risk stays manageable. The structural advantage is document freshness: the retrieval index updates independently of the model weights, which matters acutely in contexts where information has a short shelf life.

Customer support pipelines with tiered escalation also show durable performance. When the retrieval corpus is a curated product documentation set, and the generation is scoped to answer-or-escalate, the failure modes are containable. Teams running these systems are reporting meaningful deflection rates without the brittleness of older intent-classification approaches.

Where the Architecture Is Breaking Down

The failure surface is more revealing than the wins. Chunking strategy remains a surprisingly stubborn problem. Most production deployments use fixed-size chunking with cosine similarity retrieval, which performs poorly on multi-hop questions where the answer requires synthesizing evidence across several non-adjacent passages. The retrieved chunks are individually plausible but collectively incomplete, and the model compounds the error downstream.

Context window utilization is the second structural weakness. When retrieval returns ten passages at 512 tokens each, the model’s attention is not uniformly distributed. Research across several labs has documented the “lost in the middle” phenomenon: information positioned in the center of a long context window is retrieved significantly less reliably by the model than information at the edges. Production teams that haven’t audited for this are likely over-reporting retrieval quality.

• Query-document mismatch: user queries are short and colloquial; indexed documents are long and formal. Embedding similarity scores do not adequately bridge this gap without query rewriting layers.
• Latency compounding: a retrieval call, a reranking pass, and a generation call in sequence produce p95 latencies that are incompatible with synchronous user-facing products at scale.
• Evaluation gaps: most teams are measuring retrieval recall against labeled datasets that don’t reflect live query distributions. The benchmark and the production system are solving different problems.

What Next-Generation Implementations Look Like

The more sophisticated production systems have moved away from single-stage retrieval toward modular pipelines. HyDE (Hypothetical Document Embeddings) addresses query-document mismatch by generating a synthetic answer first and embedding that for retrieval. RAPTOR and similar tree-structured indexing approaches tackle multi-hop synthesis by building hierarchical summaries at index time rather than at query time. Neither is a complete solution, but both represent a more honest accounting of where the naive implementation fails.

Graph-augmented retrieval is attracting sustained infrastructure investment. By encoding entity relationships explicitly rather than relying solely on embedding proximity, these systems can handle relational queries that defeat dense-retrieval-only architectures. The operational cost is index complexity and maintenance overhead, which is why uptake is concentrated in organizations with dedicated ML infrastructure teams rather than in the broader mid-market.

The Operator Read

The structural dynamic favoring infrastructure layers over application layers remains intact here. The teams capturing durable value are those building reranking models, evaluation frameworks, and retrieval pipeline tooling rather than those deploying vanilla RAG wrappers on top of foundation model APIs. The application layer compresses; the infrastructure layer where correctness is actually enforced does not. Organizations allocating engineering resources accordingly are positioning into a more defensible surface area.