Retrieval-Augmented Generation entered the enterprise conversation as the pragmatic alternative to full fine-tuning: cheaper, updatable, auditable. Two years of production deployments later, the picture is more complicated. The architecture works, under specific conditions, and fails in ways that are predictable enough to inform where capital and engineering attention is now concentrating.

Where RAG Is Actually Holding

The clearest wins are narrow-domain, high-document-density applications. Legal contract review, internal knowledge bases with structured metadata, regulated-industry compliance lookups. In these environments, the retrieval layer operates against a bounded, well-maintained corpus, and the generation step is constrained enough that hallucination risk stays manageable. The structural advantage is document freshness: the retrieval index updates independently of the model weights, which matters acutely in contexts where information has a short shelf life.

Customer support pipelines with tiered escalation also show durable performance. When the retrieval corpus is a curated product documentation set, and the generation is scoped to answer-or-escalate, the failure modes are containable. Teams running these systems are reporting meaningful deflection rates without the brittleness of older intent-classification approaches.

Where the Architecture Is Breaking Down

The failure surface is more revealing than the wins. Chunking strategy remains a surprisingly stubborn problem. Most production deployments use fixed-size chunking with cosine similarity retrieval, which performs poorly on multi-hop questions where the answer requires synthesizing evidence across several non-adjacent passages. The retrieved chunks are individually plausible but collectively incomplete, and the model compounds the error downstream.

Context window utilization is the second structural weakness. When retrieval returns ten passages at 512 tokens each, the model’s attention is not uniformly distributed. Research across several labs has documented the “lost in the middle” phenomenon: information positioned in the center of a long context window is retrieved significantly less reliably by the model than information at the edges. Production teams that haven’t audited for this are likely over-reporting retrieval quality.

• Query-document mismatch: user queries are short and colloquial; indexed documents are long and formal. Embedding similarity scores do not adequately bridge this gap without query rewriting layers.
• Latency compounding: a retrieval call, a reranking pass, and a generation call in sequence produce p95 latencies that are incompatible with synchronous user-facing products at scale.
• Evaluation gaps: most teams are measuring retrieval recall against labeled datasets that don’t reflect live query distributions. The benchmark and the production system are solving different problems.

What Next-Generation Implementations Look Like

The more sophisticated production systems have moved away from single-stage retrieval toward modular pipelines. HyDE (Hypothetical Document Embeddings) addresses query-document mismatch by generating a synthetic answer first and embedding that for retrieval. RAPTOR and similar tree-structured indexing approaches tackle multi-hop synthesis by building hierarchical summaries at index time rather than at query time. Neither is a complete solution, but both represent a more honest accounting of where the naive implementation fails.

Graph-augmented retrieval is attracting sustained infrastructure investment. By encoding entity relationships explicitly rather than relying solely on embedding proximity, these systems can handle relational queries that defeat dense-retrieval-only architectures. The operational cost is index complexity and maintenance overhead, which is why uptake is concentrated in organizations with dedicated ML infrastructure teams rather than in the broader mid-market.

The Operator Read

The structural dynamic favoring infrastructure layers over application layers remains intact here. The teams capturing durable value are those building reranking models, evaluation frameworks, and retrieval pipeline tooling rather than those deploying vanilla RAG wrappers on top of foundation model APIs. The application layer compresses; the infrastructure layer where correctness is actually enforced does not. Organizations allocating engineering resources accordingly are positioning into a more defensible surface area.