Every foundation model that ships eventually faces the same structural problem: it has to run continuously, at scale, against unpredictable demand, while someone pays the compute bill. The training narrative dominates capital conversations, but the infrastructure serving inference is where margin is made or destroyed. That gap in attention is, itself, an observation worth sitting with.

The Core Bottleneck Is Memory, Not Compute

Inference workloads are fundamentally memory-bandwidth-constrained, not FLOP-constrained. The weights of a large model must be loaded into GPU VRAM for every forward pass, and the KV cache — the stored attention state for each token in a sequence — grows linearly with context length. A 70B-parameter model running 128K-context requests is largely an exercise in memory management, not raw arithmetic.

This explains why GPU utilization figures from hyperscalers are often misleading. A chip reporting 80% utilization can still be memory-starved, spending most of that time waiting on data transfer rather than executing operations. Continuous batching techniques, pioneered in open-source serving frameworks like vLLM, address part of this by interleaving requests to improve memory throughput — but the ceiling imposed by VRAM capacity remains a hard architectural constraint.

Where the Infrastructure Stack Is Fracturing

The serving layer is not consolidating; it is stratifying. Three distinct infrastructure categories are emerging with meaningfully different economic structures. First, hyperscale API endpoints — OpenAI, Anthropic, Google Vertex — where the operator buys simplicity and absorbs variable pricing risk. Second, dedicated deployment platforms like Together AI, Fireworks, and Baseten, which serve the segment that needs lower latency and more predictable per-token costs than tier-one APIs deliver. Third, on-premises or private cloud deployments using open-weight models, increasingly viable as quantization techniques compress 70B-class models into single-node configurations.

Each tier creates different supplier dependencies and unit economics. The middle tier is particularly structurally interesting: it absorbs the operational complexity of inference optimization — speculative decoding, tensor parallelism tuning, prefill-decode disaggregation — so that product teams do not have to build it internally. That operational abstraction has historically produced durable software businesses.

Hardware Alternatives Are Creating Real Optionality

The GPU monoculture is showing visible cracks. Groq’s LPU architecture demonstrates that purpose-built inference silicon can produce deterministic, low-latency token generation that general-purpose GPU clusters structurally cannot match. Cerebras, with its wafer-scale approach, addresses memory bandwidth differently — the entire model fits on-chip, eliminating the VRAM transfer problem at the cost of a very different deployment footprint.

Neither displaces NVIDIA in the near term. But both reveal that the inference problem is architecturally distinct enough from training to warrant hardware designed specifically for it. Capital flows are acknowledging this: dedicated inference silicon attracted meaningful institutional attention across 2023 and 2024, and the competitive surface area for CUDA’s dominance is narrower in serving workloads than in training ones.

The Operator Read

The investable observation here is not a single company — it is a structural layer. Inference infrastructure sits between commodity compute and application logic, and that position historically produces defensible businesses when switching costs accumulate in the form of optimization work, tuned deployments, and latency SLAs. Operators building on top of this layer are watching for the point where per-token costs compress enough to unlock net-new applications that are currently uneconomic at prevailing prices. That compression curve, not any single model release, is the variable worth tracking.