AI’s inference boom turns memory into the chip industry’s main event

The AI industry has quietly changed what it does for a living: less building models, more running them. That shift from training to inference is now forcing a rethink of the chips and memory everything else depends on, and it is redrawing who buys silicon from whom. The sharpest datapoint in the new landscape: researchers found Nvidia H100 GPUs running open-source models sit idle 50 to 80 per cent of the time, starved by memory that cannot feed them fast enough. The world's most coveted accelerator spends a good slice of its working life as a very expensive footrest.

Watch Desk analysis

What happened

Training has receded to background noise. "It's like training is yesterday's news," Matt Kimball, principal analyst at Moor Insights & Strategy, told IEEE Spectrum. "All that any chief information officer wants to talk about is inference." Jensen Huang called GTC 2026 the "inflection point of inference". The demand side explains why: reasoning models re-prompt themselves in chains of thought, with high-effort settings producing up to 20 times as much text as low-effort ones, and agentic systems now run inference around the clock rather than only when somebody types a query.

The scramble has produced alliances nobody would have drawn up two years ago, as the full feature sets out. OpenAI and Amazon have deployed Cerebras's dinner-plate wafer-scale chips, even though Amazon builds its own Trainium silicon; AWS splits inference in two, with Trainium on the compute-heavy half and Cerebras on the memory-heavy half. Nvidia paid a controversial $20bn for the talent and intellectual property of inference startup Groq, and the resulting Groq 3 language-processing unit arranges on-chip SRAM in the exact order computation needs it. Anthropic, meanwhile, is paying LLM rival SpaceXAI more than a billion dollars a month to lease spare compute.

Underneath the dealmaking sits a technical squeeze. Generating each token of a reply means reading the whole model from memory, plus a KV cache of prompts and prior output that can swell to dozens of gigabytes. Memory bandwidth, not raw compute, sets the pace, which is why those H100s idle and why the fix-it money is flowing towards memory.

Why it matters

For anyone buying or building AI services, this changes what to shop for. If inference performance is set by how fast data moves rather than how many operations a chip can perform, peak FLOPs are the wrong price tag, and the idle-GPU finding suggests plenty of money is spent on compute that mostly waits. Shahriar Rabii, formerly head of silicon engineering at Meta and now cofounder of Majestic Labs, says the GPU approach leaves you "greatly over-provisioning compute and starved on memory", which is precisely what is driving the memory scale-out.

The strategic map has been redrawn too. Amazon owns a chip ambition of its own and still buys Cerebras. Anthropic rents from a rival. Nvidia spent $20bn on a startup whose ideas now sit inside its own LPU. When inference is the product, hardware choices stop being procurement detail and become strategy, and the fight between cheap DRAM and better HBM will set the cost curve every AI bill of the next few years rides on.

Our read

Everyone agrees on the diagnosis and nobody agrees on the cure. d-Matrix shortens the wire, Majestic lengthens it, and the HBM camp, Samsung and SK Hynix included, bets better memory arrives on schedule. Commodity DRAM at a third of the price is a serious wedge, and the upstarts need only a slice of the market to matter. If you run inference workloads, price the plumbing: memory bandwidth and cost per terabyte moved will tell you more than a peak-performance slide. Keep a little scepticism for rack-scale claims until they ship with numbers attached.

What to watch

• Whether HBM4 ships with Nvidia's Vera Rubin in the second half of 2026 and actually breaks the memory bottleneck in deployed systems.
• Whether d-Matrix's Raptor and Majestic Labs' long-reach racks move from claims to customers with published performance numbers.
• Whether the odd alliances hold: OpenAI and Amazon on Cerebras silicon, Anthropic renting compute from SpaceXAI at over a billion dollars a month.
• What Nvidia builds on Groq's technology beyond the Groq 3 LPU, after a $20bn deal the feature itself calls controversial.

Discussion spark: If the bottleneck is memory rather than compute, should AI buyers still pay a premium for raw FLOPs, or is the smart money on whoever moves data cheapest?

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.