AI inference is splitting into four jobs, and the data centre must keep up

AI inference is no longer one neat burst of computation. A new SemiAnalysis article explains how modern models divide serving into distinct stages, each placing different demands on chips, memory, networking and software. The practical consequence is that faster AI may depend as much on moving data efficiently as on buying more raw computing power.

Watch Desk analysis

What happened

The article examines inference for mixture-of-experts and modern transformer models, separating it into prefill, midfill, decode attention and decode experts. Those stages do different work and therefore put different pressure on compute, memory bandwidth and networking.

SemiAnalysis also connects the model architecture to the machinery around it: orchestration layers, inference servers, caching strategies and the physical layout of hardware nodes. In other words, the AI system is not just a model sitting on a chip. It is a token factory, and the conveyor belts matter.

Why it matters

This helps explain why headline accelerator specifications do not tell the whole performance story. A system can have ample compute but still lose time and money shifting model weights, cached context and intermediate results between processors and memory.

The distinction is especially relevant for mixture-of-experts models, which activate only parts of a larger network for each token. That can reduce some computation, but it also makes routing, memory placement and interconnect design more important. The bottleneck has not vanished. It has acquired a more complicated job title.

Our read

SemiAnalysis offers a useful map of the problem rather than a single shopping list. The important takeaway for engineers and buyers is to match infrastructure to the workload stage: prefill, where large inputs are processed, may favour different resources from decode, where responses are generated token by token.

That is a more useful frame than asking which accelerator is “fastest” in the abstract. Before spending heavily, teams should measure the workload they actually run, including context length, batching, cache behaviour and network traffic. AI performance is increasingly a systems question, not a trophy cabinet for impressive FLOPs.

What to watch

• Whether inference platforms expose clear measurements for each serving stage rather than one blended throughput figure.
• How caching and memory movement affect the cost of long-context and agent workloads.
• Whether specialised hardware wins in production once orchestration and networking overheads are included.
• How inference servers adapt as mixture-of-experts models become more common. The useful argument here is not whether compute still matters. It plainly does. It is whether the industry has spent too long treating data movement as plumbing when it is becoming part of the product.

Discussion spark: If data movement is becoming the main constraint in AI inference, should buyers prioritise memory and networking design over headline accelerator performance?

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.