Huawei says its new OceanStor M900 system can extend AI models’ key-value cache beyond GPU and system memory, pooling it across a cluster. The pitch is aimed at a real infrastructure pinch point: keeping long-running inference supplied with the context it needs without relying only on scarce, expensive accelerator memory.
Watch Desk analysis
What happened
Huawei introduced OceanStor M900 at HUAWEI CONNECT 2026 in Shanghai. The Register’s sponsored article describes a dedicated context-memory layer using the company’s UnifiedBus interconnect to share KV-cache resources across a cluster. Huawei says the system can scale to 64 petabytes and gives NPUs a single-hop route to stored cache data.
The company claims access latency of 60 microseconds, with the potential to double token throughput and halve time to first token in typical AI coding workloads. It also says its cache-aware storage can reach endurance of up to 24 drive writes per day. Those are Huawei’s performance and durability claims, not independently tested results in the report.
Why it matters
KV caches hold information generated during inference so a model can reuse earlier computations. As conversations, reasoning tasks and agent workflows grow longer, those caches can expand; GPU and system memory are fast but capacity-constrained, while conventional storage may add latency. Huawei’s proposal is to put a shared tier between those options, making storage part of the inference design rather than a cupboard the compute team remembers later.
If the claimed latency and capacity hold up in real deployments, the system could let operators retain more context without keeping all of it in expensive accelerator memory. That would matter most for workloads where context size and access speed affect throughput or response time. The article is sponsored by Huawei, so the numbers deserve testing rather than a ceremonial round of applause.
Our read
This is a substantive AI-infrastructure announcement, not just a new storage box with “AI” attached: Huawei is proposing a distinct shared tier for inference context and cache. The useful question is whether that architecture delivers its claimed speed and capacity under actual workloads, and at what cost. Operators should treat the figures as a vendor proposition until independent deployments put them through their paces.
What to watch
- Independent measurements of latency, throughput and time to first token.
- Whether the claimed 64PB capacity and 24-drive-writes-per-day endurance appear in deployed configurations.
- Which AI operators adopt the system, and how it compares with keeping KV cache in accelerator or system memory.
Discussion spark: Would you trust a shared storage tier with inference context if it cut costs or improved throughput, or is keeping that data close to accelerator memory worth the premium?
Sources and evidence
- AI inference is running into a memory bottleneck. Huawei wants to tackle it with OceanStor M900 (19 October 2026, 08:00 UTC)
Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.