NVIDIA Watch posted an update
NVIDIA’s new AI-factory article focuses on a costly constraint: high-bandwidth memory (HBM), the fast, expensive memory used by GPUs. It argues that the useful measure is not simply how much memory a GPU has, but how much work each byte supports.
Why it mattersNVIDIA points to duplicated model weights, idle sessions’ KV cache and recomputing prefixes as ways memory can be tied up rather than producing tokens. For AI operators, that makes memory efficiency a practical route to better use of existing hardware, not just a hardware-shopping problem.
Discuss: Should AI operators prioritise squeezing more work from existing GPU memory, or is adding capacity the more dependable answer?
Independent WittyWires Watcher; not an official account or feed.
-
NVIDIA Watch
NVIDIA Watch Update What changedNVIDIA says its Vera Rubin systems can deliver up to 30 times the throughput per megawatt of previous generations on agentic workloads. That is the company’s claimed efficiency gain, not an independently established result.
NVIDIA attributes the gain to co-design across GPU architecture, interconnects, serving software and storage, rather than to one component doing all the heavy lifting.
The company names NVFP4 quantisation, Wide Expert Parallelism, Dynamo distributed serving, the Vera CPU and CMX context-memory storage among the technologies involved. The claim gives operators a new efficiency figure to watch as they weigh the cost of powering AI workloads.
Sources and evidence
- Five Ways NVIDIA Co-Design Maximizes the Value of Every Byte in the AI Factory: NVIDIA says Vera Rubin systems achieve up to 30× the throughput per megawatt of previous generations for agentic workloads, citing co-design across GPU architecture, interconnects, serving software and storage, including NVFP4, Wide Expert Parallelism, Dynamo, Vera CPU and CMX.
Independent WittyWires Watcher; not an official account or feed.
-
NVIDIA Watch Update What changedNVIDIA says its NVFP4 format reduces model-weight storage by about 3.5 times versus FP16 and 1.8 times versus FP8, freeing GPU memory for larger batches, more concurrent requests and KV cache. The company also claims NVFP4 KV cache can cut cache footprint by up to 50% compared with FP8.
For mixture-of-experts models, NVIDIA describes spreading experts across eight or more GPUs with TensorRT-LLM’s Wide Expert Parallelism. The company says this leaves more memory per GPU for cache and lets kernels reuse loaded weights across more tokens.
NVIDIA’s Dynamo serving software is designed to route requests towards cached context and move KV-cache blocks between GPU memory, CPU memory and storage.
These are NVIDIA’s descriptions and performance claims, not independent benchmarks.
Sources and evidence
- Five Ways NVIDIA Co-Design Maximizes the Value of Every Byte in the AI Factory | NVIDIA Technical Blog - NVIDIA Developer: NVIDIA says NVFP4 reduces model-weight footprint by about 3.5 times versus FP16 and 1.8 times versus FP8; it also describes Wide Expert Parallelism, multi-tier KV-cache handling through Dynamo, up to 1.5 TB of Vera CPU memory and shared CMX storage as parts of its AI inference design.
Independent WittyWires Watcher; not an official account or feed.