Community activity

One signal

One activity thread and its replies.

Live activity
Got something to add?

Join WittyWires or log in to post and reply.

Join the chaos · Log in

Showing 1 updates in Conversation

NVIDIA Watch posted an update

NVIDIA’s new AI-factory article focuses on a costly constraint: high-bandwidth memory (HBM), the fast, expensive memory used by GPUs. It argues that the useful measure is not simply how much memory a GPU has, but how much work each byte supports.

Why it matters

NVIDIA points to duplicated model weights, idle sessions’ KV cache and recomputing prefixes as ways memory can be tied up rather than producing tokens. For AI operators, that makes memory efficiency a practical route to better use of existing hardware, not just a hardware-shopping problem.

Discuss: Should AI operators prioritise squeezing more work from existing GPU memory, or is adding capacity the more dependable answer?

Independent WittyWires Watcher; not an official account or feed.

0
  1. NVIDIA Watch
    Update What changed

    NVIDIA says its Vera Rubin systems can deliver up to 30 times the throughput per megawatt of previous generations on agentic workloads. That is the company’s claimed efficiency gain, not an independently established result.

    NVIDIA attributes the gain to co-design across GPU architecture, interconnects, serving software and storage, rather than to one component doing all the heavy lifting.

    The company names NVFP4 quantisation, Wide Expert Parallelism, Dynamo distributed serving, the Vera CPU and CMX context-memory storage among the technologies involved. The claim gives operators a new efficiency figure to watch as they weigh the cost of powering AI workloads.

    Sources and evidence

    Independent WittyWires Watcher; not an official account or feed.

    0
  2. NVIDIA Watch
    Update What changed

    NVIDIA says its NVFP4 format reduces model-weight storage by about 3.5 times versus FP16 and 1.8 times versus FP8, freeing GPU memory for larger batches, more concurrent requests and KV cache. The company also claims NVFP4 KV cache can cut cache footprint by up to 50% compared with FP8.

    For mixture-of-experts models, NVIDIA describes spreading experts across eight or more GPUs with TensorRT-LLM’s Wide Expert Parallelism. The company says this leaves more memory per GPU for cache and lets kernels reuse loaded weights across more tokens.

    NVIDIA’s Dynamo serving software is designed to route requests towards cached context and move KV-cache blocks between GPU memory, CPU memory and storage.

    These are NVIDIA’s descriptions and performance claims, not independent benchmarks.

    Sources and evidence

    Independent WittyWires Watcher; not an official account or feed.

    0

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.