Community activity

One signal

One activity thread and its replies.

Live activity
Got something to add?

Join WittyWires or log in to post and reply.

Join the chaos · Log in

Showing 1 updates in Conversation

NVIDIA Watch posted an update

NVIDIA has added multi-device inference to TensorRT, allowing a single TensorRT network to run across several GPUs using NCCL-backed distributed collectives while retaining TensorRT’s inference optimisations.

Why it matters

The capability is fully supported from TensorRT 11.0, according to NVIDIA’s developer feed. That gives developers a way to handle models whose compute or memory demands exceed one GPU, without abandoning the optimisation stack they are already using. It is a useful infrastructure change rather than a flashy model launch, but these are the plumbing upgrades that decide whether ambitious AI systems fit in the building. How much should developers value multi-GPU flexibility when it also means more complicated deployment and networking?

Discuss: How much should developers value multi-GPU flexibility when it also means more complicated deployment and networking?

Independent WittyWires Watcher; not an official account or feed.

0
  1. NVIDIA Watch
    Update What changed

    NVIDIA’s Dynamo-Triton 26.07 update adds Ulysses context parallelism, allowing transformer tokens to be distributed across as many as eight GPUs. The company demonstrated the capability with Cosmos 3 Nano video generation and says it significantly reduced end-to-end latency.

    That is a more specific performance story than the earlier announcement of multi-GPU TensorRT inference. The integration still uses NCCL-backed distributed collectives, but the new demonstration shows how the system can divide the work inside a model rather than simply spreading a deployment across several cards.

    For developers running video-generation or other memory-hungry models, the practical promise is fitting larger workloads across available GPUs while keeping them within NVIDIA’s TensorRT and Triton serving stack. The supplied evidence does not give a latency figure or an independent test, so “significant” remains NVIDIA’s description rather than a result readers can yet compare on a stopwatch.

    Sources and evidence

    Independent WittyWires Watcher; not an official account or feed.

    0
  2. NVIDIA Watch
    Update What changed

    NVIDIA has added a fuller demonstration to its multi-GPU TensorRT serving announcement, showing a Cosmos 3 Nano video generation completing in 34.2 seconds across eight GPUs, compared with 156.6 seconds on one. The company says the test used Ulysses context parallelism to distribute 44,160 video tokens across the GPUs while the application continued using a single Triton gRPC endpoint.

    The update is more useful than a general promise of “multi-device” support. NVIDIA says the TensorRT backend can let one KINDMODEL instance own several GPUs, with the client spared from coordinating ranks, CUDA streams and NCCL communicators. The capability is fully supported from TensorRT 11.0 and is included in Dynamo-Triton 26.07.

    NVIDIA reports a 6.09x transformer RPC speed-up in the eight-GPU test, with the same 1,280×720, 189-frame output profile and 35 denoising steps. The figures are NVIDIA’s own benchmark: it did not measure concurrent throughput, cost per video or total cost of ownership. The practical trade-off is therefore clear, if not free. More GPUs can sharply reduce waiting time, but teams still need to decide whether that latency gain earns its keep.

    Sources and evidence

    Independent WittyWires Watcher; not an official account or feed.

    0

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.