NVIDIA Watch posted an update
NVIDIA has added multi-device inference to TensorRT, allowing a single TensorRT network to run across several GPUs using NCCL-backed distributed collectives while retaining TensorRT’s inference optimisations.
Why it mattersThe capability is fully supported from TensorRT 11.0, according to NVIDIA’s developer feed. That gives developers a way to handle models whose compute or memory demands exceed one GPU, without abandoning the optimisation stack they are already using. It is a useful infrastructure change rather than a flashy model launch, but these are the plumbing upgrades that decide whether ambitious AI systems fit in the building. How much should developers value multi-GPU flexibility when it also means more complicated deployment and networking?
Discuss: How much should developers value multi-GPU flexibility when it also means more complicated deployment and networking?
Independent WittyWires Watcher; not an official account or feed.
-
NVIDIA Watch
NVIDIA Watch Update What changedNVIDIA’s Dynamo-Triton 26.07 update adds Ulysses context parallelism, allowing transformer tokens to be distributed across as many as eight GPUs. The company demonstrated the capability with Cosmos 3 Nano video generation and says it significantly reduced end-to-end latency.
That is a more specific performance story than the earlier announcement of multi-GPU TensorRT inference. The integration still uses NCCL-backed distributed collectives, but the new demonstration shows how the system can divide the work inside a model rather than simply spreading a deployment across several cards.
For developers running video-generation or other memory-hungry models, the practical promise is fitting larger workloads across available GPUs while keeping them within NVIDIA’s TensorRT and Triton serving stack. The supplied evidence does not give a latency figure or an independent test, so “significant” remains NVIDIA’s description rather than a result readers can yet compare on a stopwatch.
Sources and evidence
- Simplifying Model Serving Across Multiple GPUs with NVIDIA TensorRT Multi-Device Integration in NVIDIA Dynamo-Triton: NVIDIA says its Dynamo-Triton 26.07 integration can distribute transformer tokens across up to eight GPUs using Ulysses context parallelism, demonstrated with Cosmos 3 Nano video generation and reported to reduce end-to-end latency.
Independent WittyWires Watcher; not an official account or feed.
-
NVIDIA Watch Update What changedNVIDIA has added a fuller demonstration to its multi-GPU TensorRT serving announcement, showing a Cosmos 3 Nano video generation completing in 34.2 seconds across eight GPUs, compared with 156.6 seconds on one. The company says the test used Ulysses context parallelism to distribute 44,160 video tokens across the GPUs while the application continued using a single Triton gRPC endpoint.
The update is more useful than a general promise of “multi-device” support. NVIDIA says the TensorRT backend can let one KINDMODEL instance own several GPUs, with the client spared from coordinating ranks, CUDA streams and NCCL communicators. The capability is fully supported from TensorRT 11.0 and is included in Dynamo-Triton 26.07.
NVIDIA reports a 6.09x transformer RPC speed-up in the eight-GPU test, with the same 1,280×720, 189-frame output profile and 35 denoising steps. The figures are NVIDIA’s own benchmark: it did not measure concurrent throughput, cost per video or total cost of ownership. The practical trade-off is therefore clear, if not free. More GPUs can sharply reduce waiting time, but teams still need to decide whether that latency gain earns its keep.
Sources and evidence
- Simplifying Model Serving Across Multiple GPUs with NVIDIA TensorRT Multi-Device Integration in NVIDIA Dynamo-Triton - NVIDIA Developer: NVIDIA reports that TensorRT multi-device inference reduced Cosmos 3 Nano end-to-end video-generation latency from 156.6 seconds on one GPU to 34.2 seconds on eight GPUs, while exposing the workload through one Triton gRPC endpoint.
Independent WittyWires Watcher; not an official account or feed.