One KIND_MODEL instance can now own multiple GPUs and serve distributed inference through a single gRPC endpoint.
NVIDIA shipped TensorRT multi-device inference in TensorRT 11.0, and Dynamo-Triton release 26.07 enables it in the TensorRT backend. A single Triton KIND_MODEL instance now owns multiple GPUs, builds per-rank execution contexts and NCCL communicators, and launches all ranks for each request. Applications call one named model over a single gRPC endpoint.
The capability targets generative AI workloads whose compute and memory needs outgrow one GPU. TensorRT keeps its inference optimizations while NCCL-backed distributed collectives move data across devices. The team demonstrated the integration with NVIDIA Cosmos 3 Nano, a video generation model, using Ulysses context parallelism with Diffusers handling prompts, scheduling, and postprocessing.
For operators, the integration moves multi-GPU coordination out of client code and into a versioned Triton model. Teams can add GPU resources to cut request latency, keep the application interface and surrounding workflow stable, and stop managing rank and communicator lifecycles themselves. Shorter generation times reduce user wait and speed review-and-refine cycles for latency-sensitive media work.
NVIDIA reports end-to-end generation latency falling from 156.6 seconds on one GPU to 34.2 seconds on eight GPUs, with transformer RPC speedup at 6.09x and visual validation passing configured quality thresholds. Builders can reproduce the results by pulling Dynamo-Triton 26.07 from NGC and reviewing the TensorRT backend multi-device guide along with the TensorRT multi-device documentation.
What matters
- TensorRT 11.0 lets a single network execute across multiple GPUs with NCCL-backed collectives.
- Builders trade extra GPUs for lower latency while keeping the client interface unchanged.
- NVIDIA Cosmos 3 Nano demonstrates a drop from 156.6 seconds to 34.2 seconds on eight GPUs.
Why it matters
NVIDIA Cosmos 3 Nano demonstrates a drop from 156.6 seconds to 34.2 seconds on eight GPUs.
This GenAI News article was prepared in original wording using reporting and materials published by NVIDIA Developer Blog. Source reference: https://developer.nvidia.com/blog/simplifying-model-serving-across-multiple-gpus-with-nvidia-tensorrt-multi-device-integration-in-nvidia-dynamo-triton/.
Drafted by the GenAI News review pipeline.