NVIDIA Announces Multi-Device Integration for TensorRT in Dynamo-Triton

NVIDIA has announced the integration of multi-device support for TensorRT in NVIDIA Dynamo-Triton, a significant advancement for AI model serving across multiple GPUs. According to the announcement, this new capability allows a single TensorRT network to execute across multiple GPUs using NCCL-backed distributed collectives, retaining TensorRT's inference optimizations. This feature is fully supported starting with TensorRT 11.0.

NVIDIA Dynamo-Triton release 26.07 introduces the TensorRT backend multi-device capability, enabling one KIND_MODEL instance to own multiple GPUs and expose a single gRPC endpoint for distributed inference. This integration is demonstrated using NVIDIA Cosmos 3 Nano video generation, where Ulysses context parallelism distributes 44,160 video tokens across up to eight GPUs while Diffusers continues to orchestrate prompts, scheduling, and postprocessing.

The integration significantly reduces end-to-end generation latency from 156.6 seconds on one GPU to 34.2 seconds on eight GPUs, with transformer RPC speedup reaching 6.09x. Visual validation confirms that the outputs meet configured quality thresholds.

For organizations deploying generative AI, this new capability closes the gap between multi-GPU acceleration and a consumable inference service. Teams can trade additional GPU resources for shorter request latency, maintain a stable application interface and workflow, and package the engine as a versioned Triton model. This can reduce user wait time and accelerate review-and-refine cycles in latency-sensitive generative media workflows.

To reproduce the results, download NVIDIA Dynamo-Triton 26.07 from NGC and review the Dynamo-Triton TensorRT backend multi-device guide for configuration details.

Source: nvidia-developer