NVIDIA reports a 10.4x throughput gain for DeepSeek-V3 MoE training on GB200 hardware with JAX and Transformer Engine.

NVIDIA added Transformer Engine support to JAX to speed up dropless Mixture of Experts training. The stack raised DeepSeek-V3 throughput on GB200 hardware from 103 TFLOPS per GPU to 1,068 TFLOPS per GPU, a 10.4x gain. Dropless MoE keeps every token in the training pass, so models avoid the quality loss that comes from dropping or padding tokens.

Mixture of Experts models like DeepSeek, Qwen, and Mixtral use conditional computation to match or exceed dense model quality at lower training compute. A learned router sends each token to Top-K expert networks instead of one shared feed-forward network. NVIDIA built Transformer Engine to accelerate Transformer models on its GPUs, and JAX gives developers a Python path to run those optimized kernels.

Operators gain a practical path to train large MoE models with less wasted compute. Grouped GEMM kernels handle variable expert token counts in one call, and NCCL EP fuses dispatch and combine stages while deduplicating tokens to cut network traffic. JAX host offloading and XLA multistreaming collectives reduce memory bottlenecks and overlap communication across NVLink and InfiniBand.

NVIDIA reported 97% scaling efficiency at 1,024 GPUs on GB300 NVL72 hardware while training DeepSeek-V3 671B. Builders can reproduce the optimized JAX MoE path with the NVIDIA NGC MaxText container and Transformer Engine enabled. The MaxText MoE Configuration guide and Transformer Engine documentation provide setup details for teams that want to test the stack on their own clusters.

What matters

• NVIDIA Transformer Engine in JAX raised DeepSeek-V3 MoE training from 103 to 1,068 TFLOPS per GPU on GB200.
• Dropless MoE processes every token without padding, preserving model quality while cutting training time.
• Watch scaling efficiency at 1,024 GPUs and NCCL EP improvements for larger MoE models on GB300 NVL72 systems.

Why it matters

Watch scaling efficiency at 1,024 GPUs and NCCL EP improvements for larger MoE models on GB300 NVL72 systems.

This GenAI News article was prepared in original wording using reporting and materials published by NVIDIA Developer Blog. Source reference: https://developer.nvidia.com/blog/accelerating-dropless-moe-training-in-jax-with-nvidia-transformer-engine/.

Drafted by the GenAI News review pipeline.