Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

NVIDIA Open-Sources Reshard for Faster RL Weight Refit

AI summary

NVIDIA Technical Blog introduced Reshard, a collective communication operation for transferring tensors between arbitrary DTensor source and destination layouts. An open-source reference implementation, NCCL M2N Reshard, transfers weights from M training GPUs to N inference GPUs and is available through NVIDIA's nccl-extensions repository. NVIDIA reports that its NeMo RL implementation achieved 2.7x to 5.7x speedups over gather-and-broadcast across benchmarks on DGX H100 and DGX GB200 systems.

In-depth insight

The open-source implementation targets a communication bottleneck in disaggregated reinforcement learning post-training, where updated model weights must move between differently sharded training and inference GPUs.

During reinforcement learning (RL) post-training, training trajectories are generated on the fly by running inference with the current policy model. For training efficiency, it is common to run inference and training in parallel on separate GPU resources in a disaggregated setup. Although the policy model is updated only on the training GPUs, the system must periodically refresh the weights on the inference GPUs with the latest parameters generated on the training GPUs. This transfer of model weights between training and inference GPUs is called refit.

Refit involves two defining characteristics that pose challenges. First, inference and training have different computational requirements, so the weights are distributed differently across inference and training instances by having different parallel mappings (data, context, tensor, expert, and pipeline parallelism). 

Second, because refit must update all model weights on the inference GPUs, the communication payload is on the order of the total size of the model’s weights. This payload is already very large for the latest frontier models and is expected to continue growing. Together, these characteristics mean that refit requires high-performance communication across a wide variety of distributed source and destination layouts. 

#NVIDIA#NCCL M2N Reshard#NeMo RL#NCCL#Qwen3.8 2.4T#Qwen 3 235B#DeepSeek v3 671B#Nemotron Ultra v3#NVIDIA DGX H100#NVIDIA DGX GB200

Original · 9 min readRead the original
Group chat

No comments yet. Start the conversation.