NVIDIA开源Reshard加速RL权重重构
NVIDIA Technical Blog介绍了Reshard,这是一项可在任意DTensor源布局与目标布局之间传输张量的集合通信操作。开源参考实现NCCL M2N Reshard可将权重从M个训练GPU传输到N个推理GPU,并通过NVIDIA的nccl-extensions仓库提供。NVIDIA称,其NeMo RL实现在DGX H100和DGX GB200系统的基准测试中,较gather-and-broadcast方法加速2.7至5.7倍。
这一开源实现针对分离式强化学习后训练中的通信瓶颈,解决更新后的模型权重需要在不同分片方式的训练与推理GPU之间迁移的问题。
During reinforcement learning (RL) post-training, training trajectories are generated on the fly by running inference with the current policy model. For training efficiency, it is common to run inference and training in parallel on separate GPU resources in a disaggregated setup. Although the policy model is updated only on the training GPUs, the system must periodically refresh the weights on the inference GPUs with the latest parameters generated on the training GPUs. This transfer of model weights between training and inference GPUs is called refit.
Refit involves two defining characteristics that pose challenges. First, inference and training have different computational requirements, so the weights are distributed differently across inference and training instances by having different parallel mappings (data, context, tensor, expert, and pipeline parallelism).
Second, because refit must update all model weights on the inference GPUs, the communication payload is on the order of the total size of the model’s weights. This payload is already very large for the latest frontier models and is expected to continue growing. Together, these characteristics mean that refit requires high-performance communication across a wide variety of distributed source and destination layouts.
#NVIDIA#NCCL M2N Reshard#NeMo RL#NCCL#Qwen3.8 2.4T#Qwen 3 235B#DeepSeek v3 671B#Nemotron Ultra v3#NVIDIA DGX H100#NVIDIA DGX GB200