REINFORCE Zero用内核对齐稳定异步RL
AI 总结
vLLM Blog报道,NVIDIA Research与Inferact提出REINFORCE Zero,通过对齐rollout与训练内核,让未裁剪重要性采样能够修正异步强化学习中的策略权重滞后。研究方称,该方法可稳定训练2万次更新,在策略滞后64时仍能学习,并达到同步训练步吞吐量的2.40倍。报道还表示,论文、训练代码、零失配配方及针对不同模型的修复均已发布。
深度见解
这项工作把异步RL中的策略权重滞后与数值内核失配区分开来,并提供了可供核查的论文、代码、配置与性能测量。
TL;DR
- Asynchronous RL is faster, but it trains on samples generated by older weights.
- The rollout-to-learner gap has two parts. Weight lag changes smoothly and unclipped importance sampling (IS) controls it. Kernel mismatch, where rollout and training compute different numbers for the same weights, fluctuates and persists despite IS.
- Align the kernels, and unclipped IS corrects the remaining weight lag without sample-rejection gates. REINFORCE Zero (NVIDIA Research and Inferact) trains stably for 20,000 updates, learns at policy lag 64, and reaches 2.40× the training-step throughput of synchronous training with aligned kernels.
- Zero mismatch now covers DeepSeek-V4.1-Flash (DAPO rollout step within 10% of the fast baseline), Kimi K3.1 on day 0, and both DeepSeek-V4-Flash and Kimi K3.1 on AMD MI355X.
- Everything is in Inferact/zero-mismatch-rl.
1. From synchronous to asynchronous
Last month, we showed that rollout and training can agree exactly. On DeepSeek-V4-Flash, vLLM and the trainer produced the same logprob for every sampled token, bit for bit, on the fast kernels, with an 8.7% reduction in rollout throughput. An agent skill now does the alignment for each new model.
#REINFORCE Zero#NVIDIA Research#Inferact#vLLM#DeepSeek-V4.1-Flash#DeepSeek-V4-Flash#Kimi K3.1#Qwen3-4B#Qwen3-30B-A3B#Nemotron-3-Nano-Omni-30B