创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

REINFORCE Zero用内核对齐稳定异步RL

AI 总结

vLLM Blog报道,NVIDIA Research与Inferact提出REINFORCE Zero,通过对齐rollout与训练内核,让未裁剪重要性采样能够修正异步强化学习中的策略权重滞后。研究方称,该方法可稳定训练2万次更新,在策略滞后64时仍能学习,并达到同步训练步吞吐量的2.40倍。报道还表示,论文、训练代码、零失配配方及针对不同模型的修复均已发布。

深度见解

这项工作把异步RL中的策略权重滞后与数值内核失配区分开来,并提供了可供核查的论文、代码、配置与性能测量。

TL;DR

  • Asynchronous RL is faster, but it trains on samples generated by older weights.
  • The rollout-to-learner gap has two parts. Weight lag changes smoothly and unclipped importance sampling (IS) controls it. Kernel mismatch, where rollout and training compute different numbers for the same weights, fluctuates and persists despite IS.
  • Align the kernels, and unclipped IS corrects the remaining weight lag without sample-rejection gates. REINFORCE Zero (NVIDIA Research and Inferact) trains stably for 20,000 updates, learns at policy lag 64, and reaches 2.40× the training-step throughput of synchronous training with aligned kernels.
  • Zero mismatch now covers DeepSeek-V4.1-Flash (DAPO rollout step within 10% of the fast baseline), Kimi K3.1 on day 0, and both DeepSeek-V4-Flash and Kimi K3.1 on AMD MI355X.
  • Everything is in Inferact/zero-mismatch-rl.

1. From synchronous to asynchronous

Last month, we showed that rollout and training can agree exactly. On DeepSeek-V4-Flash, vLLM and the trainer produced the same logprob for every sampled token, bit for bit, on the fast kernels, with an 8.7% reduction in rollout throughput. An agent skill now does the alignment for each new model.

#REINFORCE Zero#NVIDIA Research#Inferact#vLLM#DeepSeek-V4.1-Flash#DeepSeek-V4-Flash#Kimi K3.1#Qwen3-4B#Qwen3-30B-A3B#Nemotron-3-Nano-Omni-30B

原文约 9 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。