REINFORCE Zero aligns kernels for stable asynchronous RL
vLLM Blog reports that NVIDIA Research and Inferact developed REINFORCE Zero, a method that aligns rollout and training kernels so unclipped importance sampling can correct policy weight lag in asynchronous reinforcement learning. According to the researchers, the system trains stably for 20,000 updates, learns at policy lag 64, and reaches 2.40 times the training-step throughput of synchronous training with aligned kernels. The report also says the paper, training code, zero-mismatch recipes, and model-specific fixes have been released.
The work addresses a concrete obstacle to faster asynchronous RL training by separating manageable policy weight lag from numerical kernel mismatch, with paper, code, recipes, and performance measurements available for inspection.
TL;DR
- Asynchronous RL is faster, but it trains on samples generated by older weights.
- The rollout-to-learner gap has two parts. Weight lag changes smoothly and unclipped importance sampling (IS) controls it. Kernel mismatch, where rollout and training compute different numbers for the same weights, fluctuates and persists despite IS.
- Align the kernels, and unclipped IS corrects the remaining weight lag without sample-rejection gates. REINFORCE Zero (NVIDIA Research and Inferact) trains stably for 20,000 updates, learns at policy lag 64, and reaches 2.40× the training-step throughput of synchronous training with aligned kernels.
- Zero mismatch now covers DeepSeek-V4.1-Flash (DAPO rollout step within 10% of the fast baseline), Kimi K3.1 on day 0, and both DeepSeek-V4-Flash and Kimi K3.1 on AMD MI355X.
- Everything is in Inferact/zero-mismatch-rl.
1. From synchronous to asynchronous
Last month, we showed that rollout and training can agree exactly. On DeepSeek-V4-Flash, vLLM and the trainer produced the same logprob for every sampled token, bit for bit, on the fast kernels, with an 8.7% reduction in rollout throughput. An agent skill now does the alignment for each new model.
#REINFORCE Zero#NVIDIA Research#Inferact#vLLM#DeepSeek-V4.1-Flash#DeepSeek-V4-Flash#Kimi K3.1#Qwen3-4B#Qwen3-30B-A3B#Nemotron-3-Nano-Omni-30B