等算力下密集过程奖励未必胜过GRPO
一项于2026年11月10日提交至arXiv的研究预印本提出按等算力协议比较密集过程奖励与GRPO,并同时按生成token数和训练GPU时长设定预算。论文称,在六个模型共1,668次运行中,等算力比较会改变结果排序:在Qwen2.5-1.5B上,Monte-Carlo奖励在等token数时比标准GRPO低14.3个百分点,在等训练时长时低7.8个百分点。论文还报告,训练预测成功率最接近1/2的提示可提升准确率。
该预印本表明,忽略token成本可能夸大密集过程奖励的优势,因此推理模型训练评估需要关注等算力比较。
Computer Science > Computation and Language
arXiv:2611.13560 (cs)
[Submitted on 10 Nov 2026]
Title:Are Dense Rewards Worth Their Tokens in GRPO?
Authors:Vikash Singh, Weicong Chen, Debargha Ganguly, Yanyan Zhang, Mohsen Hariri, Sreehari Sankar, Wang Yang, Chaoda Song, Alexander Nemecek, Osama Zafar, Yuting Shao, Kai Ye, Biyao Zhang, Xinpeng Li, Shouren Wang, Erman Ayday, Jing Ma, Sanmukh Rao Kuppannagari, Vipin Chaudhary
View a PDF of the paper titled Are Dense Rewards Worth Their Tokens in GRPO?, by Vikash Singh and 18 other authors
#GRPO#Qwen2.5-1.5B#Qwen3-0.6B#Monte-Carlo process reward#Reinforcement Learning with Verifiable Rewards