Matched Compute Challenges Dense Process Rewards Versus GRPO
A research preprint submitted to arXiv on 10 November 2026 proposes a matched-compute protocol that compares dense process rewards with GRPO using generated-token and GPU-hour budgets. Across 1,668 runs on six models, the authors report that the ranking reverses under matched compute: on Qwen2.5-1.5B, a Monte-Carlo reward is 14.3 percentage points lower than standard GRPO at equal token usage and 7.8 points lower at equal training time. The paper also reports gains from training on prompts whose predicted success is closest to 1/2.
The preprint suggests that dense process rewards can appear stronger when token costs are ignored, making matched-compute evaluation important for reasoning-model training research.
Computer Science > Computation and Language
arXiv:2611.13560 (cs)
[Submitted on 10 Nov 2026]
Title:Are Dense Rewards Worth Their Tokens in GRPO?
Authors:Vikash Singh, Weicong Chen, Debargha Ganguly, Yanyan Zhang, Mohsen Hariri, Sreehari Sankar, Wang Yang, Chaoda Song, Alexander Nemecek, Osama Zafar, Yuting Shao, Kai Ye, Biyao Zhang, Xinpeng Li, Shouren Wang, Erman Ayday, Jing Ma, Sanmukh Rao Kuppannagari, Vipin Chaudhary
View a PDF of the paper titled Are Dense Rewards Worth Their Tokens in GRPO?, by Vikash Singh and 18 other authors
#GRPO#Qwen2.5-1.5B#Qwen3-0.6B#Monte-Carlo process reward#Reinforcement Learning with Verifiable Rewards