TPPO: Tracking-Value PPO tops standard PPO for LM reasoning
An arXiv preprint introduces Tracking-Value PPO (TPPO), which analyzes how successive prefix values relate and shows they admit a Bayesian decomposition with additive updates in log-odds space; the method uses a cumulative value parameterization that sums position-wise scalar outputs into value logits, learned from terminal-outcome supervision. The authors report aggregate mathematical reasoning gains over PPO of 7.0 and 8.4 points on Qwen3-4B-Base and Qwen3-8B-Base respectively, along with improved out-of-distribution generalization. They say gains also extend to Qwen3-4B, Qwen2.5-Math-1.5B and Qwen3.5-9B, plus countdown and scientific coding tasks, with code released publicly.
If reproducible, this kind of critic value-tracking change could give reasoning RL under sparse terminal rewards more reliable advantage estimates.