ProAS targets finer credit assignment in agentic reinforcement learning
The ProAS paper proposes a critic-free framework for fine-grained credit assignment in multi-turn LLM agent reinforcement learning. It uses a dynamic Agentic Reference Graph, Masking Unreasonable Incentives and Asymmetric Penalty Redistribution to avoid assigning trajectory-level advantages uniformly across actions. The authors report peak success rates of 91.9% and 94.3% on ALFWorld for Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct, and 66.7% and 70.8% on WebShop, with convergence accelerated by up to 2.44 times.
ProAS offers a concrete way to address uneven action-level credit in multi-turn agent training without adding a critic, while the paper reports stronger benchmark results and faster convergence.
Computer Science > Machine Learning
arXiv:2610.18683 (cs)
[Submitted on 13 Oct 2026 (v1), last revised 25 Oct 2026 (this version, v2)]
Title:ProAS: A Progress-Aware Advantage Shaping Framework for Multi-Turn Agentic Reinforcement Learning
Authors:Bowen Ding, Mingda Wan, Yuhan Chen, Jiyao Yuan, Qi Zhu, Futing Wang, Heyuan Deng, Fei Mi, Lifeng Shang, Tao Lin
View a PDF of the paper titled ProAS: A Progress-Aware Advantage Shaping Framework for Multi-Turn Agentic Reinforcement Learning, by Bowen Ding and 9 other authors
#ProAS#Agentic Reference Graph#Qwen2.5-1.5B-Instruct#Qwen2.5-7B-Instruct#ALFWorld#WebShop