HICO-Bench scores social agents on verifiable plans, not just fluent chat
The paper introduces HICO-Bench, a benchmark of 180 everyday coordination scenarios such as resource allocation, scheduling and authorization, in which two agents holding private information converse and then independently record the plan they agreed on. A program checks Feasibility (the two records match and satisfy both participants' private conditions), while an LLM judge checks Integrity (the conversation supports the recorded plan); passing both counts as Verified Success. Across thirteen LLMs, the authors report that 36% of conversations rated highly for goal completion fail Verified Success. The paper also proposes Paired-Credit Policy Optimization (PaCPO), which asks the agent for a tentative plan after each turn and checks it against both versions of the scenario; the authors report it raises Qwen3-8B's Verified Success from 20.4% to 28.3%.
The authors argue that existing social-agent evaluations mainly rely on LLM judges that rate the conversation without checking the resulting plan, which this benchmark aims to address with programmatically verifiable outcomes.
Computer Science > Computation and Language
arXiv:2610.20195 (cs)
[Submitted on 14 Oct 2026]
Title:What Makes a Social Agent's Interactions Effective? Learning from Verifiable Outcomes in Paired Worlds
Authors:Caixin Kang, Ke Wang, Yuchuan Wu, Xiaoqiang Wang, Ling Li, Bingxue Qiu, Yongbin Li, Yoichi Sato
View a PDF of the paper titled What Makes a Social Agent's Interactions Effective? Learning from Verifiable Outcomes in Paired Worlds, by Caixin Kang and 7 other authors