DecisionPivot benchmark probes decision failures hidden by aggregate scores
A new arXiv preprint introduces DecisionPivot, a diagnostic benchmark of 48 tasks and 40,033 items across seven decision roles for models that handle routing, guardrails and output verification in LLM systems. Its core track holds 7,800 items from 13 generated task families arranged in contrast groups with answers verified by independently implemented solvers, while a reference track carries 32,233 items from 35 public test sets under their original metrics. The authors evaluated 40 decision models and 14 general models and report that two 27B models only 2.6 points apart on the reference track reached contrast-group accuracies of 70.1% and 44.2%; Perplexity Decider ranked the positive member above the negative in 92.3% of 2,400 binary pairs but answered both correctly in 74.1%, and GLM-5.3 scored 93.5% on a 1,296-item subset at a higher reasoning setting versus 81.5% at its lowest effort.
The authors argue item accuracy alone cannot establish whether decision models make correct choices across controlled variations or assign reliable confidence, so fine-grained diagnostics are needed for the routing, guardrail and verification roles these models fill.
Computer Science > Artificial Intelligence
arXiv:2610.28522 (cs)
[Submitted on 20 Oct 2026]
Title:DecisionPivot: A Diagnostic Benchmark for Decision Models in LLM Systems
Authors:Xialie Zhuang, Mingcong Lei, Zunhai Su, Ri Liu, Jianjin Li, Peixian Ma, Jia Pan, Ngai Wong, Guanhua Chen, Chuan-Wei Kuo
View a PDF of the paper titled DecisionPivot: A Diagnostic Benchmark for Decision Models in LLM Systems, by Xialie Zhuang and 9 other authors