DecisionPivot

Headlines · 2

1. Oct 19
   DecisionPivot benchmark probes decision failures hidden by aggregate scores

   A new arXiv preprint introduces DecisionPivot, a diagnostic benchmark of 48 tasks and 40,033 items across seven decision roles for models that handle routing, guardrails and output verification in LLM systems. Its core track holds 7,800 items from 13 generated task families arranged in contrast groups with answers verified by independently implemented solvers, while a reference track carries 32,233 items from 35 public test sets under their original metrics. The authors evaluated 40 decision models and 14 general models and report that two 27B models only 2.6 points apart on the reference track reached contrast-group accuracies of 70.1% and 44.2%; Perplexity Decider ranked the positive member above the negative in 92.3% of 2,400 binary pairs but answered both correctly in 74.1%, and GLM-5.3 scored 93.5% on a 1,296-item subset at a higher reasoning setting versus 81.5% at its lowest effort.

   Hugging Face · Papers · 🔥 0
2. Oct 18
   StartLux-Decision-v1.1 tops Hugging Face decision-model leaderboard

   Reports from 36Kr, TMTPost and InfoQ Chinese say StartLux's StartLux-Decision-v1.1-27B Instant ranked first on Hugging Face's updated Decision Index 0.3.2 with 69.1 points, which InfoQ gives as 69.12; the reports differ on whether 173 or 174 entries were listed. The new generation offers Instant, which returns option probabilities in a single pass, and AutoThink, which generates extra reasoning only when initial confidence is insufficient; the team also introduced the RLCD² training method and the DecisionPivot benchmark, and says 27B AutoThink scored 73.37 on its own Decision Index 0.3 evaluation versus 57.96 for hosted Jev. Model weights and inference code are published, but the weights use a CC BY-NC 4.0 license limited to non-commercial use, with commercial deployment requiring separate authorization.

   InfoQ 中文 AI&大模型 · 🔥 0

Experience and discussion from the community

Share my DecisionPivot experienceAsk about DecisionPivot