JevAgentHarness delegates agent tool choices to small decision models
An arXiv paper submitted 15 Oct 2026 by Kunlun Zhu and co-authors introduces JevAgentHarness, which delegates the multiple-choice decisions LLM agents make at each step — such as which tool to call next — to decision models like Jev, falling back to the large LLM when a per-choice confidence threshold fitted on the agent's own trajectories is not met. The authors built AgentDecisionBench and report that the decision model most accurate on isolated questions solved only 69.5% of agent tasks, and that delegating tool choices without a fitted confidence threshold lowered success by 19.0 points. They also trained AgentDecision-4B, an open 4B decision model, on post-mistake agent states labeled with Jev's answers; per the authors, on tasks unlike its training data the agent solved 91.5% with AgentDecision-4B and 82.0% with Jev, while on tasks like the training data such models made agents 81.1–89.0% cheaper and up to 2.5 times faster without losing 3 or more points of task success. The models, data and harness are said to be released.
According to the authors, most harness calls end in a multiple-choice decision, and in their in-distribution tests routing those choices to smaller decision models cut agent cost by 81.1–89.0% and made runs up to 2.5 times faster — pointing at per-step overhead as a target for agent design.
Computer Science > Artificial Intelligence
arXiv:2610.21412 (cs)
[Submitted on 15 Oct 2026]
Title:JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs
Authors:Kunlun Zhu, Tao Feng, Xu Han, Heng Wang, Ling Yang, Xiangru Tang, Jiaxuan You, Heng Ji
View a PDF of the paper titled JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs, by Kunlun Zhu and 7 other authors
#JevAgentHarness#AgentDecisionBench#AgentDecision-4B#Jev#DecisionGym