Doc2Harness benchmarks long-horizon harness engineering for coding agents
Yuyao Ge and co-authors have posted Doc2Harness on arXiv, a two-stage benchmark that evaluates long-horizon harness engineering for coding agents. In Stage 1, a build model inside a fixed builder harness receives an interface-only contract skeleton and a development document consolidating behavior across twelve frontier harnesses, then constructs a candidate harness in an isolated container; in Stage 2, a scripted mock model drives the frozen candidate harness end to end in separate containers, with scoring across model requests, files on disk, process exit codes and command output. The authors report that across nineteen frontier models, three independent builds each and 328 test cases, GPT-6 Astra had the highest pass rate at 53.05%, while the weakest model reached 27.44%—a gap of roughly 26 percentage points—and even the strongest model failed nearly half the cases.
The authors argue that agent performance depends as much on the harness around the model as on the model itself, and that existing benchmarks fix the harness and leave harness engineering unexplored; they describe Doc2Harness as the first deterministic benchmark for measuring long-horizon harness engineering.
Computer Science > Software Engineering
arXiv:2610.20344 (cs)
[Submitted on 14 Oct 2026]
Title:Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents
Authors:Yuyao Ge, Shenghua Liu, Peizhong Gao, Chendi Ge, Chenxu Liu, Yiwei Wang, Lingrui Mei, Baolong Bi, Jiayu Yao, Jiafeng Guo, Xueqi Cheng
View a PDF of the paper titled Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents, by Yuyao Ge and 10 other authors