LoLBench Tests Coding Agents on Large-Scale Feature Delivery
The LoLBench paper introduces a benchmark for evaluating coding agents that must turn community feature proposals into cross-module implementations in large, evolving repositories. Its 100 tasks cover 29 systems, eight languages and five domains, with repositories averaging about 2.4 million lines of code. The paper reports that the best of 28 model-and-scaffold combinations resolved 14% of tasks, while all combinations had a median resolution rate of 2%.
The benchmark shifts coding-agent evaluation from isolated bug fixes or code generation toward end-to-end feature delivery, exposing gaps in repository understanding and coordinated implementation.
新智元 2026-11-04 18:03 北京
新智元报道
如今Agent已经能熟练调用工具、修改文件、运行测试，但面对一个长期演进的大型软件系统，它能否从一份新需求出发，交付完整功能？
产品经理提出新需求，开发者拿到的往往不是文件清单或函数签名。需要先理解约束，追踪模块依赖，再判断改哪里、怎么改、怎样证明改对了。这些分析影响后续交付。
复旦大学、香港城市大学、香港中文大学等机构提出LoLBench，把Agent放进平均约240万行源代码的软件系统，评估从功能提案到模块交付的完整过程，检验把需求落到系统中的感知能力，以及完成跨模块修改的实现能力。
#LoLBench#Claude Opus 5#Claude Code#mini-SWE-agent#GPT-5.6 Sol#Codex#DeepSeek V4 Flash#OpenCode#Fudan University#City University of Hong Kong#Chinese University of Hong Kong