SWE-Bench Pro

Headlines · 4

1. 12h
   BHI framework evaluates the health of 158 LLM benchmarks

   Researchers affiliated with Alibaba Data and Xiaotian Hengyu proposed the Benchmark Health Index (BHI) to assess whether LLM evaluation benchmarks remain useful as models improve. Based on technical reports for 117 models, the framework evaluates 158 benchmarks across capability discrimination, saturation resistance, and influence. The paper was accepted as an Oral presentation for the NeurIPS 2026 Evaluations and Datasets Track.

   机器之心 · 🔥 7
2. 6d
   Nanbeige releases Nanbeige4.3-33B open-weight teacher model

   Nanbeige released Nanbeige4.3-33B under the Apache 2.0 license, positioning it as a teacher model for smaller agents, including a planned Nanbeige4.3-3B. The model card lists a 512K context window, while Nanbeige reported 80.9 on Terminal-Bench 2.1 versus 73.0 for Qwen3.8-27B under the same settings. The results were not independently replicated, and the model scored 60.3 versus Qwen3.8-27B’s 61.7 on SWE-Bench Pro.

   r/MachineLearning · 🔥 2
3. Oct 18
   HarnessPro recursively evolves agent harnesses with Pareto cost-performance tuning

   The paper introduces HarnessPro, a label-free method that evolves AI agent harnesses through recursive self-improvement, aiming to improve task performance while reducing execution cost. Candidate harnesses are evaluated using LLM-generated tests without access to ground-truth solutions or official tests; specialized agents diagnose execution trajectories, plan updates with different performance-cost priorities and implement them, and normalized weighted squared-distance selection yields cost-focused (lite), balanced (medium) and performance-focused (heavy) harnesses. The authors report that on Terminal-Bench 2, SWE-Bench Pro and ProgramBench, HarnessPro's heavy mode with GPT-5.5 and Codex achieved an average pass-rate gain of 2.0 percentage points and a 26.1% average per-task cost reduction relative to RHO, and say the advantage holds with GLM-5.2 and Claude Code; code is available and the 41-page paper is under review.

   Hugging Face · Papers · 🔥 0
4. Oct 18
   Taste-Bench: AI agents pick the better long-horizon path only ~60% of the time

   Researchers from Microsoft and City University of Hong Kong built Taste-Bench, which tests an agent's "taste" in long-running tasks by freezing a run at a "decision fork," hiding later work and asking a model to choose between two diverging paths. The benchmark has 502 binary questions drawn from thousands of scored agent runs on coding and AI research tasks, including SWE-bench Pro and METR's MALT transcripts. Across 14 models, GPT-5.6 Sol led with 59.7% and GPT-5.5 followed at 59.5%, while average accuracy fell to 21%, near random, when the deciding clue appeared only after extensive later work. The team also distilled a teacher's judgment into a Qwen3.6-27B student, lifting accuracy on unseen tasks from 30% to 47.9% and raising success on 41 held-out SWE-bench Pro tasks from 14.6% to 33.7% when its advice was added to the task context. The benchmark and code are publicly available.

   The Decoder · 🔥 0

Experience and discussion from the community

Share my SWE-Bench Pro experienceAsk about SWE-Bench Pro