ProgramBench

Headlines · 4

1. 5d
   Import AI 478 spotlights research automation, coding agents and AI security

   Import AI 478 reviews several AI developments, led by Resolution’s agent-driven autoformalization of mathematical statements connected to statistical learning theory. Resolution says the project took about a week, consumed roughly 100 billion input tokens and 300 million output tokens, cost an estimated $135,000, and produced about 600,000 lines of Lean. The issue also reports Claude Opus 5.5 reaching 18.5% on ProgramBench, Mirendil-Base being trained by one researcher in 3.5 months with less than $10 million in compute, and OpenAI describing agents that communicated through processor-cache timing.

   Import AI · 🔥 0
2. Oct 19
   Study finds dynamic concurrency in coding agents helps long tasks but hurts short ones

   Researchers from Nanjing University and UIUC ran paired experiments on the dynamic concurrency mechanisms of three coding agents — Codex, Claude Code and Kimi Code — across 354 software engineering tasks and 2,124 executions, analysing full main-agent and sub-agent traces. Results varied by task type and system: on LoopsBench, Claude Code's completion rate rose from 7.1% sequentially to 21.4% with concurrency, and Codex rose from 34.6% to 57.7% on NL2Repo; on SWE-bench Verified, Claude Code fell from 83.0% to 59.0% and Kimi Code from 77.0% to 59.0%, while Codex stayed at 71.0% in both modes. Trace analysis produced a taxonomy of 4 categories, 13 subcategories and 28 concurrency failure modes (such as missing owners, pseudo-concurrency, load imbalance, mistimed result merging and shared-environment conflicts), plus three mechanisms where concurrency paid off: complementary work, parallel exploration and independent verification.

   新智元 · 🔥 0
3. Oct 18
   HarnessPro recursively evolves agent harnesses with Pareto cost-performance tuning

   The paper introduces HarnessPro, a label-free method that evolves AI agent harnesses through recursive self-improvement, aiming to improve task performance while reducing execution cost. Candidate harnesses are evaluated using LLM-generated tests without access to ground-truth solutions or official tests; specialized agents diagnose execution trajectories, plan updates with different performance-cost priorities and implement them, and normalized weighted squared-distance selection yields cost-focused (lite), balanced (medium) and performance-focused (heavy) harnesses. The authors report that on Terminal-Bench 2, SWE-Bench Pro and ProgramBench, HarnessPro's heavy mode with GPT-5.5 and Codex achieved an average pass-rate gain of 2.0 percentage points and a 26.1% average per-task cost reduction relative to RHO, and say the advantage holds with GLM-5.2 and Claude Code; code is available and the 41-page paper is under review.

   Hugging Face · Papers · 🔥 0
4. Oct 11
   Study: AI agent teams cost 1.8–5.1x more for little quality gain

   Evals company Vals AI tested GPT-6 Sol and Claude Opus 5.5 on its Vibe Code Bench both as single agents and as agent teams, at medium and maximum reasoning effort. Teams cost between 1.8x and 5.1x more than single agents, yet only one of four comparisons showed a statistically significant improvement: GPT-6 Sol at medium reasoning, where the team scored 7.3 points higher; at maximum reasoning, neither Sol's nor Opus 5.5's team gained a real advantage. The report also notes that in two of Anthropic's own tests with Opus 5.5, quality gains shrank as more agents were added, while OpenAI researcher Noam Brown has said multi-agent systems mainly buy speed rather than better quality.

   The Decoder · 🔥 0

Experience and discussion from the community

Share my ProgramBench experienceAsk about ProgramBench