SWE-bench Verified

Headlines · 7

1. 2d
   Google Researchers Open-Source EnvHarness for Adaptive Agent Training

   Google researchers have open-sourced EnvHarness, a framework that adapts training environments to an agent’s weaknesses without modifying the agent, underlying environment, or evaluation system. It wraps existing environments through standard reset() and step() interfaces and includes EnvRigger to identify weaknesses and generate targeted changes. The researchers reported gains of 2.9–3.7 percentage points across WebArena, SWE-bench Verified, and ALFWorld for several tested models.

   InfoQ 中文 AI&大模型 · 🔥 3
2. 3d
   LMSYS trains Qwen3 coding agent with asynchronous RL on AMD MI355X

   LMSYS Blog reports that its team trained Qwen3-30B-A3B-Instruct for coding-agent tasks using fully asynchronous agentic reinforcement learning on AMD Instinct MI355X, with Miles, SGLang, Megatron-LM, and Harbor. On a vendor-run, 100-task stratified subset of SWE-bench Verified, pass@1 rose from 7.3% to 27.5% after supervised fine-tuning and agentic RL. The system used separate inference and training nodes, capped trajectories at 64K tokens, and moved sandboxes to E2B cloud microVMs, supporting 384 concurrent trajectories and about 3,480 trajectories per hour.

   LMSYS Blog · 🔥 0
3. Oct 24
   Coding-Agent Harness Changes Raise Costs Without Reliable Gains

   Shandong University researchers Liu and Han compared Claude Code, OpenCode, and mini-SWE-agent while holding the language model constant. Most performance differences were similar to rerun noise, while the harnesses consumed two to three times as many tokens as the leanest option; OpenCode showed a significant deficit on the larger task pool. A small cross-check on Terminal-Bench 2.1 showed the same pattern.

   The Decoder · 🔥 0
4. Oct 23
   ModelAnalysis.si launches LLM cost, benchmark, and VRAM comparison tool

   Developer Zack Ku announced ModelAnalysis.si, a free dashboard for comparing LLM benchmarks, API token pricing, VRAM requirements, and self-hosting economics. The tool supports side-by-side and radar-chart comparisons, API spend simulation, cloud-GPU break-even calculations, and deployment commands for Ollama and vLLM. According to Ku, the site draws on benchmark and pricing data covering more than 250 models, while a later post says it covers over 280 models. These coverage and accuracy claims have not been independently verified.

   r/OpenAI · 🔥 0
5. Oct 19
   HarnessAdapt-RL trains coding agents by adapting their harness during RL

   A new arXiv preprint by Lingxiao Tang, Wei Tao, Yiwen Guo and Lingfeng Bao introduces HarnessAdapt-RL, a reinforcement-learning method that trains a coding agent's policy while also adapting its harness. A frozen teacher generates task-specific mechanism-level skills for flawed repair strategies, while a separate teacher model evolves tools and hooks, keeping each change only if it improves the observed pass rate or observed trainable rate without lowering either. With a 9B base model, the authors report pass@1 gains over GRPO of 9.8, 12.6 and 6.4 percentage points on SWE-bench Verified, Multilingual and Pro, and say it outperforms GRPO with the human-designed OpenHands harness; the results are self-reported and not independently verified.

   Hugging Face · Papers · 🔥 0
6. Oct 19
   Study finds dynamic concurrency in coding agents helps long tasks but hurts short ones

   Researchers from Nanjing University and UIUC ran paired experiments on the dynamic concurrency mechanisms of three coding agents — Codex, Claude Code and Kimi Code — across 354 software engineering tasks and 2,124 executions, analysing full main-agent and sub-agent traces. Results varied by task type and system: on LoopsBench, Claude Code's completion rate rose from 7.1% sequentially to 21.4% with concurrency, and Codex rose from 34.6% to 57.7% on NL2Repo; on SWE-bench Verified, Claude Code fell from 83.0% to 59.0% and Kimi Code from 77.0% to 59.0%, while Codex stayed at 71.0% in both modes. Trace analysis produced a taxonomy of 4 categories, 13 subcategories and 28 concurrency failure modes (such as missing owners, pseudo-concurrency, load imbalance, mistimed result merging and shared-environment conflicts), plus three mechanisms where concurrency paid off: complementary work, parallel exploration and independent verification.

   新智元 · 🔥 0
7. Oct 8
   Inside the AI data boom: from labeling to RL environments

   In this Silicon Valley 101 podcast episode, He Yunzhong, who works on post-training and evaluation research at Scale AI, and Sun Yiyou, a UC Berkeley postdoc on the Agents’ Last Exam (ALE) project, unpack the fast-growing but opaque market for AI training data. They note AfterQuery was valued at $300 million in its April A round and, per media reports, $3.2 billion by September, while Bespoke Labs announced $40 million in combined seed and A funding in July, and trace the shift from crowd labeling and pre-training toward rubrics, verifiable signals and RL environments bundling tasks, tools and verification. They also discuss procurement bottlenecks in vertical domains, the blurred line between benchmarks and data sales — including a claim that some data firms sold evaluation data — expert fabrication and contamination, and OpenAI's February decision to stop reporting SWE-bench Verified scores. Guest views are personal.

   36氪 人工智能 · 🔥 0

Experience and discussion from the community

Share my SWE-bench Verified experienceAsk about SWE-bench Verified