Terminal-Bench 4.0

Headlines · 4

1. 5h
   DeepSeek V4.1 Flash's Terminal-Bench score differs across model card, AA and Vals

   A post on r/LocalLLaMA compares three Terminal-Bench 4.0 scores for DeepSeek-V4.1-Flash-1017: 42.3% reported on the model card, 34% from Artificial Analysis and 31.82% from the Vals index. The poster notes that the model card says the Claude Code harness was used and guesses that AA and Vals instead use mini-swe-agent, which he suggests could explain the lower results; he also points to the benchmark's eight-hour task limit and provider token throughput as possible sources of skew. Taking the 42.3% figure, he says the model would be on par with Fable 5, released June 9, 2026.

   r/LocalLLaMA · 🔥 5
2. 5d
   Terminal-RSI: recursive self-improving terminal agents

   An arXiv preprint introduces Terminal-RSI, which alternates Recursive Self-Rewrite (RSR) of verified trajectories with critic-inheriting Recursive Policy-Refine (RPR), and adds Turn-Boundary Skip-Observation GAE, Family Dense Reward and Success-Prioritized Reward alongside a warm-up critic model, recursive critic handoff and stage-specific learning rates. The authors report that a five-stage run from a Qwen3.8-27B base (RSR-1, RPR-1, RSR-2, RPR-2) yields Terminal-RSI-27B, scoring 19.84% on Terminal-Bench 4.0, 86.52% on Terminal-Bench 2.1, 16.95% on Terminal-Bench 3.0 and 79.00% on Terminal-Bench Hard. On LHTB they report process reward rising from 0.21 to 0.37 and strict completion from 0/46 to 5/46, while SWR100 success doubles from 3.00% to 6.00%; these figures are the paper's own claims and have not been independently verified.

   Hugging Face · Papers · 🔥 0
3. Oct 7
   Anthropic launches Claude Haiku 5.5 at $0.10 per million input tokens

   Anthropic released Claude Haiku 5.5 on October 7, describing it as the fastest, cheapest and most capable model in its Haiku small-model line. For requests up to 100,000 tokens, it costs $0.10 per million input tokens and $0.50 per million output tokens, rising to $0.50 and $2.50 above that threshold; Anthropic says it costs about 75% less to run on average than Haiku 4.5. Anthropic-reported figures show OSWorld 2.1 offline subset rising from 15.7% to 72.4% and GDPval-AA v2.1 at 1620, while Terminal-Bench 4.0 is reported at either 39.2% or 33% across sources, still well below Sonnet 5.5. Haiku 5.5 is the first Haiku model with adjustable effort settings and is positioned as a subagent for Opus 5.5 and Sonnet 5.5; Anthropic also cut Sonnet 5.5 cache-read pricing from $0.20 to $0.10 per million tokens.

   IT之家 AI · 🔥 0
4. Oct 5
   HERMES: modular executable Dev-Primitives for software engineering agents

   An arXiv paper introduces Dev-Primitives, an abstraction that pairs each repository artifact — source files, configs, tests, dependencies — with a resident LLM, giving it an agent-native interface for natural-language reasoning, inter-component communication and localized self-modification. Built on top of it, the HERMES harness-engineering framework activates these primitives at repository scale via dependency-aware dynamic activation and a bug-diagnosis mechanism that maps execution evidence back to the components needing revision. The authors report HERMES beats matched baseline harnesses by 12.4% on average across four software engineering benchmarks, stays within 4.5% of a homogeneous GPT-5.6 Sol configuration even with Qwen3-8B Dev-Primitives, and cuts inference cost by 26.2% on Terminal-Bench 4.0.

   Hugging Face · Papers · 🔥 0

Experience and discussion from the community

Share my Terminal-Bench 4.0 experienceAsk about Terminal-Bench 4.0