AutomationBench
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 6
- Gear open-source framework targets recursive AI self-improvement
Gear developers introduced an open-source, meta-learning-inspired framework in which a task agent produces execution traces and a meta-agent uses those traces and evaluation feedback to improve prompts, tools, workflows, and context management. The framework separates search decisions from execution and evaluation and includes a working GEPA variant with shared failure diagnosis and staged evaluation. Its stated goal is to make methods such as GEPA, RRSI, and Harness² easier to implement and compare. Developers reported higher task pass rates with Gear-optimized configurations using GPT 5.6 Luna at medium reasoning effort, but supplied no independent measurements.
r/MachineLearning · 🔥 5 - BHI framework evaluates the health of 158 LLM benchmarks
Researchers affiliated with Alibaba Data and Xiaotian Hengyu proposed the Benchmark Health Index (BHI) to assess whether LLM evaluation benchmarks remain useful as models improve. Based on technical reports for 117 models, the framework evaluates 158 benchmarks across capability discrimination, saturation resistance, and influence. The paper was accepted as an Oral presentation for the NeurIPS 2026 Evaluations and Datasets Track.
机器之心 · 🔥 7 - Personal AI agents face product, cost and ecosystem tests
A Best Ideas community compilation published by Overseas Unicorn examines personal AI agents, including Meta Muse, OpenAI dots, Instinct and Anthropic's Claude Duet. It argues that proactive task execution, cloud hosting and easier setup mark progress over chatbots, while the field remains early and lacks an “iPhone moment” for sustained mainstream use. The compilation also identifies high per-user service costs, platform restrictions, account authorization, permissions and privacy as unresolved commercial and ecosystem constraints. Its product, adoption and cost claims are community or practitioner views and estimates rather than independently verified findings.
虎嗅 AI · 🔥 0 - Prime Intellect and Nvidia post-train Nemotron 3.5 Super VL for enterprise agents
Prime Intellect and NVIDIA built 2,500 verified training tasks aimed at teaching long-horizon enterprise agent work. According to the Prime Intellect blog, 100 GRPO steps — about 11 hours of training on 128 NVIDIA GB200 GPUs — took NVIDIA Nemotron 3.5 Super VL from 34.0% to 57.7% on AutomationBench-AA (public split), a 23.7-point gain over its starting SFT checkpoint. Each task is scored on actual changes to simulated business applications and files, with objectives the agent must complete and guardrails it must not break. The environment library, trainer and templating used for validation are open source, and the same loop runs as a hosted service on Prime Intellect Lab.
Prime Intellect Blog · 🔥 0 - SpecAgent has weak models draft tool-call trajectories and strong models verify
An arXiv paper introduces SpecAgent, a collaboration protocol that turns a strong model from the author of each agent step into the reviewer of steps drafted by a weaker model. The weak model executes read-only tool calls while drafting so each step is grounded in real observations, and stops at the first action that would modify the environment; the strong model then verifies the drafted segment in a single prefill pass, commits the longest prefix it judges semantically valid, and repairs the first rejected step. The authors report SpecAgent comes within one point of the strong model's success rate on τ²-Bench while roughly halving memory traffic and wall time, recovers 44–75% of the quality gap on AutomationBench and VitaBench, and in an edge-cloud MCP-Atlas deployment closes 53–75% of the weak-strong quality gap with the strong model generating at most half as many tokens.
Hugging Face · Papers · 🔥 0 - DAEDALUS: Bootstrapping agent memory from self-generated tasks
An arXiv paper introduces DAEDALUS, a method that bootstraps reusable agent memory from self-generated practice without existing tasks or an oracle verifier. An explorer agent creates challenging but solvable tasks while a solver agent attempts them; heuristics derived from solver failures are accepted only after repeated in-context success, then consolidated into a memory bank. Across AppWorld, τ²-bench and AutomationBench, it improves mean success rates by up to 15.9 points and pass^5 by up to 2.2x over a no-memory baseline, at lower inference cost than most comparable methods.
Hugging Face · Papers · 🔥 0
Experience and discussion from the community
Share my AutomationBench experienceAsk about AutomationBench
Nobody has shared their experience with AutomationBench yet.