CloudOpsBench 1.0评测AI智能体云运维能力
据 The Decoder 报道,开源基准 CloudOpsBench 1.0 包含45项AWS任务,由资深DevOps工程师构建,用于评测AI智能体修复服务、删除资源、限制访问和部署更新等能力。测试显示,Claude Fable 5.1完成60%的任务,GPT-6 Astra完成53%,没有模型解出全部任务。密歇根大学团队的分析称,约80%的失败源于任务范围判断错误、遗漏相关服务或过早停止,约20%则涉及错误操作。
CloudOpsBench为评估AI智能体能否在真实AWS环境中执行具有后果的运维工作提供了开源基准。
New open-source benchmark reveals reasoning as the biggest bottleneck for AI in cloud operations
Maximilian Schreiner View the LinkedIn Profile of Maximilian Schreiner
Nov 4, 2026
CloudOpsBench is a new open-source benchmark that measures how well AI agents handle typical DevOps tasks in the cloud, from fixing services and removing resources to restricting access rights, rolling out updates, and decommissioning projects. Version 1.0 includes 45 tasks on AWS and was built by experienced DevOps engineers.
Even the best models fail at many of the tasks. Claude Fable 5.1 solved 60 percent, GPT-6 Astra managed 53 percent. According to the team led by Ang Chen at the University of Michigan, the main problem isn't a lack of knowledge but reasoning. About 80 percent of failures happen because agents misjudge the scope of a task, don't check all the relevant services, or stop too early. Roughly 20 percent of errors are wrong actions, some with irreversible consequences for the infrastructure.

#CloudOpsBench#Amazon Web Services#Claude Fable 5.1#GPT-6 Astra#University of Michigan