开源基准测试揭示推理是云操作中AI最大的瓶颈
CloudOps Bench是一种新的开源基准测试，它测量AI agents在云中如何很好地处理典型的DevOps任务，从固定服务和移除资源到限制访问权，推出更新，以及退役项目。 1.0版本包括AWS上的45项任务，由有经验的DevOps工程师建造。
即使是最好的模型也在许多任务中失败。 Claude Fable 5.1解决了60%,GPT-6 Astra管理了53%. 密歇根大学的安陈率领的团队认为，主要问题不是缺乏知识，而是推理。 大约80%的失败都是因为agents错误判断任务范围，不要检查所有相关服务，或者停得太早。 大约20%的错误是错误的行动，有些对基础设施有不可逆转的后果。
所有任务在真实的AWS云中和模拟器中运行相同。 计划为GCP,Azure,OCI进行扩展。 GitHub上的项目根据Apache 2.0获得许可，详细故障分析也可用。
AI News Without the Hype – 由人类治愈
订阅THE DECODER免费阅读，每周AI通讯，我们独家的"AI雷达"前沿报告每年6次，全面存档访问，访问我们的评论部分。
现在订阅
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
New open-source benchmark reveals reasoning as the biggest bottleneck for AI in cloud operations
CloudOpsBench is a new open-source benchmark that measures how well AI agents handle typical DevOps tasks in the cloud, from fixing services and removing resources to restricting access rights, rolling out updates, and decommissioning projects. Version 1.0 includes 45 tasks on AWS and was built by experienced DevOps engineers.
Even the best models fail at many of the tasks. Claude Fable 5.1 solved 60 percent, GPT-6 Astra managed 53 percent. According to the team led by Ang Chen at the University of Michigan, the main problem isn't a lack of knowledge but reasoning. About 80 percent of failures happen because agents misjudge the scope of a task, don't check all the relevant services, or stop too early. Roughly 20 percent of errors are wrong actions, some with irreversible consequences for the infrastructure.
All tasks run identically in the real AWS cloud and in an emulator. Extensions for GCP, Azure, and OCI are planned. The project on GitHub is licensed under Apache 2.0, and a detailed failure analysis is also available.
AI News Without the Hype – Curated by Humans
Subscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive "AI Radar" frontier report six times a year, full archive access, and access to our comment section.
Subscribe now
Auto-fetched and translated by AI Atlas. All rights belong to the original author.