Benchmarking decision models for AgenticOps root-cause analysis
A first-party report compares 12 decision models across 15,788 scored cases drawn from STAROps operational scenarios and tests candidate models at different integration positions in 10 root-cause analysis tasks from rca-benchmark. On 1,745 shared single-question inputs, the internal Pilot model was 2.52 percentage points more accurate than Qwen3.8 Flash, with 80.3% lower P95 latency. Rankings varied by task: Decision 1 at task dispatch improved RCA scores by 6.40 points, Clef 27B at progress assessment by 7.20 points, and Pilot at root-cause validation by 2.40 points while reducing time by 4.0%. The authors recommend assigning suitable repeated judgments to specialist models. The article was included in the Alibaba Cloud AI Agent Handbook.
作者:朴象
本文已收录到开源项目 Alibaba Cloud AI Agent Handbook 的决策模型专题,该专题已汇总了决策模型综述,以及我们在智能体评测、AgenticOps、工单分诊助手等方面的实践,欢迎大家将 Handbook 作为你的 Agent 外接知识库。文末点击阅读原文,访问该开源项目。
若您无法访问 GitHub,也可在此处下载 PDF 版:
developer.aliyun.com/ebook/8630
一条慢查询告警,Agent 查了 10.16 分钟,调用 123 次诊断模型,最后拿到 94 分。结果不差,但是,先查什么,这条证据有没有用,下一步该往哪走。这些判断,每次都需要让大模型生成一遍答案吗?以前我们在选更强的调查模型,同时持续优化我们的 harness,决策模型的出现拓宽了我们的优化思路,我们想看看能不能把调查中的一部分 LLM 判断交给专门的决策模型。
Some source content is omitted. Continue reading on the original site.
#Alibaba Cloud#STAROps#Pilot#Qwen3.8 Flash#Microsoft Decision 1#Clef SFT-27B#Decider#Drex Max#Liquid D1 Ultra#rca-benchmark#Alibaba Cloud AI Agent Handbook