作者：银桑
阿里云 STAROps 是一个生产运维 Agent，通过连接多域数据、运维工具与专家经验，从被动响应走向智能自治，7×24 保障业务连续、稳定运行。本文将分享 STAROps 的评测和调优实践。
本文已收录至开源项目 Alibaba AI Agent Handbook，您可在此处获得更佳的阅读体验：aliyun.github.io/ai-agent-ha…
背景与评测对象
运维评测集的快速发展
新的大模型或 Agent 发布时，外界判断它的能力上限，主要依据的是一份份评测报告，背后是公开或私有的评测集。评测集覆盖哪些任务、标准答案是否可靠，直接决定评估结论能不能信，这一点对大模型和 Agent 都成立。公开评测集便于横向比较，企业要评估自己的 Agent，还需要一套贴合自身业务和场景的评测集。
以运维评测集为例，图里整理了 30 多项运维相关的评测资源，横轴是年份，纵轴是任务领域。数量上，新的评测集逐年集中出现，大部分是近两年的成果。覆盖面上，早期集中在遥测诊断数据和少数可执行环境，之后出现了专门考核根因与因果过程的评测，并扩展到网络、数据库和 AI 基础设施。变化最大的是评估方式：早期多基于日志、指标等离线数据，只看 Agent 给出的最终答案，像是在考场上拿着试卷答题；现在越来越多的评测把 Agent 放进可执行的运维环境里完成任务，更像是放进真实的猎场里看它的本事，考核也从答案延伸到它找到的证据，以及能不能把问题修好。
考核延伸到执行过程之后，一条数据要回答的问题也变了：以前只问 Agent 最后给的根因对不对，现在还要问它查了哪些数据、为什么排除了某个假设、有没有用错时间窗、能不能把问题修好。这些问题都要靠数据里保存的过程信息和证据来回答，后面各节的做法都从这里出发。
一次根因分析过程
这种变化，从一次具体的根因分析过程里能看出来。STAROps 是一个生产运维 Agent，根因分析是它的主要能力之一。一次根因分析可以抽象成三部分：输入，多轮「Agent 判断、工具取证」的循环，以及最后的诊断结论。
图左边是输入，这次是三条告警：下单错误、商品查询错误、前端异常。右边是诊断结论，包括根因实体、传播链、影响面和处置建议。中间的多轮循环是重点。 每一轮里，Agent 先定范围，从告警出发判断问题大概落在哪些服务、哪段时间；再提假设，列出可能的根因；然后排除，用取到的证据否定不成立的假设；最后定下一步，决定接下来查什么。判断需要证据时，Agent 调用工具去查指标、日志、调用链、事件、拓扑和变更记录。各轮之间靠上下文衔接：已经取得的证据和已经排除的假设，决定下一轮往哪里查；历史排查经验，影响先查什么、后查什么。
评一次根因分析，要看每一轮的判断和取证站不站得住，因为结论正确不等于过程可信。具体有三件事要评，每一件都对评测集提出了要求。第一是这套分析思路能不能通用：同一个 Agent 在一类故障上表现很好，换一个故障场景、换一类异常可能就不行，评测集要覆盖足够多的场景，才看得出这种差异。第二是结果错在哪一步：结果不对时，要分清是工具没查到，还是查到了没用对，为此数据里要能对照每一步：Agent 调了什么工具、工具返回了什么、它据此做了什么判断。第三是执行效率：结果对了，也要看有没有无效的工具调用、上下文有没有越堆越乱，这需要保留完整的调用过程。
还有一类错误更难发现，就是图下方标的时间窗查错。这时工具调用成功，返回的也是真实数据，Agent 据此做的推理可能前后自洽，结论甚至可能碰巧是对的，但这些数据对应的不是这次故障发生的时间段，支撑不了结论。从外面看一切正常，所以这类错不容易发现。要发现它，评测数据里必须记下这次故障真实的时间窗，拿 Agent 实际用的时间窗去对照。 这也是后面「时间绑定」和标准答案里「告警锚点」这两项要求的由来。
评测数据的要求
对评测集的一些需求
一次任务要看完整过程，整体能力还要看覆盖和稳定性。所以评测数据有三条基本要求：覆盖要足够广，能重复执行，每一条还要证明自己有效。 图上三栏的数字都来自外部的公开研究，不是本评测集自己的统计。
先看覆盖。一版公开的运维根因数据集，在三个系统上覆盖了 27 类注入机制、500 个可评测实例，这个覆盖面还不能代表真实环境，真实环境里的故障类型更多。再看重复执行。同一套题在相同或者不同的环境下，执行结果也会有差异：一项研究里，同一个 Agent 对同一个场景跑 10 次，只有 6 次诊断成功。原因包括大模型本身的不确定性和环境的变化，所以一个场景只跑一次、只留成功的那一次，是不够的。
最后是有效性。有些数据集虽然有故障的输入和输出，却缺乏清晰的因果关系，要考核的那层推理其实并不存在。比如某个服务出现异常，因为这次注入的是 CPU 故障，就直接把 CPU 当成标准答案；但真实运行时，同样的异常完全可能由其他原因导致。另一项研究从九千多次初始注入里，最后只保留了 1,430 条，其余的按它的标准没有造成可感知的异常。可见执行了注入不等于产生了可用数据。
所以公开数据集不能直接替代面向自身系统构建的评测集，这三条要求落到自己的构建上，各有对应的做法。覆盖广，要求题目按自己系统的能力域和难度去铺，不能拿一份公开集合当全部，后面能力地图那一节讲的就是这件事。 重复跑，要求能区分大模型本身的不确定性和环境变化带来的差异，环境本身也要足够稳定、数据能原样重跑，后面环境和质量检查两节会分别讲。验有效，要求注入之后先证明故障真的发生了、因果链确实存在，再去写标准答案，这是构建循环里信号捕获和证据提取两步要做的事。
一条运维评测数据的构成
一条评测数据要保存的，是一次评测的完整记录，核心要求是能还原一次完整的分析轨迹。图上第一行是轨迹，分五步，拿前面那次根因分析对照来看。目标是要考核的东西，那次是从下单错误、商品查询错误、前端异常三条告警出发定位根因。观测是这次任务里 Agent 要看的数据，来自指标、日志、调用链、事件、拓扑和变更记录。工具是查这些数据用到的工具。动作是排查路径上的每一次工具调用或推理，包括定范围、提假设、排除、定下一步这几轮判断，以及每一次取证调用。结果是最后的结论，也就是根因实体、传播链、影响面和处置建议。
轨迹和答案的区别也在这里。答案只是最后一步的结果，轨迹还包括前面的目标、观测、工具和动作。只保存答案，评测时只能比结论；保存了轨迹，才能看出 Agent 在哪一步查错了数据、在哪一步做错了判断。轨迹不要求 Agent 按固定路径执行，但关键判断、关键证据和关键动作必须明确，否则答案对了也可能是碰巧对的。
图上第二行是轨迹之外必须一起保存的两样东西：配套条件是环境、工具和观测手段，核验依据是评价标准。它们要和轨迹放在一起保存，是因为评测集要反复拿来评不同版本的 Agent。 配套条件没有一起保存，下一个版本来跑同一道题时，可能已经换了环境、换了工具，跑出来的轨迹和上一次不可比；评价标准没有固定下来，同一条轨迹这次判对、下次判错，分数的变化就说不清是 Agent 变了还是判分的人变了。
最下面四个是运维场景特有的约束。一条数据保存下来的，是某个时间窗口、某套环境下的一次完整快照，脱离上下文的静态答案没有用处，这四个约束后面各有一项做法对应。环境绑定：故障离开真实环境就不存在，所以故障要在真实的服务、流量和观测里发生，数据和产生它的环境一起保存，对应按需扩展的评测环境。时间绑定：证据有保留期，调用链和日志过几天就会被清掉，要及时把时间窗和当时查出来的证据保存下来，否则过一段时间就无法回头核验，对应线上回流时的「先保全」和标准答案里的告警锚点。完整因果链：因果链上的每一步都要有证据，对应标准答案里的因果链字段，原因、传播和影响分开写。低频高风险：生产故障不能拿线上反复试，就用故障注入在评测环境里按同一机制造出来，可以反复执行。
根因分析数据的构建流程
STAROps 评测集的构建流程
有了这些要求，再看一条根因分析评测数据是怎么构建出来的。图上这条线共十二步，分四个阶段，图下方标了每一段由谁来做。注入只是其中一步，注入前后的准备、取证和核验才是大头，构建一条数据的时间也主要花在这里：等告警、核信号、取证、写答案、过两道检查，任何一步不过还要退回重来。
第一段是设计，由 Agent 判断。先选场景，题目来自后面要讲的两个来源：能力地图上还空着的格子，以及线上回流的真实问题；再设计故障，定下注入什么、注入在哪，以及它应该沿哪条链传播、影响到哪里。这两步交给 Agent，是因为选哪个场景、故障设计成什么样，都要结合业务和架构去想。
第二段是注入、验证与恢复，由工具执行：这几步是确定的动作，每次执行都要一样，才能重复。注入准备要确认环境处在未注入时的干净基线上，流量在跑、观测可查，否则注入之后看到的异常，分不清是故障造成的还是原来就有的。验证生效和等待告警是两件事：注入动作执行成功，只说明命令跑完了，故障是否真的生效要看信号，后面构建循环里「信号捕获」的六项就是这一步要查的东西；告警是否触发，要等告警真的出现，告警是后面诊断的起点，也是标准答案里的锚点。最后把环境恢复回去，下一次注入要从干净的环境开始。
第三段是取证与写答案，又回到 Agent 判断。取证在故障期间和恢复之后两个阶段都要做。图上的观测取数排在恢复之后，但故障期间的现场同样要取：验证生效、等待告警时看到的信号、告警和调用链，反映的是故障正在发生时的状态，要当场保存下来。恢复之后再取一次，这时故障从发生、传播到恢复的整段数据都有了，收集和分析这段时间的指标、日志、调用链等数据，才能把注入点、根因、传播链、影响面完整还原出来。之后按后面会讲的七个字段写标准答案，把一段时间的数据组织成一份因果链完整的答案，需要读懂信号之间的关系。
第四段是检查与入库，由程序和人把关，两道检查查的东西不一样。程序门禁查能写成规则的事：字段是不是齐全，答案里的每个数字有没有附上可执行的查询、重跑出来是不是同一个值。准入复核查需要判断的事：因果链说不说得通，相似的候选有没有排除干净；这一步先由 Agent 审一遍，最后由人决定准不准进，人看到的已经是通过了程序门禁、又经 Agent 审过一遍的答案。通过之后，数据进入评测集。
这条线的终点是「STAROps 完成被评诊断」：数据进入评测集之后，被评的 Agent 拿着同样的输入去做一次诊断，它的输出再和标准答案逐个字段对照。所以前面每一步都会影响评分：故障没有真正发生，标准答案就是错的；证据没有取全，评分就缺依据；检查没有把住，错答案会把正确的诊断判成错。
持续构建的主要问题
构建评测集的四个核心问题
单条流程跑通只是起点。要持续构建、长期使用，实际还有四个问题要解决，后面逐个回答。第一是如何选择合适的题目，也就是题目从哪来、怎么设计，由「按能力域和难度规划场景」和「线上问题还原成评测集」两节回答。第二是如何构建环境：评测数据要从真实的环境里产生，Agent 是和环境交互的，脱离真实环境就没有可评的东西，由「按需扩展评测环境」一节回答。第三是如何保证评测集的有效性，由「基于故障注入的评测集构建循环」一节回答。第四是如何高质量地持续迭代，由「评测集的质量检查」一节回答。这四个问题互相依赖：环境不稳，有效性就无从验证；题目来源不受控，效率再高也只是在批量生产熟悉的那几类。所以要用一套方法同时回答它们，下面按题目、环境、构建、监督的顺序展开。
构建方法与核验
评测集构建中 Agent 分工
以往这些事情靠人逐条梳理，规模上不去；拆给 Agent 之后可以并行推进。构建拆成四段，每一段对应前面的一个问题，由一个 Agent 负责，各有四项具体工作。题目由 ScenarioAgent 负责，做缺口识别、变更分析、架构还原和现场回流：缺口识别是对照能力地图找还没覆盖的格子，变更分析和架构还原是从业务系统的变更和架构出发梳理要验证的分析能力，现场回流是把线上问题还原成评测数据。环境由 EnvironmentAgent 负责，做拓扑构建、业务仿真、流量注入和观测接入，也就是把服务按需要的依赖关系搭起来，让这些服务承载一条真实的业务链，让业务链上一直有请求在跑，再把指标、日志、调用链和告警接通，故障发生时看得到。
构建由 BuildAgent 负责，四项工作就是构建循环的四步：故障注入、信号捕获、证据提取、准入复核。监督由 SupervisorAgent 负责，做质量体检、失效识别、回归复测和难度进阶：质量体检是五项检查，失效识别是把跑不出来的数据下架，回归复测是被评 Agent 修复后在仍然有效的数据上复测，难度进阶是分数都接近满分时主动提高难度。准入复核也由 BuildAgent 来做，审核时不沿用构建时的上下文，重新取证据核对，否则复核会顺着已有的答案走。
图中最上面一条写的是人的位置：定目标、定验收、终审。定目标，是决定要构建什么数据：哪些能力域、哪些难度要优先补，地图上的空格先做哪一格。定验收，是决定什么样的数据算合格：标准答案要有哪些字段、证据要到什么程度。终审，是数据进入评测集之前的最后一道决定。这三件事都要结合业务和评测目的来判断，留给人；逐条构建、逐条核对这类重复的工作，交给 Agent。人不退出，但不再逐条去梳理。
评测集是用来评 STAROps 的，但构建评测集这件事本身，也是一项复杂的 Agent 任务。一条数据从场景理解、任务规划、环境构建、故障执行、工具调用、观测取证、因果还原，到标准答案构建、独立复核和质量准入，任何一环都不能孤立完成。所以能不能自动化地构建出一套可信的评测集，反过来也能看出一个组织的 Agent 工程能力：有没有能力让一组 Agent 在真实环境里持续完成复杂工作，而且最后交出来的东西是可信的。从这个角度看，它也是对 Agent 能力和基础设施成熟度的一次压力测试。
这场压力测试同时考验四个方面。一是 Agent 自身的能力：能不能理解业务和架构，能不能正确调用工具，能不能基于证据做推理，能不能发现自己的结果存在问题。二是 Agent 之间的协同：题目、环境、构建、监督这几个角色之间要有清晰的输入输出和状态传递，前一个角色的产出必须能成为后一个角色可靠的输入。三是支撑它们的基础设施，这一层往往最关键：工具是否标准化，环境是否可重建，上下文和状态怎么保存，长任务怎么恢复，失败怎么重试，不同角色之间怎么交接，证据怎么追踪。四是治理和质量机制：Agent 自己说完成了不算数，还要有程序检查、证据校验、独立复核和人工终审。至于用一个 Agent、多个 Agent，还是 Agent 加 Skill 来实现，都可以，关键在于组织有没有把这些能力、工具和环境建设到 Agent 能自主调用、相互协同的程度。
按能力域和难度规划场景
先看第一段：题目从哪来。出题有两个来源，第一个是主动规划的能力覆盖：基于自身的业务系统，梳理需要验证的分析能力，有点类似于混沌工程。这里尽量借用 Agent 本身的领域知识，以及它能掌握到的业务和架构来做分析。
具体做法是按能力域和难度把场景铺成一张能力地图。横轴是能力域：代码与变更、应用运行时、数据库、容器、中间件、云产品、主机、网络。纵轴是难度：单点故障、有传播链、多故障并发、告警风暴。纵轴不是互斥的分类，它表示要排除多少干扰才能定位：单点故障，一个异常一条线；到了告警风暴，同时有十几个信号在响，真正的根因藏在里面。
地图上每一格放的是具体场景。单点故障这一行，从代码与变更到网络依次是配置改错、接口内部错误、慢查询、容器反复重启、队列积压、CPU 打满、磁盘写满、丢包；有传播链这一行，有发布引发报错、依赖超时堆积、锁等待致超时、节点驱逐降级、缓存击穿压库、主备切换断连；多故障并发这一行，目前有发布叠加限流、限流叠加超时；告警风暴这一行还没有具体场景，图上右上角标的就是下一批构建任务。
拿几格对比就能看出难度差在哪。数据库这一列，单点的慢查询是一个异常一条线；往上一格的锁等待致超时，数据库里的锁等待传导成调用方的超时，要沿着链往回找。代码与变更这一列，单点的配置改错只有一个原因；多故障并发的发布叠加限流，同时有两个机制在起作用，要把它们分开。这样一格一格铺下来，空着的地方就是还没覆盖的，由人来确认优先级，再交给 Agent 构建。
线上问题还原成评测集
出题的第二个来源是线上真实发生过的问题。能力地图组织的是计划性的覆盖，地图上的格子是按已知的能力域和难度规划出来的；线上发生的问题不受这张地图限制，它可能落在某个已有的格子里，作为这一格的一条真实样本，也可能是地图里原来没有记录的机制，那就要在地图上补一格，再按这一格去规划后续的构建。
这条路的顺序很要紧：先保全，再还原。线上出问题的时候，先用工具把证据保全下来，包括轨迹、调用链和当时的时间窗，之后还原机制、比对重放、写标准答案都要用到。先保全，是因为调用链这类证据保留期短，处理晚了，这条数据就取不回来了。
保全之后分三步。机制还原由 Agent 做，弄清因果关系和必要条件，也就是这个故障在什么条件下才会发生，再把它重建成评测环境里的一个场景；同样的报错可能来自不同的原因，只看现象，重建出来的场景就不对。等价重放由工具做，用同一机制重新执行；只把报错复现出来不够，要让同一个原因在评测环境里再发生一次。重建验证由程序做，线上与重放的信号要对齐：信号出现的先后反映故障传播的顺序，信号和时序对不上，说明重建出来的传播路径和线上不一样，要退回重建。验证通过之后，交给 BuildAgent 统一做答案核验，让线上还原出来的数据和注入构建的数据用同一套标准：同样的七个字段、同样的证据要求、同样的检查，进入评测集之后才能放在一起比较。
按需扩展评测环境
题目定下来，才知道要建什么环境。环境扩展由覆盖需求触发，单纯规模变大，不作为新建环境的理由：能力地图出现空白、线上出现新机制，或者故障需要隔离时，才扩展环境。图左上角写的是人在这一步的位置：先由人确定能力域和隔离边界。隔离是为了避免不同故障在同一个时间窗里互相干扰，这和构建循环里「噪音干扰」那一项是同一个问题。
图上是环境的演进。起点是一个开源微服务样板，用脚本驱动一条下单链路，只能做应用层单点故障。之后每新增一类故障域就扩一次，补什么由要覆盖的故障决定：增加主机故障域时，要有主机层的注入手段，还得能看到主机指标，否则注入了也看不出影响；增加云产品故障域时，要接入云产品自己的观测与告警，因为这类故障最先表现在云产品上。这两步对应能力地图上新增的主机、云产品两列。
做跨集群和多故障并发时，就得真的有两套集群和跨集群的调用，于是克隆了第二套集群，流量也从脚本扩成常态加场景化两层，这一步对应更高的难度档。常态流量让环境像真实系统一样一直有请求在跑，故障发生时才有业务被影响、才有信号；场景化流量按具体场景驱动特定的链路，让需要验证的那条传播路径真的有请求经过。告警阈值用历史数据回放校准，让告警在该触发时触发、不该触发时不乱报。现在已经有多套集群、多条业务链。
一次扩展分五步走。先由 Agent 读取依赖和已有环境，确定要补哪些服务、流量路径与观测，是否需要隔离；再由 Agent 编排任务，工具按模板生成并部署服务，或者克隆集群；然后接通观测与两层流量，建立告警、回放校准阈值，并固化可重复的注入和恢复配置；接着由程序检查未注入时的基线，常态流量必须干净，观测必须可查，检查不过就停止构建；通过后，产出可复刻的环境配置代码。
基线检查卡得严，是因为基线本身有异常时，注入产生的信号会和原有的异常混在一起，分不清哪些是故障造成的；观测查不到时，故障发生了也看不到。产出的环境配置代码覆盖拓扑、业务、流量、观测，加上固化好的注入和恢复配置，同类故障随时可以复刻，需要几套就复刻几套，不必重新摸索。这些环境长期要做到稳定承载故障、流量持续且接近真实业务、观测完整接入，三者里优先保证观测：缺少它，连数据中的故障是否真实都无法确认。
基于故障注入的评测集构建循环
环境就位之后，才谈得上把一道题真正造出来。BuildAgent 的任务是让每条数据都有可以逐项查证的标准答案，它做的四件事是一个循环：故障注入、信号捕获、证据提取、准入复核。准入复核先由 Agent 审核，最后由人决定准不准进入评测集；四步里任何一步没过，就退回重做。
故障注入有三条路。 复用已经做好的注入方式，适合已经造过、注入和恢复配置已经固化的故障，可以直接重复执行；改配置，适合由配置引起的故障，比如地图上「配置改错」这一类；改代码，适合需要让程序本身出错的故障，比如「接口内部错误」。不管走哪条路，注入完成后都要逐项证明故障真的发生，不能因为注入命令执行成功就认为故障发生了。
信号捕获有六项必查。告警源头，看告警是不是由这次注入的对象触发；环境变更，看同一时间有没有别的变更混进来；指标异常，看根因相关的指标有没有真的异常；调用链路，看异常有没有沿调用链传播出去；影响范围，看哪些上下游受到了影响。其中最需要留意的是噪音干扰：同一个时间窗里如果有别人的故障也在跑，这次注入产生的信号就说不清是谁的，这条数据直接作废。
证据提取要从现场把注入点、根因、传播链、影响面四个维度都取出来，再组织成标准答案。 标准答案的格式决定评估能查到什么。有的标准答案只记录故障时间、根因组件和故障原因，这样的格式没有记录因果链：无法回查判断依据，也难以区分相似候选；数字没有固定时间窗可以重查，能比对的就只剩组件名。所以在这一步，标准答案由 Agent 组织成七个字段。
根因实体要写到具体的资源或服务，评分时才能判断 Agent 定位得准不准。故障类型记录可以验证的机制，不是表面的现象。 因果链把原因、传播和影响分开，Agent 把传播路径上的某个服务当成根因时，评分能看出它错在哪里。鉴别点写为什么排除了相似的候选，让答案除了结论，还说明了为什么不是别的。告警锚点保存实际触发的事件和时间窗，前面讲的时间窗查错，就靠它来对照。关键数字附上原查询，谁都可以重新查一遍。最后一个字段是评分规则：按字段独立打分，一个答案可以根因对、传播链错，分数能如实反映出来。
证据提取的四个维度和这些字段是对应的：注入点和根因，落在根因实体和故障类型上；传播链，落在因果链里原因到传播的那一段；影响面，落在因果链里影响的那一段。四个维度取全了，答案的骨架就有了；再加上鉴别点、告警锚点和关键数字，才是一份可以逐项核验的答案。
其中数字必须单独核验。有一条答案曾写着错误率约 25%，人工阅读没有发现异常；工具重新查询连续五个时间窗后，错误数均为零。还有两类数单看都像是对的：查询接口有返回上限时，返回条数不能直接当作总数；基线窗和故障窗长度不同时，比值也不能直接用。所以数字要连同查询和时间窗一起写，能重新执行，读的人才能判断这个数是怎么来的。写不出查询的数字，不能进入标准答案。这样写出来的标准答案，是一组可以重新验证的事实：每一步都附证据，谁都可以重新查一遍。
质量检查、规模与下一步
评测集的质量检查
入库前核验通过，不代表以后一直有效。环境和观测配置会变化，所以评测集本身还要持续检查。图左边是五项检查：质量，看证据是否完整，七个字段齐不齐、每个数字有没有可执行的查询；覆盖，看类型与难度分布，对照能力地图哪些格子还空着；区分度，看强弱分差能否拉开；规模，看最低样本是否足够，整体样本量够不够下结论；新鲜度，看抽样重跑能否跑通。五项按短板检查，不能用平均分抵消局部问题。
新鲜度不能只看数据建成多久。决定一条数据能不能用的，是当初的信号今天还能不能复现：基础镜像、告警规则或观测配置一变，昨天刚建好的数据也可能跑不出来。所以新鲜度主要看抽样原样重跑能不能跑通，数据的年龄只作辅助参考。
图下方是两种情况的处理办法。数据失效的，下架处理：当初的信号已经无法复现，评估却还在照常出分，被评 Agent 在这些题上答不对，会被记成能力问题，研发可能去修一个并不存在的缺陷。被评 Agent 修复之后，在仍有效的数据上复测，确认改进是否保持；复测用的数据里如果混着失效的题，分数的变化就说不清是修复带来的，还是数据带来的。所以先判断是数据的问题还是 Agent 的问题，再决定是下架还是复测。
前面那个错误率约 25% 的例子，后来变成了一条规则：写进标准答案的每个数字都要附上可执行的查询、时间窗和实体，由程序把查询重跑一遍，重跑值和答案不一致就退回；给不出查询的数字直接删掉。检查只认重跑出来的结果，不认数字写得像不像。
五项检查里最不直观的是区分度，图右边的曲线讲的就是它。左半段是原数据，能力强和能力弱的两条曲线一起往上爬，最后几乎重合在顶部，分数都接近满分，分不出强弱；中间的竖虚线是「提高难度」；右半段是加难后的数据，能力强的曲线略有回落后又往上走，能力弱的明显落下去，两条线重新分开。
评测集用久了，强弱模型都能拿高分，分数已经不能说明谁更强。分数接近上限时，Agent 能力提升和数据难度不足会呈现相同的结果，单看分数无法决定该发版还是补数据。做法是主动提高难度再比较：把简单的换成有传播链的、多故障并发的，告警风暴场景建好后也可以叠加上去，让曲线重新分开；如果分差重新出现，就说明原有数据已经难以区分能力。 加难之后分数下降，是题目变难了，不能直接读成能力退步。如果现有数据的分差仍能拉开、数据也能复现，就可以降低新增投入；五项里哪一项下降，就针对那一项再补一批。
近来也有几项研究专门检查评测集本身，正好对应五项检查里的不同项。2026 年 8 月，PAIChecker 复查 SWE-bench Verified 的 500 条数据，发现其中一成多的任务描述与参考修复不对齐，属于质量；2026 年 6 月，CORE-Bench 在几十条修订任务上，表现最好的几种配置已接近满分，说的是区分度；2026 年 7 月的一项研究按置信区间决定需要评多少数据，是规模；ICML 2026 的 MMBench-Live 在关键词检索中只取一年内上传的图像，再生成评测数据，是新鲜度。STAROps 评测集本身也在持续接受这五项检查。
STAROps 评测集规模
截至 9 月，积累的评测数据有 1,400 多条，含已复核与在建的条目。其中根因分析 838 条，已复核有效，不含在建和已淘汰，覆盖 100 多类故障类型；根因评测集 3 月还只有 58 条，半年增长到原来的 10 倍以上。另有取数 500 多条：除了造故障，Agent 取数取得对不对，同样要有数据来评。
公开数据集已经不少，仍然要自己建评测集，原因有三个，前面各节都有对应的做法。一是服务拓扑和告警规则是自己的，公开集合里的服务拓扑跟自己的系统对不上，所以题目要按自己系统的能力域和难度去铺。二是线上新出现的机制公开集合里没有，而它们恰恰是最该评的，要靠线上问题回流补进来。三是环境一变，已有的数据要能立刻重跑，这一点自己维护评测集才容易做到，靠的是环境配置代码和定期的抽样重跑。
构建评测集的成本投入
这些积累背后需要持续投入。图是投入结构的示意，不是实测的成本曲线，画的是人工、环境、模型调用三项投入的相对走向，时间分成三段。起步阶段以人工为主，先把少量场景的构建和验证流程跑通，环境刚建起一套，模型调用还很少。补齐环境与判据阶段，环境、流量、观测和告警要逐项补齐，环境投入快速追上人工；判据靠试跑和返工逐条明确，人工投入也在这一段见顶。复用已有构建路径阶段，环境能够重建、构建步骤和校验规则可以复用，同类场景不必每次从头手工处理，人从逐条构建转向标准维护、异常处理和质量审核，重复的人工投入回落；构建侧的模型调用随构建量上升，在这一段超过人工。
到最后，环境成为三项里最高的一块：集群、流量和观测常年在跑，覆盖越广这部分越大，还会更多。所以用了 Agent，总成本不一定下降：环境运行、模型调用和质量审核都在持续花钱，真正减少的是重复的人工操作。
公开研究里也有类似的做法。2026 年 6 月的一项研究用自动化流水线构建评测任务，构建、验证和评估自动完成，人工负责质量审核，报告构建一条被接受的任务约 40 美元、1 到 2 小时。2026 年 8 月的另一项研究同样用 Agent 流水线筛选材料、组装任务，专家还参与判据和评分规则的校准。这两项都说明构建可以自动化并保留人工把关；成本结构本身要各自实测，不能照搬。
开源开放共建
图上是开放共建的三个方向。开源，是把已建好的评测集逐步开放出去：RCA100、RCA200 已开放，后续逐步开放更多数据和评测方式。 共建，是更多团队提供真实环境，接入真实场景和领域经验，为后续构建提供来源。重建，是把公开场景描述搬进真实环境，重建成完整评测集。其中共建和重建目前还是方向，没有对外承诺时间。
开放之外，构建本身也在往更自动的方向走。重复执行中的很多工作已经交给 Agent，但下一批何时发起，目前仍由人决定：由人发起，评估侧 Agent 执行构建与核验，人工准入。下一步是让评估侧 Agent 根据地图空白、某类分数长期接近上限，以及线上出现的新
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Author: Ginsan
Alibaba Cloud STAROPs are a production dimension, Agent, which moves from passive response to intelligent autonomy by connecting multi-domain data, transport tools and expert experience to secure business continuity and stability. This paper shares the assessment and optimization practices of STAROPs.
This is recorded in the Open Source Project Alibaba AI Agent Handbook, where you can get better reading experience: ali io io......
Background and subject of evaluation
Rapid development of the operational assessment system
When the new LLM or Agent Z94Z is judged to be the upper limit of its capacity, it is primarily based on an evaluation report, which is followed by a public or private assessment. The assessment of which tasks are covered, the reliability of the standard answers, and the direct determination of whether or not the evaluation findings are credible are valid for both LLM and Agent. The public assessment set facilitates cross-cutting comparisons, and enterprises need a set of assessments that fit their business and scene.
In the case of the Transport Assessment Collection, for example, more than 30 environment-related assessment resources have been organized, with a cross-axis of years and a vertical axis being the mission area. Quantitatively, the new assessments are coming together year by year, mostly as a result of the last two years. In terms of coverage, the early concentration on telemetric diagnostic data and a few implementable environments was followed by assessments of causality and causal processes, which were specifically assessed and extended to networks, databases and AI infrastructure. The most significant change was the way in which the assessment was made: earlier, based more on offline data, such as logs, indicators, and only on the final answer given by Agent, such as holding a quiz in the examination field; now, more and more assessments are being made to place Agent in an enforceable environment, more like in a real hunting grounds to see what it can do, and the examination extends from the answer to the evidence it finds, and whether the problem can be fixed.
After the review extended to the implementation process, a question to be answered by the data changed: Agent was asked what was the reason for the final offer, and now it is asked what data it was looking for, why it excluded a hypothesis, whether it used the wrong window of time and whether it could fix the problem. These questions are answered by process information and evidence maintained in the data, and the practice in the subsequent sections begins here.
A root cause analysis process
This change can be seen in a specific root cause analysis process. STAROPs is a productive business, Agent, and root analysis is one of its main capabilities. A root analysis can be abstracted in three parts: input, multiple rounds of "Agent judgment, tool forensics" and final diagnosis.
To the left of the figure is input, and this time there are three alarms: an error in the list, an error in the commodity search, an anomaly at the front end. On the right are diagnostic findings, including root causes, transmission chains, impact and disposal recommendations. The middle multiple cycle is the focus. In each round, Agent set the scope, judging from the point of view of the services and the period of time that the problem would probably fall to the whistle-blower; then set out the assumptions, listing the possible root causes; then exclude, using the evidence taken, the presumption that is not valid; and finally set next step, deciding what to look at. When determining the need for evidence, Agent calls tools to check indicators, logs, call chains, events, poppings and changes in records. The context interface between the rounds: the evidence obtained and the assumptions that have been excluded determine where to look for the next round; and the historical inventory experience affects what to look at first and what to look at later.
An evaluation of the root causes depends on the judgement of each round and the evidence station, since the correct conclusion does not amount to a credible process. There are three specific matters to be evaluated, each of which requires evaluation. The first is whether this analysis is universal: The same Agent performed well on one type of failure, and a different type of malfunction might not work, and the assessment would have to cover enough of the scene to see the difference. The second is where the result is wrong: when the result is wrong, it is clear whether the tool is not found or whether it is not useful, so that the data can be compared to each step: what the tool has been adjusted by Agent, what the tool has returned and what it has been judged by. The third is the efficiency of implementation: the result is correct, depending on whether there is an invalid tool call, context or not, and this requires maintaining the full call process.
There is another type of error that is more difficult to detect, which is an error in the time window under the chart. At this point, tool call succeeded and returned real data, the reasoning followed by Agent may have been consistent, and the conclusions may even have happened correctly, but the data do not relate to the time period during which the failure occurred and do not support the conclusion. Everything is normal from the outside, so it's not easy to detect such mistakes. To find it, the evaluation data must record the real window of time for this malfunction and compare it with the window of time that Agent actually used. This is also the origin of both the requirements of "time-bound" and the standard answer, "the alarm anchor".
Data assessment requirements
Some needs for assessment collections
A mission depends on the full process and the overall capacity on coverage and stability. There are thus three basic requirements for assessing data: that coverage be sufficiently broad, that it be repeated and that each article prove to be effective. The figures in the three columns above are from outside public research and not from this assessment.
Read the cover first. An open version of the Wigand data set, covering 27 type injection mechanisms and 500 assessable examples on three systems, is not yet representative of the real environment, where there are more types of failures. Repetition. The same set of questions varies in the same or different contexts: in one study, the same Agent runs the same scene 10 times, and only six successful diagnoses. The reasons include the uncertainty of LLM itself and changes in the environment, so it is not enough that a single scene runs once, leaving only the successful one.
Finally, effectiveness. Some data sets lack clear causal relationships despite faulty input and output, and the layers of reasoning to be tested do not exist. For example, there is an anomaly in a service, because this is a CPU failure, and the CPU is used as a standard answer; but when it really works, the same anomaly can be caused by other causes. In another study, from more than 9,000 initial injections, only 1,430 articles were retained, and the rest, by its standards, did not cause perceived anomalies. Visible injections do not amount to the generation of available data.
Thus, open data sets are not a direct substitute for assessments built for their own systems, and these three requirements fall on their own constructions, each with its own practices. Broad coverage requires that the subject be covered according to the capabilities and difficulties of the system, and that an open collection not be used as a whole, as is the case in the section on the map of the latter capabilities. Repeated running requires a distinction between the uncertainty of LLM itself and the differences caused by environmental change, and the environment itself must be sufficiently stable and data capable of reruning in its original form, with the latter two sections on environmental and quality checks going separately. Validation requires proof that the failure has occurred, that the causal chain does exist, and that standard answers are written, which are two steps to build a cycle of signal capture and evidence extraction.
Composition of an operational assessment data
An evaluation data is to be kept as a complete record of an evaluation, and the core requirement is to restore a complete analytical trajectory. The first line of the figure is a track, in five steps, compared with the previous root cause analysis. The objective was to test what was at the root of the error in the order, the error in the search for goods and the three alarms on the front end. Observations are data for Agent in this mission, from indicators, logs, call chains, events, popping and changing records. The tool is the tool used to check the data. The actions were each tool call or reasoning on the route, including the determination of the scope, assumptions, exclusions, determinations of the next step rounds of judgement, and each evidentiary call. The result is the final conclusion, i.e. the root causes, the communication chain, the impact and the disposal recommendations.
The difference between the trajectory and the answer is here. The answer is only the result of the last step, and the trajectory also includes the objectives, observations, tools and actions ahead. Only the answer can be saved and the evaluation can be compared to the conclusion; only the trajectory can be preserved to see where Agent was wrong about the data and where it was wrong to judge. The trajectory does not require Agent to proceed on a fixed path, but the key judgement, the key evidence and the key action must be clear, otherwise the right answer may also happen.
The second line of the figure is two things that must be preserved together outside the trajectory: the accompanying condition is the environment, the tools and the means of observation, which are based on the evaluation criteria. They have to be stored together with the trajectory because the assessment set is used repeatedly to evaluate different versions of Agent. The accompanying conditions were not preserved together, and the next version might have changed the environment, the tools, the trajectory and the last time; the evaluation criteria had not been fixed, and the same trajectory had changed this time, the next error, and it was not clear whether Agent had changed or the person who had passed the score.
The bottom four are the constraints specific to the environment. A data saver is a complete snapshot of a time window, a set of environments, and a static answer from context is not useful, and one of the four constraints follows. Environmental binding: failure does not exist outside the real environment, so it occurs in real services, traffic and observations, data are preserved together with the environment that produces it, and the environment that should be assessed as being expanded. Timing: The evidence has a retention period, the call chain and logs will be cleared in a few days, the window of time and the evidence found at the time will be saved in time, otherwise there will be no re-checking in time, the “preserve” on the line and the alarm anchor in the standard answer. Complete causal chain: each step of the causal chain must be supported by evidence of the causal chain field in the corresponding standard answer, with causes, transmissions and impacts separated. Low-frequency high risk: failure of production cannot be tested repeatedly on the line, which is created by the same mechanism in the assessment environment and can be repeated.
Build process for rooting data analysis
Build process for STAROPs
With these requirements, see how one root cause assessment data is constructed. The figure is 12 steps along this line and is divided into four stages, with each segment indicated below. Injection is only one step, and pre- and post-injection preparation, forensics and verification are the main ones, and the time spent building a single data is mainly here: waiting for alarms, nuclear signals, forensics, writing answers, cross-checks, any one of which has to be reversed.
The first is design, judged by Agent. First the scene, the topics came from two sources to be discussed later: The blank cells on the capability map, and the real problem with the up-line flow; re-design failure, determination of what to inject, where to inject, and which chain it should spread and where to affect. These two steps were given to Agent because the selection of the scene and the failure were designed to be considered in the context of business and architecture.
The second is injection, validation and recovery, implemented by tools: These steps are definite actions, and each execution must be the same to be repeated. Injection is prepared to confirm that the environment is on a clean baseline when it is not injected, and the flow is running and observed, otherwise the anomalies observed after injection are not clear whether they were caused by malfunction or were there. Validation and waiting for alarms are two things: an injection of action is a success, which means only that the order is running out, whether the failure really works depends on the signal, and six of the next "sign capture" in the construction cycle is what this step is about; if the alarms are triggered, they are the starting point for the subsequent diagnosis and the anchor in the standard answer. Finally, the environment is restored and the next injection begins with a clean environment.
The third paragraph is evidence and answers, and goes back to Agent's judgment. The evidence was to be performed during the breakdown and after recovery. The observational extracts on the chart follow recovery, but the site during the malfunction is also required to take them: validate the validity of the signal, wait for the alarm, alert and call chain, reflecting the state in which the malfunction is occurring, to be preserved on the spot. Once recovered, the entire data range of the malfunctions from occurrence to transmission to recovery is available, and the collection and analysis of data on indicators, logs, call chains, etc. for this period will enable the full recovery of injection points, root causes, transmission chains, and impacts. The seven standard answers field, which will follow, organize the data for a period of time into a complete answer to the causal chain, requiring understanding of the relationship between signals.
The fourth paragraph concerns inspection and inventory, procedures and person-to-person clearance, and the two approaches differ. The program shut down the rules: Whether field is complete or whether each number in the answer is accompanied by an enforceable query and reruns out of the same value. Access re-verification requires judgement: the causal chain does not make sense, and similar candidates are not cleared; this step is examined first by Agent, who finally decides not to enter, and what one sees is an answer that has passed through the door, and again through Agent. After adoption, the data enter the assessment set.
The end point of this line is "STAROPs complete the assessed diagnosis": after the data enter the assessment, the evaluated Agent uses the same input for a diagnostic, and its output is compared to the standard answer one by one field. So every step ahead affects the score: failure does not really occur, standard answers are wrong; evidence is incomplete and the basis for the score is missing; inspection fails and wrong answers miscalculate the correct diagnosis.
Key issues for ongoing construction
Four core issues for building the assessment set
A single-process run is just the starting point. There are four practical questions to be addressed for continued construction and long-term use, each of which is answered later. The first is how to select the right topic, which is where it comes from and how it is designed, and is answered by the sections "Planning the scene by scope and difficulty" and "Online questions back into assessment collections." The second is how the environment is built: the assessment data are generated from the real environment, where Agent interacts with the environment, and there is nothing to assess from the real environment, which is answered by the section on “A needs-based extension of the environment”. The third is how to ensure the validity of the assessment set, which is answered by the section on the "Construction cycle based on failure injections". The fourth is how high-quality and continuous iteration is answered by the Quality Review section. These four issues depend on each other: environmental instability does not allow validity to be tested; the source of the subject matter is unregulated, and efficiency is limited to the types familiar with bulk production. They are therefore answered in a set of methods, which are presented below by title, environment, construction and monitoring.
Build Method and Verify
Agent Division in Assessment Build
These things used to be one-on-one, one-on-one, one-on-one, one-on-one, one-on-one, one-on-one, one-on-one, one-on-one; one-on-one, one-on-one, one-on-one, one-on-one, one-on-one; and one-on-one, one-on-one, one-on-one. The construction is broken into four sections, each of which corresponds to an earlier issue and is the responsibility of one Agent, each with four specific tasks. The topic is run by ScenarioAgent, which is responsible for gap identification, change analysis, structure reduction and site return: The gap identification is the search for uncovered cells against the capability map, the change analysis and architecture restoration, which was originally to refine the analytical capability to be validated from changes in the business systems and structures, and the on-site backflow, which was to return the problem online to the assessment data. The environment is the responsibility of EnvironmentAgent, which is to build up, imitate, inject and observe access, i.e. to tie up services on a demand-driven basis, to carry a real business chain, to keep requests running on the business chain, to connect indicators, logs, call chains and alarms, to be seen in case of failure.
The construction was carried out by BuildAgent, four steps of the construction cycle: failure injection, signal capture, evidence extraction, access review. Supervision is performed by SupervisorAgent for quality check-ups, invalidation, regression and difficulty steps: The quality check is five tests, the failure recognition is the drop-down of the data that cannot escape, the regression check is the re-testing of the data that Agent has recovered and is still valid, and the difficulty step is the increase of the difficulty when the score is close to full. Access reviews are also conducted by BuildAgent, which does not follow the build-up context and re-checks evidence, otherwise the review will follow the available answers.
The top one in the figure is the location of the person: target setting, acceptance check, final review. The goal is to determine which data to build: which capacity domains, which difficulties to fill, and which spaces to make first on the map. acceptance check determines what data is satisfactory: what field is required for standard answers and to what extent evidence is required. Final review is the final decision before the data enters the assessment. These three things are judged in the context of business and evaluation purposes and are left to others; the article-by-article, article-by-article, cross-checking of such duplication is assigned to Agent. People don't quit, but they don't comb.
The assessment is used to evaluate STAROPs, but the construction of the assessment is a complex Agent task. A data set cannot be done in isolation from scenario understanding, mission planning, environmental construction, failure implementation, tool call, observational forensics, causal recovery, standard answer construction, independent review and quality access. So is it possible to build a credible set of assessments automatically, and in turn to see an organization's Agent engineering capability: is there a capacity for a group of Agent to continue to do complex work in the real environment, and what is finally delivered is credible. From this point of view, it is also a stress test of Agent's ability and infrastructure maturity.
The stress test tested four aspects simultaneously. The first is Agent's own ability: to understand business and architecture, to use tools correctly, to reason on the basis of evidence, and to discover that there are problems with its own results. The second is the synergy between Agent: the subject, the environment, the construction, and monitoring of the roles require clear input output and state transmission, and the output of the former must become a reliable input for the latter. Third is the infrastructure that underpins them, which is often the most critical: whether tools are standardized, whether the environment can be rebuilt, how context and the state can be preserved, how long missions can be restored, how failures can be retry, how different players are connected and how evidence can be tracked. Fourth, governance and quality mechanisms: Agent himself said that it had completed the count, with procedural checks, evidence validation, independent review and manual final review. It is possible to do so with one Agent, several Agent, or Agent plus Skill, but the key is whether the organization builds these capacities, tools and environments to the extent that Agent is autonomous and synergistic.
Planning the scene by area of competence and difficulty
First paragraph: Where did the title come from? There are two sources of the issue, the first being the capacity coverage for proactive planning: based on its own business systems, the analytical capacity to streamline the need for validation, which is somewhat similar to the confusion engineering. The analysis is done by drawing on Agent's own field of knowledge, as well as the business and architecture it has.
This will be done by mapping the scene into a capability map by capacity area and difficulty. The cross-axis is the capability domain: code and change, application time, database, container, intermediate, cloud product, host, network. The vertical axis is difficult: single-point failure, transmission chains, multiple failures, alarm storms. The vertical axis is not a cross-cutting classification, and it indicates how much interference is to be removed in order to locate: a single point of failure, an abnormal line; to a warning storm, with more than a dozen signals ringing and the real root causes hidden inside.
Each cell on the map is a specific scene. The row of single-point failure, from code and change to network, is the error of configuration, internal error of the interface, slow query, repeated rebooting of the container, backlog of queues, CPU full, disk full, and bags thrown; the row of publish causing error, relying on timeout piles, lock waiting to cause timeout, node debarment, cache piercing the compressor, primary cut-off chain; the row of multiple malfunctions, which currently has publish folded with rate limit, rate limit folded with timeout; and the line of alarm storm is not yet specific, with the top-right mark on the top of the map being the next build.
A few comparisons show how difficult it is. In this column of the database, the slow query of a single point is an abnormal line; the lock to the previous cell is waiting for timeout, and the lock in the database is waiting to be transmitted to the caller timeout, to be searched back along the chain. There is only one reason why the code is wrong with the configuration of the single point; the multiple malfunctions of publish are superheavy with rate limit, with two mechanisms working and separating them. So it's a one-size-fits-all, empty space that's not covered, and people will confirm the priority and give it to Agent.
The online problem is back to the assessment set.
The second source of the question is the real problem on the line. The capability map is planned for coverage, and the grids on the map are planned according to known capabilities and difficulty; problems on the line are not limited by this map, which may fall into an existing grid, as a genuine sample or as a mechanism not originally recorded in the map, so that a cell is added to the map to plan for subsequent construction.
The order of the road is very important: keep it safe, then restore it. When there is a problem on the line, the evidence is preserved by means of tools, including a trajectory, a call chain and a window of time at the time, followed by the restoration mechanism, replay and standard answers. Preserve first because evidence such as the call chain is kept for a short period and processed late, and this data cannot be retrieved.
Three steps after preservation. The mechanism was created by Agent to determine the causal link and the necessary conditions, i.e., under what conditions the failure would occur, and to recreate it as a scenario in the environment; the same error could be attributed to different reasons, depending on the phenomenon, which would not be right. Equivalence is done by means of tools and re-executed using the same mechanism; it is not enough to repeat the reported errors, so that the same reason is repeated in the assessment environment. Reconstruction certification is process-driven, and online and re-played signals are aligned: the sequence in which the signals appear reflects the sequence in which the malfunction spreads, the signals and the sequences are not matched, indicating that the re-established transmission paths and lines are different and have to be reversed. After validation is passed, the same set of criteria is given to BuildAgent to verify the answer, so that the data returned online and the constructed data are used: The same seven fields, the same evidentiary requirements, the same checks, are not compared together until they have entered the assessment set.
Extension of the assessment environment on demand
It's not until the title is set that you know what environment to build. Environmental expansion is triggered by coverage needs and is simply larger, and does not justify the creation of new environments: gaps in capability maps, new mechanisms on-line or when failure requires isolation. The top left corner of the figure describes the position of the person in this step: first, the determination of the capability area and the isolation of the border. Segregation is the same thing as "noise interference" in the construction cycle to avoid different malfunctions interfering in the same window of time.
The picture shows the evolution of the environment. The starting point is an open source microservice template, which drives a lower single link using scripts and can only be used for application layer single point failure. Each of the new types of malfunctions is then expanded, and the answer to what is to be covered by the failure is determined by the failure: when the host malfunction is added, there is an injection of the host layer and the host indicator is visible, otherwise the impact is not visible; when the cloud product is added, the cloud product is connected to its own observations and alarms, as such failure is first manifested in the cloud product. The two steps corresponded to the addition of two columns of mainframe, cloud products on the map of capacity.
When cross-clustering and multi-facility were combined, two clusters and cross-cluster calls had to be made, and the second cluster had been cloned, and the flow had been expanded from scripts to normal scenery, two layers, a step that corresponded to a more difficult slot. Normal flows keep the environment running as if it were a real system, with business being affected and signals only when a failure occurs; scene-based traffic drives a specific link to a specific scene, making the transmission path that needs to be verified a genuine request. The alarm threshold is applied back to the calibration using historical data so that the alarm triggers at that time and does not report when it should. There are now multiple clusters and business chains.
A five-step extension. Agent first reads dependencies and existing environments and determines which services, traffic paths and observations are to be filled and whether there is a need for segregation; then Agent organizes missions, tools are to be generated and deployed by templates, or clone clusters; then connects observations to two layers of traffic, establishes warning, calibration thresholds, and fixes reversible injection and recovery configurations; then the program checks the baseline without injection, and the normal flow must be clean and must be searchable, but checks to stop construction; after adoption, the output can reset the environmental configuration code.
The baseline checks are rigorous because when the baseline itself is abnormal, the signal generated by the injection is mixed with the original anomaly and it is not clear which was caused by the failure; when observations are not detected, the failure is not visible. The output ' s environmental configuration code covers scaling, operations, traffic, observations, and solid injection and recovery configurations, so that the same type of failure can be repeated at any time, requiring several sets of retraces without having to retrace them. For these environments to be stable in the long term, with constant traffic and close to real operations, and full observation access, priority is given to observations: without it, even the trueness of the malfunctions in the data cannot be ascertained.
Assessment and measurement cycle based on failure injection
It is only when the environment is in place that a single issue can be made. The task of BuildAgent is to have standard answers to each data item by item. Four things it does are a cycle: failure injection, signal capture, evidence extraction, access review. Access reviews were first reviewed by Agent, and ultimately the decision was taken to deny access to the assessment; no step was taken in four steps, and they were returned to re-work.
There are three paths to a malfunction. Reuse a well-established injection that is suitable for failures that have been built, injected and restored, and can be performed directly and repeatedly; reconfigure, suitable for faults caused by configuration, such as "configuration error" on the map; and recode, suitable for malfunctions that require errors in the program itself, such as "interface error". Either way, after the injection has been completed, the failure must be proved on a case-by-case basis and cannot be considered to have occurred because of the successful execution of the injection order.
There are six mandatory checks for signal catches. The source of the alarm is to see if the alarm was triggered by the object of the injection; the environment is to be changed, to see if there is any other change at the same time; the indicators are abnormal, to see if the underlying indicator is really abnormal; the connections are to be called to see if the anomaly is transmitted along the transfer chain; and the impact ranges are to see which upstream or downstream are affected. The most important thing to note is noise interference: if someone else's fault is running in the same window at the same time, the signal from the injection is not clear who it belongs to, and the data is directly disabled.
The evidence is extracted from the site by taking the injection point, the root causes, the transmission chain, the four dimensions of the impact, and then organizing it into a standard answer. The format of the standard answer determines what the assessment can find. Some standard answers only record the time of failure, the root causes and the cause of the failure, and this format does not record the causal chain: it is impossible to review the basis of judgement and to distinguish between similar candidates; the numbers are not subject to a fixed window of time to recheck, and only the names of the components are matched. So at this point, the standard answer is Agent organized into seven fields.
Because the entity has to write down specific resources or services, the rating is used to determine the exact location of Agent. The mechanism for recording the type of failure is verifiable and is not a superficial phenomenon. The chain separates causes, transmissions and impacts, and Agent points out where it is wrong when it considers a service on the transmission path as the root cause. The question of why the identifier excluded a similar candidate made it clear that the answer was not anything other than conclusions. The alarm anchor keeps the actual trigger event and the time window, which is used to contrast the previous time window. Key figures are attached to the original query and anyone can re-examine it. The last field is the scoring rule: a single field score can be duly reflected in an answer that is based on correctness, a communication chain error.
The four dimensions of the evidence extracted corresponded to those of field: injection points and root causes, which fell on the root entity and type of failure; transmission chains, which fell on the part of the causal chain that spreads; and impact areas, which fell on the part of the causal chain. Four dimensions are complete, and the skeleton of the answer is there; plus points of identification, alarm anchors and key numbers, the answer is one that can be verified on a case-by-case basis.
The figures must be independently verified. In one case, the error rate was about 25 per cent, and manual reading did not reveal anomalies; the error count was zero after the tool re-surveyed five consecutive time windows. There are two other types of numbers that look right: The query interface returns on limited-time and returns the number of bars cannot be considered as a total; the baseline window and the malfunction window have different lengths and the margin cannot be used directly. So the numbers have to be written together with the query and the time window, so they can be re-executed, so the reader can judge how it came from. The number of queries cannot be written and the standard answer cannot be obtained. The standard answer thus written is a set of facts that can be retested: each step is supported by evidence, and everyone can re-examine.
Quality check, size and next step
Quality check of assessment collections
Pre-entry verifications are valid and do not mean that they will remain valid. Environmental and observational configurations change, so the assessment is itself subject to continuous inspection. To the left of the figure is five checks: quality, to see whether the evidence is complete, seven fields are incomplete and there are no enforceable queries for each number; coverage, to see type and difficulty distribution, and which grids are empty against the capability map; differentiation, to see whether the difference between strength and weakness can be pulled; size, to see whether the lowest sample is sufficient and the total sample is sufficient to conclude; freshness, to see whether the sample runs through. Five of the checks are short-screened and cannot offset local problems with average points.
Freshness cannot only depend on how long the data are built. What determines whether a data can be used is whether the initial signal will be reproduced today: a change in basic mirrors, alarm rules or observational configurations, and the data just built yesterday may not be available. So freshness depends mainly on whether the sample runs again, and the age of the data is used only as a reference.
Below the figure are two scenarios. If the data fail, the lower shelf is processed: the initial signal is not recreated, the assessment is still on track, Agent is rated wrong on these questions and will be recorded as a capability problem, and R & D may fix a non-existent flaw. After Agent ' s restoration, it was measured on the data still in force to confirm whether the improvements were maintained; if the questions of failure were mixed in the data used, the change in fractions was not clear whether the repairs were made or whether the data were. So the question of whether it's data or Agent, then the question of whether it's down or back.
The previous example of an error rate of about 25 per cent became a rule: each number written into a standard answer must be accompanied by an enforceable query, a window of time and an entity, reruns the query by the program and returns it if the value and the answer are not consistent; the number that cannot be deleted directly. The results of the check-up are only re-runs, not the numbers.
The least intuitive of the five checks is the difference, which is the curve on the right. The left half is the original data, the two curves of power and power weakness climb together, and end up almost at the top, with the fractions close to full and weak; the middle half is the "problem"; and the right half is the harder data, with the strong curve slightly backward and moving upward, with the obvious fall of weak capacity and two lines falling apart.
It's been a long time since we've had a strong and weak model, and it doesn't mean who's stronger. The scores are close to the upper limited-time, and Agent capacity enhancement and data difficulty are expected to show the same results, and a single score cannot determine whether the release or the supplement data are available. This is done by proactively making it more difficult to compare: by replacing simple with a transmission chain, with multiple malfunctions, the storm scene can be built up and folded to allow the curves to be redivided; if the differentials re-emerge, it means that the original data are difficult to distinguish between capabilities. The fall in scores after the additional difficulty is that the title has become difficult and cannot be read directly as a capacity retreat. Additional inputs can be reduced if the gaps in available data remain open and re-emerged; either of the five drops can be added to that.
Several recent studies have been devoted to the examination of the assessment itself, corresponding to different items in the five inspections. In August 2026, PAIChecker reviewed 500 data from SWE-Bench Verified and found that more than one in a dozen mission descriptions and reference restorations were not in line with the quality; in June 2026, CORE-Bench, on dozens of revised missions, showed that the best-performing configurations were close to the full score, referring to the difference; in July 2026, a study determined how much data would need to be evaluated by the confidence zone, which was size; and in key word retrieval, ICML 2026, MMBench-Live, which took images from the top of the key word search and generated new data. The STAROPs themselves continue to undergo these five inspections.
STAROPs Scale
As of September, more than 1,400 assessment data had been accumulated, including entries reviewed and under construction. The review was valid because of the analysis of 838 articles, excluding construction in progress and phase-out, covering more than 100 types of malfunctions; the root cause was only 58 in March, increasing to more than 10 times the original level. There are more than 500 more: Agent takes the right number, but it also needs data to evaluate.
There are already many publicly available data sets that still have to be assessed for three reasons, as in previous sections. The first is that the rules of service scale-up and whistle-blowing are their own, and that the services in the open assembly are out of line with their own systems, so that the topics are laid according to the capabilities and difficulties of their own systems. The second is that the emerging mechanisms on the line are not in the public pool, and they are precisely the most deserving of comment and need to be matched by online problems. Thirdly, the fact that the available data are capable of running back immediately is easy to do by maintaining their own assessment, relying on environmental configuration codes and regular sampling reruns.
Cost input to build the assessment set
These accumulations require sustained investment. The figure is an indication of the structure of the input and is not a realistic cost curve, drawing the relative direction of the manual, the environment, the model using the three inputs, divided into three periods of time. The start-up phase is manual, starting with a process of constructing and validating a small number of scenes, with a new set of environments and few model calls. The environmental, flow, observation and prosecution phases are to be completed on a case-by-case basis, and the environmental inputs are to be quickly followed up; the sentences are to be determined by test runs and back-to-work regulations, and manual inputs are to be found at the top of this paragraph. Re-use the built path phase where the environment is able to rebuild, build steps and validate rules can be reused. The same scenario does not have to be handled manually from the head to the article-by-article to standard maintenance, abnormal treatment and quality clearance, and duplicated manual input retreats; the build side of the model calls with the build volume rising, and in this period more than manual.
In the end, the environment became one of the highest in three: clusters, flows and observations are running year by year, and the larger the coverage of this segment, the greater it will be. As a result, Agent was used, and total costs did not necessarily decrease: environmental operations, model calls and quality clearances were being spent on a continuous basis, and what really reduced was repetitive manual work.
Similar practices have been observed in public research. A study in June 2026 built an automated streaming line assessment mission, built, validated and evaluated automatically, performed manual quality clearance, and reported on the construction of an accepted task of approximately $40, or 1 to 2 hours. Another study in August 2026 also used the Agent streaming line to screen materials, assemble tasks, and experts were involved in the calibration of the rules of reference and rating. Both suggest that construction can be automated and manual; the cost structure itself has to be measured and not replicated.
Open-source Co-building
The figure shows three directions of openness. The open source is the gradual opening up of already established assessments: RCA 100, RCA 200 has been opened, and more data and assessment methods have been gradually opened. Together, more teams provide a real environment, access real scenes and field experience, and provide a source for subsequent construction. The reconstruction was to move the public narrative into the real environment and to complete the assessment. Co-building and reconstruction are still the direction, with no time for external commitment.
Beyond openness, construction itself is moving in a more automatic direction. Much of the work under way has been handed over to Agent, but when the next batch will be launched, it is still up to the people to initiate, evaluate and perform construction and verification, and manual access. next step is for the assessment side, Agent, based on map blanks, long-term approaches to the upper limit for certain categories, and new online developments.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.