作者：任懿
Agentic SOC 是⼀个以多 Agent 协作为核⼼的安全运营产品。在建设过程中，我们逐步形成了由真实问题、统⼀评测、修改验证和⽣产反馈组成的产品改进闭环。随着 Agent 承担更多⼯作，团队也开始调整协作⽅式，让产品、安全、研发和测试围绕同⼀个客户结果共同推进。
本⽂结合这段实践，介绍评测如何驱动产品持续进化、⼈机协作的边界如何设定，以及团队如何通过统⼀⽬标、结果标准和责任分⼯，减少跨岗位交接，降低沟通损耗，把技术能⼒转化为实际交付效果。
本文已收录至开源项目 Alibaba AI Agent Handbook，您可在此处获得更佳的阅读体验：aliyun.github.io/ai-agent-ha…
从安全运营任务出发
Agentic SOC 是⼀个以多 Agent 协作为核⼼的安全运营产品。⾯对分散在多云、多产品中的告警，它⾸先需要把信息归并成可以调查、可以处置的安全事件，再由不同 Agent 分⼯完成调查、研判、响应与阻断，并⽀撑威胁狩猎和⽇常安全运营。
对⽤户来说，看到⼀份分析报告只是过程。更重要的是：发⽣了什么，证据在哪⾥，影响了哪些对象，接下来应该采取什么⾏动。报告、时间线和处置建议，需要共同回答这些问题。
图 1：Agentic SOC 将多源告警归并为安全事件，由多个 Agent 协作完成调查与响应。
因此，产品的改进也必须回到真实任务。线上暴露了什么问题，哪⼀种能⼒没有达到要求，修改后是否解决了问题，有没有影响原本正确的场景，都需要有可检查的依据。
我们逐渐形成了⼀条持续迭代的链路：⽣产问题形成失败样本，评测帮助定位能⼒差距，Agent 提交候选修改，验证与审批决定变更是否⽣效。 新的⽣产结果，再进⼊下⼀轮改进。
这条链路能否成⽴，取决于两个基础：有⼀套稳定的标准判断产品是否变好，也有⼈持续对最终结果负责。
一次角色识别错误，如何变成回归资产
在⼀次调查中，实际发⽣的是 A 攻击 B，报告却把两者的⻆⾊写反了。攻击者和受害者反转，会影响读者对整个事件的理解。更麻烦的是，如果⼏个模块各⾃做判断，即使报告中的⻆⾊被修正，时间线或处置建议仍可能沿⽤另⼀套结论。
排查发现，问题出在三个模块缺少共享判定。沿着这个原因，修复⽅案从“改正⼀段报告”扩展为⼀次结构化改造：统⼀⽣成攻击者、受害者及判定结果字段，再让报告、时间线和处置建议复⽤这些字段。
图 2：从错误识别、根因定位、实际修改，到复测、⽣效与持续回归，⼀次缺陷成为后续版本的验证依据。
复测时，我们重新运⾏原来的场景，检查 A 是否被识别为攻击者、B 是否被识别为受害者，同时核对三个模块的输出是否⼀致。这样才能确认，修改解决了造成错误的机制，⽽不只是修好了某⼀处表述。
⻆⾊字段与读取逻辑随后进⼊正式版本，对应回归⽤例也纳⼊评测基线。以后⽆论改模型、提示词还是规则，都需要继续⽤这个样本检查⻆⾊判定及其在各模块中的⼀致性。
⼀次修复留下两份结果：⽣效的产品能⼒，以及可以反复运⾏的验证样本。 后者使团队不必每次都依赖某个⼈记得这个问题，也让后续改进有了可以⽐较的起点。
这就是我们所说的产品⾃进化：真实问题持续进⼊改进流程，修改通过评测和⻛险判断后⽣效，失败经验成为下⼀轮的约束。它需要可追踪的修改、可靠的验证和清晰的发布边界。
用统一评测判断每次修改是否有效
修好⼀个案例，还不⾜以说明版本整体变好了。⼀次修改可能解决⻆⾊识别问题，却让其他场景的能⼒退化。因此，候选变更需要放到同⼀套基线中⽐较。
围绕安全调查任务，我们建设了 SOCBench。在这次分享所记录的阶段，它包含 211 个真实调查任务，覆盖 16 类事件，从 9 个维度评估 5 个调查阶段。题集、执⾏配置和评分规范都纳⼊版本管理，便于复现结果、⽐较版本。
图 3：评测覆盖不同事件类型和调查阶段，题集、配置与评分规范统⼀进⾏版本管理。
安全调查是⼀个⻓程任务，只检查最后的结论，会遗漏很多问题。我们还需要看环境定位是否准确、⼊⼝是否确认、攻击主链是否还原、影响范围是否得到检查，以及最终结论是否有证据⽀撑。
对应到能⼒评估，除了报告完整性，也要检查⽇志覆盖、时间准确、⼊⼝还原、进程链还原、影响扩展、证据质量、推理与定级，以及效率与稳定性。这样，评测才能说明“哪个环节需要改”，⽽不只是给出⼀个总分。
同时看调查质量和执⾏成本
⻓任务的效果还需要结合查询量与执⾏耗时来理解。查询更多，并不必然意味着调查更充分；耗时更短，也不代表任务已经完成。
图 4：左侧⽐较版本能⼒轮廓，右侧观察⻓任务中的查询调⽤与阶段得分。
分享中的⼀个⻓任务案例⾥，Agentic SOC-Latest 得到 86 分，进⾏了 43 次查询，耗时 28 分钟。对⽐项中，也有 93 分、95 次查询、48 分钟的结果。这个案例帮助我们观察质量、查询开销和时间之间的关系；它是单案例表现，不能直接作为全量任务的平均效果。
从候选修改到发布，我们关注的是：原问题是否复测通过，约定评测范围内是否出现退化，⻛险是否可接受。缺陷样本复测之外，还需要结合独⽴测试集评估、⽣产任务重放和灰度验证。未通过的修改退回调整，暴露出的失败样本继续沉淀到回归集中。
统⼀评测由此承担三项⼯作：定位弱项、验证改进、持续回归。 每⼀次变化都应当留下能够解释、复查和复⽤的结果。
让 Agent 在明确边界内自主工作
随着 Agent 能承担的步骤增加，⼈的参与⽅式也要变化。逐步指导每⼀次操作，容易让⼈变成任务分发和结果搬运的中间环节；完全放开，⼜⽆法满⾜安全产品对⽣产变更的要求。
我们的做法，是在任务开始前明确四类边界，让 Agent 在这些边界内完成分析、修改与验证。
图 5：⼈预先定义⽬标、质量、权限和停⽌条件，Agent 在约定范围内推进任务。
- ⽬标边界回答允许解决什么问题。任务⽬标、优先级、输⼊范围和预期产物需要明确，超出范围的⼯作需要重新确认。
- 质量边界回答什么状态才算完成。原问题必须复测通过，约定评测范围不得退化，证据不⾜时不能输出确定结论。
- 权限边界回答允许访问和改变什么。可⽤数据、⼯具和环境需要限定，⾼⻛险操作必须在预授权范围内，⽣产变化必须可追踪、可回滚。
- 停⽌边界回答什么时候必须转交⼈⼯。出现异常、越权、关键指标恶化，达到重试或资源消耗上限，或者⽆法继续作出可靠判断，都应触发暂停。
这些约束越清楚，Agent 可以⾃主推进的空间也越清楚。⼈的精⼒则集中在⽬标设定、关键判断和⻛险处置上。我们期待的演进路径，是从辅助执⾏到⾃主完成单步任务，再到边界内完成⾃进化闭环。
把跨岗位交接改成共同结果负责
产品内部开始围绕任务结果持续迭代后，组织中的另⼀种⼯作⽅式显得更加突出：产品完成需求，安全完成规则，研发完成实现，测试完成验收。每⼀段都有交付物，但客户的问题要等整条链路⾛完才算解决。
这时，仅仅提⾼某⼀岗位的执⾏速度，能压缩的总时间是有限的。⽬标在交接中被多次解释，不同岗位分别排队，验收集中在末端，问题⼜可能沿原路返回。
⽤⼀个需求的排期对⽐说明了这种差异。分段交接⽅式中，诉求澄清、设计、安全⼝径、后端、前端和集中验收依次展开，合计51 天。另⼀种⽅式先确定 12 条验收条件，由⼀个结果负责⼈贯穿全过程，把产品体验、⼯程交付和安全效果组织为并⾏任务，形成了 14 天的周期提升。
图 6：51 天的分段排期与 14 天的并⾏周期。
这个对⽐中，值得关注的是验收与责任的位置。原来要等到末端才集中判断的标准，被放到了任务开始时；原来随交付物逐段转移的责任，由⼀个 Owner，也就是结果负责⼈，从⽬标确认⼀直跟到结果验证。
围绕这种⼯作⽅式，可以把组织协作归纳为四个要求。
⾸先，统⼀客户结果。产品、安全和研发需要共同理解这次⼯作要改善什么，最终⽤什么真实使⽤效果判断完成。
其次，统⼀结果标准。指标和证据⼝径提前确定，避免各岗位分别完成⾃⼰的清单，却在最后才发现⽬标不⼀致。
第三，贯穿结果责任。负责⼈持续跟进问题、依赖和验证结果，推动任务收敛。并⾏⼯作可以由多个⼈和多个 Agent 承担，结果仍然需要有⼈负责。
最后，按⻛险安排专家介⼊。⾼⻛险和不可逆变更必须由专业⼈员把关，其他环节根据任务需要参与，减少⽆必要的等待。
这些要求也解释了为什么团队合并和岗位融合会成为重要⼀步。共享⼀个⽬标和完成标准，需要落实到⽇常分⼯中。
岗位融合以后，专业能力怎样发挥作用
在这次实践中，岗位变化主要表现为责任范围扩⼤：向前理解问题，向后跟进交付，同时借助 Agent 完成更多执⾏⼯作。
产品经理从 PRD 和原型继续向实现延伸，能够交付部分前端与可运⾏原型，⽤运⾏结果验证产品判断。安全⼯程师从专注安全能⼒，扩展到产品设计和实现，把专业经验转化为场景、判定标准与效果验证。研发⼯程师⼀⽅⾯向全栈闭环扩展，减少前后端分别排期和联调带来的等待；另⼀⽅⾯向客户与业务理解延伸，理解需求背后的路线、价值和商业影响。
图 7：产品、安全与研发扩⼤责任范围，具体由⼈还是 Agent 执⾏，取决于标准清晰度与⻛险。
这⾥需要把“能够多做⼀些⼯作”和“所有⼈都承担所有⼯作”区分开。对于标准明确、可以评测、可以回滚的任务，可以由 Agent 执⾏、⼈审批；对于跨领域但⻛险可控的⼯作，由结果负责⼈借助 Agent 完成闭环；⾼⻛险、⾼歧义或不可逆的⼯作，仍需要专业⼈员主导。
因此，专业能⼒仍然重要，只是越来越多地体现在定义问题、判断⽅案、验证效果和承担责任上。 安全⼯程师需要把经验表达成系统可执⾏、可检查的规则和代码；产品经理需要判断可运⾏结果是否解决真实问题；研发需要兼顾实现质量与⽣产可靠性。
三类⻆⾊也有共同需要增强的能⼒：理解业务和价值，以及调度 AI 完成复杂⼯作。任务如何拆分、哪些可以并⾏、需要提供什么信息、哪些结果必须复核，都会影响最终交付质量。
阶段性成果，以及仍要验证的事情
分享材料中的阶段统计显示，需求从创建到完成的平均周期由 50.9 天降⾄ 13.8 天，按这两个数值计算，缩短约 72.9%；窗⼝内吞吐率由 63.9% 提升⾄ 98.9%。
图 8：阶段性结果与后续验证⽅向。
接下来需要验证的事情也很明确：同样的机制能否从个别项⽬复制到更多模块；能否把依赖个⼈主动性的做法沉淀为团队的稳定能⼒；能否让共同结果标准进⼊⽇常评价，⽽不只在某⼀次项⽬推进时有效。
对于希望尝试类似实践的团队，可以先从⼀个真实问题开始：明确要交付的客户结果和验收证据，指定⼀个持续负责的⼈，运⾏完整个过程，再检查时间花在了哪⾥，哪些交接可以减少，哪些⻛险仍需要专业判断。
Agent 围绕任务结果持续进化，团队也需要围绕同⼀个客户结果共同⼯作。 这是我们在 Agentic SOC 实践中正在推进、也仍需继续验证的⽅向。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Submitted by: Ren Zheng
Agentic SOC is a secure operating product centred on multiAgent collaboration. In the course of construction, we have developed a product-improvement loop consisting of real problems, uniform assessment, revision of certification and production feedback. As Agent took on more work, the team began to adapt its collaborative approach to allow products, safety, research and development and testing to advance around the same client outcome.
In conjunction with this practice, this paper describes how assessments drive the continuous evolution of products, how boundaries for human collaboration are set, and how teams reduce cross-post transitions, reduce communication losses and translate technical capabilities into actual delivery results through harmonization of objectives, outcome standards and division of responsibilities.
This is recorded in the Open Source Project Alibaba AI Agent Handbook, where you can get better reading experience: ali io io......
From a secure operation.
Agentic SOC is a secure operating product centred on multiAgent collaboration. In the face of alarms scattered over clouds and products, it needs first to combine information into security incidents that can be investigated and dealt with, then to be investigated, studied, responded and blocked by different Agent divisions of labour, and to support the threat to hunting and daily safe operations.
For users, it is only a process to see an analysis. More importantly, what happened, where the evidence was, who was affected and what should be done next. The report, timeline and recommendations for disposal require a common response to these questions.
Figure 1: Agentic SOC returns multiple source alerts as security incidents, and multiple Agent collaborates to complete investigations and responses.
Therefore, product improvements must also return to real task. What problems are revealed online, which capabilities fall short of what is required, whether the problem is resolved after the modification, and whether there is an impact on the original correct scene, need to be checked.
We have developed a continuous and iterative link: failure in production Z.111Z, assessment of the gaps in positioning capacity, Agent's submission of candidate changes, validation and approval of the validity of changes. New production results, moving into the next round of improvement.
The establishment of this link depends on two foundations: a stable set of criteria to determine whether the product is better or not, and sustained accountability for the end result.
A role recognition error, how to turn into a return asset.
In one investigation, what actually happened was an attack on B, and the report reversed the color of both. The inversion of the attackers and victims affects the reader ' s understanding of the whole event. More troublesome is that if several modules are judged separately, a different set of conclusions may be followed in timelines or disposal proposals even if the color of the report is amended.
The screening found that the problem was that three modules lacked shared determinations. Along these lines, the rehabilitation programme was expanded from a “correction of a report” to a structural transformation: the unified generation of the attackers, the victims and the verdict field, and the re-use of field for reporting, timeline and disposal recommendations.
Figure 2: From error identification, rooting, actual modifications, to repeat, effective and continuous return, a defect became the basis for validation of the subsequent version.
When we rerun the scene, we check if A is identified as the assailant, B is identified as the victim, and the output of the three modules is checked for consistency. This would confirm that the modification resolved the mechanism that caused the error and not just fixed a certain expression.
The colored field and the read logic were then brought into the official version, and the corresponding regression examples were included in the assessment baseline. We'll need to use the model, prompt or the rules. This sample check the tan determination and its consistency in the modules.
A rehabilitation left two results: an effective product capacity and a validation of sample that can be repeated. The latter makes it unnecessary for teams to rely on one person each time to remember the issue and gives a comparable starting point for subsequent improvements.
This is what we call the self-evolution of products: real problems continue to move into a process of improvement, changes become effective through assessments and risk judgements, and failures become constraints in the next round. It requires traceable modifications, reliable validation and a clear publish border.
Use a uniform assessment to determine the validity of each change
Fixing a case is not enough to show that the whole version has changed. A change may solve the problem of colour recognition, but it degrades the capacity of other scenarios. Candidate changes therefore need to be compared in the same set of baselines.
We built SOCBench around the security investigation mission. At this stage of sharing the records, it consisted of 211 genuine investigation missions covering 16 types of events, from 9 dimensions to 5 investigation phases. Questions collections, implementation configurations and rating specifications are incorporated into the version management to facilitate the replication of results and comparison versions.
Figure 3: Review coverage of different types of events and investigation phases, and uniform version management of theme sets, configuration and rating specifications.
The security investigation is a short-term mission, examining only the final conclusions and leaving many issues out. We also need to see whether the environmental position is accurate, whether the entrance is confirmed, whether the main chain of attack is restored, whether the impact range is checked and whether the final conclusion is supported by evidence.
Corresponding to the assessment of capacity, in addition to reporting integrity, checks are made for log coverage, timing, entry reduction, process chain restoration, impact extension, quality of evidence, reasoning and rating, and efficiency and stability. In this way, the assessment would indicate “which part of the chain needs to be changed” rather than just giving a total score.
The quality of the survey and the cost of implementation
The effects of missions also need to be understood in the context of the amount of queries and the time spent on implementation. More inquiries do not necessarily mean that the investigation is more adequate; it is shorter and does not mean that the task has been completed.
Figure 4: left comparison of capability profiles, right observation of the call and phase of queries.
In one of the shared missions, Agentic SOC-Latest received 86 minutes and conducted 43 inquiries, which took 28 minutes. There are also 93 minutes, 95 queries, 48 minutes of results in the comparison. This case helps us to observe the relationship between quality, inquiry into spending and time; it is a single case and cannot be directly used as the average effect of the full task.
From the candidate changes to publish, we are concerned about whether the original problem has been repeated, whether degradation has occurred within the scope of the agreed assessment, and whether the insurance is acceptable. In addition to the sample repeat of the defect, there is a need to combine the assessment of the independent test set, the re-location of the production task and the greyscale validation. Unadopted modifications returned adjustments and revealed failures of sample continued to sink to return concentration.
As a result, three tasks were undertaken: targeting weaknesses, validation improvements and continuous return. Each change should leave behind results that can be explained, reviewed and reused.
Let Agent work on his own within a clear border.
As the steps that Agent can take have increased, the way people participate changes. The gradual direction of each operation makes it easier for people to become midway between task distribution and result handling; it is completely liberalized and does not meet the requirements of safety products for changes in production.
Our approach is to identify four categories of borders before the start of the mission and to allow Agent to complete analysis, modification and validation within those boundaries.
Figure 5: People predefined targets, quality, permission and conditions of cessation, Agent advanced tasks within the agreed limits.
- The target boundary answers what is allowed. Task objectives, priorities, input scope and expected product need to be clarified and work beyond scope need to be reconfirmed.
- Quality boundary answer what status is complete. The original problem must be re-examined, the agreed scope of assessment must not be degraded, and a definitive conclusion cannot be produced when insufficient evidence is available.
- permission border reply to allow access and change what. Available data, tools and environmental needs are limited, high-risk operations must be within pre-authorized limits and production changes must be traceable and roll back.
- When the boundary response must be handed over manually. The suspension should be triggered by anomalies, excesses, deterioration of key indicators, reaching the retry or resource consumption ceiling, or failure to continue to make reliable judgements.
The clearer these constraints, the clearer the space that Agent can move by himself. Human energy is focused on targeting, critical judgement and risk management. The evolutionary path we are looking for is from auxiliary implementation to autonomous completion of a one-step mission, to self-evolutional closure within borders.
Change responsibility for cross-hatting to common results
As the internal product begins to evolve around mission results, another way of working in the organization becomes more prominent: completion of product demand, safe completion rules, completion of research and development, and testing of acceptance check. Each segment is delivered, but the customer ' s problems will not be resolved until the entire link is completed.
At this point, the total amount of time that can be reduced by simply increasing the speed of implementation of a given post is limited. The target was explained on several occasions during the handover, with different positions lined up, acceptance check concentrated at the end, and the problem could return along the same path.
This difference is illustrated by a needs scheduling comparison. In the transfer of subparagraphs, the request is then made for clarification, design, security calibre, back end, front end and concentration of acceptance check, for a total of 51 days. In another way, 12 acceptance check conditions were first identified, with a result manager running through the entire process, organizing product experience, project delivery and safety effects into parallel missions, resulting in a 14-day cycle upgrade.
Figure 6: 51-day segmenting and 14-day parallel cycle.
Of concern in this comparison is the location of acceptance check and responsibility. The criteria, which had to wait until the end of the line, had been placed at the beginning of the task; the responsibility that had been transferred paragraph by paragraph with the delivery had been carried out by one Owner, the outcome holder, from target confirmation to outcome validation.
In this way of work, organizational collaboration can be grouped into four requirements.
First, harmonize client results. Product, safety and R & D require a common understanding of what the effort is to improve, and ultimately to judge with what real use results.
Second, harmonization of results standards. Indicators and evidence are determined in advance so as to avoid individual jobs completing their own lists, but at the end they are found to be inconsistent.
Thirdly, accountability for results cuts across the board. The principals continue to follow up on issues, rely on and validate results, thus contributing to the reduction of tasks. Parallel work can be carried out by multiple individuals and by multiple Agents, with the result that responsibility remains.
Finally, expert intervention is arranged on a risk basis. High risk and irreversible change must be handled by professionals, and other elements must be involved according to the requirements of the mandate, reducing unnecessary waiting.
These requirements also explain why team integration and job integration are important steps forward. Sharing one goal and completion criteria need to be implemented in the daily division of labour.
How does professional competence work when jobs are integrated?
In this practice, the change of position is mainly reflected in the broadening of the scope of responsibilities: to understand the issues forward, to follow up delivery, and to do more with Agent.
The product manager continues to extend from the PRD and prototypes to deliver some front-end and operational prototypes and validates product judgement with the results of the operation. The security engineers have expanded from a focus on safety capabilities to product design and realization, translating professional experience into scenes, determination standards and results validation. On the one hand, R & D engineers extend to the whole of the inn to reduce the waiting times associated with separate schedules and interconnections at the back and back end; on the other hand, extend to clients and operations to understand the routes, values and business implications behind demand.
Figure 7: Product, safety and R & D expand the scope of responsibility, whether implemented by people or by Agent, depending on standard clarity and risk insurance.
A distinction needs to be made between “can do more” and “all work for all”. For tasks that are clear, measurable and roll-backable, they can be carried out by and approved by Agent; for cross-cutting but controllable work, the outcome manager can use Agent to close the loop; and for high-risk, divisive or irreversible work, professional leadership is still required.
Professional capacity therefore remains important, albeit increasingly in terms of definitional issues, programme judgement, validation of effectiveness and accountability. Safety engineers need to arrive at system-enforceable, searchable rules and codes; product managers need to judge whether operational results solve real problems; and research and development needs to balance quality with production reliability.
The three categories of colour also have a common need for enhanced capabilities: understanding of operations and values, and movement control AI to perform complex tasks. The quality of the final delivery is affected by how split missions are conducted, what can be done in parallel, what information is required and what results must be reviewed.
Phased results, and things still to be verified
Phase statistics in shared materials show that the average period between creation and completion of demand decreased from 50.9 to 13.8 days, a reduction of about 72.9 per cent in both values; and the rate of ingestion in windows increased from 63.9 to 98.9 per cent.
Figure 8: Phase results and direction for subsequent validation.
The following is clear: The same mechanism could be replicated from individual projects to more modules; could rely on individual initiative be reduced to a team ' s stabilizing capacity; and could common outcome standards be allowed to move into day-to-day evaluation rather than being effective only when a particular project advanced.
For those who wish to try similar practices, a real question can begin with: clearly identifying the client results to be delivered and the evidence of acceptance check; designating a person responsible on an ongoing basis; running the whole process; checking where time is spent; which connections can be reduced; and which risks still require professional judgement.
Agent continues to evolve around mission results, and teams need to work together around the same client results. This is the direction that we are moving forward, and that still needs to be tested, in Agentic SOC practice.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.