作者：来自 Elastic Carlos Crespo
发现 agent 会原样重新运行每条已触发规则自身的 ES|QL 查询来检索日志，因此告警能够指出无法访问的服务，并附上可证明这一点的查询。
在你的数据所在之处构建和观测 人工智能 agent。从 Elastic agent 构建器（ Elastic Agent Builder ）开始。你也可以立即开始免费云试用（ free cloud trial ），或在你的本地计算机（ local machine ）上试用 Elastic。
两条 ES|QL 规则因同一次支付故障触发，而重大事件（ Significant Events ）功能只发出一条告警。它针对日志重新运行每条规则自身的查询，统计出过去一小时内有 1,156 条失败日志，并指出无法访问的是支付服务。告警中的每个数字都附有其背后的查询，因此你可以自行检查证据。本文是三篇系列文章的最后一篇。第一部分（ Part 1 ）对日志源进行采样并提取知识指标，即有关服务、 依赖关系 和故障模式的、有证据支撑的事实。第二部分（ Part 2 ）将这些知识指标转化为经过验证的 ES|QL 查询，并将其中最有力的查询转化为规则。本文介绍这些规则触发后会发生什么。
第二部分最后展示了一条查询：商店正常运行时没有匹配结果，支付开始失败后则匹配到 676 份文档。同一支付路径上的第二条查询也会触发。一次故障触发两条规则，仍会产生两条告警，而且两者都无法告诉你故障是否真实存在，或是否仍在持续。
变点检测（change point detection）会发现规则的行为发生了变化。随后，发现 agent 会针对日志重新运行每条发生变化的规则自身的 ES|QL 查询，衡量失败规模，将描述同一事件的规则归为一组，并评估事件的严重程度。分组是减少噪声的关键，也正因如此，两条支付规则最终产生一条告警，而不是两条。
贯穿本文的示例与第一部分和第二部分相同：在 Kubernetes 上运行的 OpenTelemetry 演示应用（ OpenTelemetry demo ），将容器日志发送到 logs.otel。在采集过程中，我们将结账服务连接支付服务的端口改为错误端口，并将负载生成器的用户数增加到原来的三倍，导致每笔订单都在支付环节失败。下文的检测结果和告警来自对该场景的一次重放。
代码与 模型 按照第一部分和第二部分所述的方式分工。代码决定哪些规则发生了变化、哪些检测结果会传递给 agent、告警处于什么状态，以及存储哪些内容。模型则判断日志是否证实规则检测到的问题、哪些规则应归为一组，以及事件有多严重。模型无法编辑规则的查询，因此不能改写查询来迎合自己的假设。它提出的每项判断都必须能够追溯到查询结果或知识指标（ Knowledge Indicator，KI ），也就是第一部分从日志源中提取的、有证据支撑的事实。
查询知识 KI 如何转化为 Kibana 告警规则
规则阶段：将第二部分中的查询知识指标转化为 Kibana 告警规则。
第二部分中的每个查询知识指标都是一条带有严重程度评分的 ES|QL 查询，其中最有力的查询会转化为规则。得分足够高的匹配查询会自动转化为 Kibana 告警规则（ Kibana alerting rule ），在 Kibana 的第二版告警框架中运行。其余查询仍作为知识指标 存储 ，你可以手动将其中任何一条转化为规则。
转化后的规则会按计划运行查询，并记录每次发现的结果。流水线的后续阶段监测的正是这些逐次累积的记录。在本次采集中，四条查询被转化为规则：两条均被评为关键级别的支付查询，以及两条分别监测邮件投递失败和 Kubernetes 重启的查询。
变点检测如何发现真正的变化
事件阶段：变点检测将规则的触发模式转化为检测结果。
随着时间推移，每条规则的结果会形成一个时间序列，问题也从 “这条规则是否匹配到了内容？” 变成 “这条规则的行为是否发生了变化？”
日志源可能整天都在记录同一种可容忍的错误。使用固定阈值，需要为每条规则设置一个事先无人知晓的基线，而第二部分生成的查询并不附带基线。因此，检测工作流（ workflow ）会将每条规则与其自身的近期历史进行比较。它将每条规则的时间序列交给 Elasticsearch 的 change_point 聚合（ change_point aggregation ），判断序列是否发生变化，例如突增、骤降、阶跃变化或趋势转变，并报告变化的显著程度。 工作流 按计划运行，将规则分散到多次运行中处理，确保每条规则都能定期检查，无需一次分析全部规则。
对规则的监测精细程度取决于第二部分赋予它的严重程度评分。关键级别规则采用较短的近期时间窗口和细粒度时间桶进行分析，以捕获短暂变化。其余规则则采用较长时间窗口和较粗粒度时间桶，以较慢的响应换取较不重要规则的更少误报。
当规则行为发生变化时，工作流会记录一条检测结果：哪条规则发生了变化、变化属于什么类型，以及变化有多显著。如果一条规则在整个故障期间始终保持同一状态，就只会产生一条检测结果，而不是每次检查都产生一条。
在本次采集中，两条支付规则在同一次检测运行中发生了相同的变化：
`
1. [2. {3. "rule_name": "Payment gRPC Connection Failure (frontend → payment)",4. "change_point_type": "spike",5. "p_value": 0.0000127,6. "severity_score": 857. },8. {9. "rule_name": "Payment Service gRPC Unavailable (connection refused or i/o timeout)",10. "change_point_type": "spike",11. "p_value": 0.0000127,12. "severity_score": 8513. }14. ]
`AI代码解释
两条查询在故障期间匹配到相同的 676 份文档，因此它们的时间序列同时突增。约为八万分之一的 p 值意味着，如此大的突增极不可能只是噪声。
代码决定哪些规则会产生检测结果，因此模型只会处理行为发生变化的规则。大多数规则都不会进入这一步：
检测结果会进入队列，等待发现 agent 处理，并根据严重程度和等待时长，优先处理最紧急的结果。只有 agent 报告处理完成后，检测结果才会被标记为完成。如果过程中任何环节失败，包括告警写入失败，对应规则都会留在队列中，等待下一次运行。
在本次采集中，第二部分中转化为规则的四条查询，有两条产生了检测结果，即两条支付规则。邮件和 Kubernetes 重启规则始终没有匹配到内容，因此从未进入 agent 的处理环节。第二部分中最精确的结账服务启动查询也没有进入这一环节。它的得分为 25，从未转化为规则，因此也从未触发。
发现 agent 如何将检测结果转化为重大事件告警
发现 agent 将检测结果转化为重大事件告警。
发现 agent（discovery agent）是一个 Elastic agent 构建器（ Elastic Agent Builder ） agent。它的初始消息只有这批检测结果，每条规则对应一行，包含规则名称、日志源、变化类型、变化的显著程度，以及该规则近期发生变化的频率。
变点只能说明计数发生了变化。agent 会根据日志源中的日志，以及第一部分（ Part 1 ）和第二部分（ Part 2 ）构建的信息，判断其余情况：规则所描述的故障是否正在发生、有多严重，以及是否仍在持续。
对于每条检测结果，agent 首先查找规则背后的查询知识指标，再通过它找到生成该查询所依据的特征知识指标。这些指标明确了规则涉及的服务、依赖关系和基础设施。
然后，它会重新运行规则的查询。用于确认的查询就是知识指标中存储的查询，只是在前面加上时间筛选条件，并要求返回最新的记录：
`
1. FROM logs.otel
2. | WHERE @timestamp >= ?_tstart AND @timestamp < ?_tend
3. | WHERE MATCH_PHRASE(body.text, "rpc error: code = Unavailable")
4. AND (MATCH_PHRASE(body.text, "connection refused") OR MATCH_PHRASE(body.text, "i/o timeout"))
5. | SORT @timestamp DESC
6. | LIMIT 5
`AI代码解释
与第二部分一样，查询显示数据流名称，并针对日志源的视图运行，?_tstart 和 ?_tend 将时间窗口限定为检测发生前 15 分钟至当前。agent 不得修改 WHERE、重命名字段或更改 FROM。第二部分已针对日志源验证该查询，并确保其中每个短语都有数据依据，因此原样运行它，就能检验规则所声称的情况。
随后，agent 会完整阅读返回的每一行。只有当日志行体现出知识指标所描述的服务、错误和故障机制时，才会被计入。如果某行虽然包含这些短语，却描述了其他情况，例如重试后成功，或调用了不同的服务，就不属于该规则所检测的故障。
如何衡量故障的影响范围、趋势和持续时间
运维人员要决定是否在凌晨 3 点起床处理问题，就需要知道故障有多严重。因此，对于每条返回日志确实显示故障的规则，agent 都会再运行三条查询。每条查询都保留规则的 FROM，设置各自的时间窗口，并复用规则查询中的两个部分：
- <scope> 选取该操作写入的所有日志行，无论成功还是失败，例如每次银行卡扣款调用。
- <failure> 是将日志行判定为失败的条件。对于支付规则，就是其中的 MATCH_PHRASE 子句。
过去一小时的影响范围：该操作的失败比例是多少，涉及多少个服务。
`
1. FROM logs.otel
2. | WHERE @timestamp >= ?_tstart AND @timestamp < ?_tend
3. | WHERE <scope>
4. | STATS affected = COUNT(*) WHERE <failure>, total = COUNT(*) BY service.name
5. | STATS failed = SUM(affected), total_lines = SUM(total),
6. services_hit = COUNT(*) WHERE affected > 0, services = COUNT(*),
7. top_failed = MAX(affected),
8. top_names = TOP(affected, 5, "desc", service.name) WHERE affected > 0
`AI代码解释
如果规则查询中没有能够单独限定该操作的范围，例如支付规则，agent 就会采用另一种形式，先按 <failure> 筛选，再报告数量，不计算比例：
`
1. FROM logs.otel
2. | WHERE @timestamp >= ?_tstart AND @timestamp < ?_tend
3. | WHERE <failure>
4. | STATS affected = COUNT(*) BY service.name
5. | STATS failed = SUM(affected), services_hit = COUNT(*),
6. top_failed = MAX(affected),
7. top_names = TOP(affected, 5, "desc", service.name)
`AI代码解释
过去 24 小时的趋势：失败计数如何变化，最多划分为 50 个时间桶。
`
1. FROM logs.otel
2. | WHERE @timestamp >= ?_tstart AND @timestamp < ?_tend
3. | WHERE <scope>
4. | WHERE <failure>
5. | STATS count = COUNT(*) BY bucket = BUCKET(@timestamp, 50, ?_tstart, ?_tend)
6. | SORT bucket ASC
`AI代码解释
最长 14 天内的首次和末次出现时间：故障何时开始，以及是否仍在发生。
`
1. FROM logs.otel
2. | WHERE @timestamp >= ?_tstart AND @timestamp < ?_tend
3. | WHERE <scope>
4. | WHERE <failure>
5. | STATS earliest = MIN(@timestamp), latest = MAX(@timestamp)
`AI代码解释
这三条查询都是聚合查询，因此无论日志源有多大，都只会返回少量记录。它们共同告诉 agent：
- 故障是否仍在发生。如果最近一次发生在一分钟前，且最后一个时间桶仍有完整的失败记录，就说明故障仍在持续。如果最近一次发生在一小时前，则说明故障已经结束，无论计数如何变化。
- 从何时开始。最早出现时间给出了故障至少已持续到何时的依据，因为故障至少从那个时间点起就已经存在。
- 规模多大，影响哪里。失败比例和受影响的服务数量，可以区分是某个服务产生大量噪声，还是所有调用方都遭遇了故障。如果规则查询没有能够单独限定该操作的范围，例如只匹配错误文本的支付查询，agent 就会报告失败计数。对于支付规则，它统计出过去一小时内有 1,156 条失败日志，全部来自前端，并注明未测量失败比例。该计数与第二部分中的 676 来自不同的运行和时间窗口，因此两个数字不同。
这些数字用于确定严重程度并过滤噪声。如果故障在 agent 查看之前就已结束、返回的日志行显示调用成功，或者匹配的日志行只是系统以稳定频率容忍的后台错误，测量结果就会揭示这些情况，不会发出告警。
Agent 会以固定格式将这些数字写入每个信号的描述，让运维人员每次都能按相同的结构阅读。下文“告警中会写入什么”一节中的告警展示了一个示例。
根据日志源的拓扑确定影响范围
最后一步依据核实，会确定故障影响超出规则查询范围后还延伸到了哪里。agent 仅使用已获取的特征知识指标，将每个已确认的故障定位到日志源拓扑中，并明确受影响的实体：发生故障的实体，以及故障波及的服务、依赖关系和基础设施。只有已确认的日志行和知识指标自身的证据提供支持时，才会列出某个实体。
支付查询关联了 frontend-payment-grpc、checkout-payment- grpc 和 grpc-js，这些都是第一部分中的稳定特征标识符，因此今天的拓扑与昨天的告警使用相同的实体标识。在本次采集中，agent 只列出了 frontend-payment-grpc。所有已确认的日志行都来自前端，因此结账服务的依赖关系虽然关联到该查询，却没有得到确认的证据支持。
减少告警噪声：将多个信号合并为同一事件
经过依据核实后，agent 会为每条规则得到一个_信号_，即该规则经过检查和测量的结果。接下来要判断哪些信号描述的是同一事件。两个信号可通过三种关系归属于同一事件：
- 拓扑匹配：它们影响相同的实体，例如同一个故障服务、同一条依赖关系，或同一目标的两个调用方。
- 症状匹配：它们体现相同的故障机制，例如针对同一目标出现相同错误，且影响表现一致。
- 级联关系：一个实体的故障可以沿依赖关系解释其他实体的问题。后端错误和调用方超时，是从两端观察到的同一次故障，即使日志行的内容不同。
分组具有传递性，因此，如果甲与乙属于一组，乙与丙属于一组，三者就属于同一事件。仅共享服务名称的两个信号仍会分开。
事件的影响范围会扩大，因此，在创建新事件之前，agent 会先检查仍然有效、处于活动或恢复中状态的重大事件。如果某个信号对应的规则已属于一个有效事件，那么故障消失时，就记录该规则已恢复；如果故障仍与同一事件相符，就延续该事件。如果信号来自事件尚未包含的规则，且其受影响实体出现在该事件的拓扑中，故障也符合该事件，就会加入其中。事件随之获得新证据，拓扑范围也会扩大。只有无法延续任何有效事件的信号，才会被分组为新事件。
第二部分中的两条支付规则在拓扑和症状上都匹配。它们都关联到 frontend-payment-grpc，两者确认的日志行也都表明支付服务是无法访问的目标，因此它们会合并为一个重大事件。这两条规则是捕获同一次故障的两种方式：一条匹配前端表示“银行卡扣款失败”的 failed to charge card 日志行，另一条匹配其底层的 gRPC 错误。第二部分保留了两条规则，因为它们是不同的查询；到了发现阶段，才识别出它们属于同一事件。如果一条监测结账服务下单操作的规则在一小时后开始触发，也会作为级联故障加入该事件，因为其失败的调用指向支付服务。
如何根据证据评估严重程度和置信度
agent 会根据事件所包含的所有信号的实测影响，评估整个重大事件的严重程度。规则的严重程度评分决定了检测阶段的监测精细程度及其在队列中的位置，在这里不参与评估。p 值、变化类型和告警数量也不参与。评估按以下等级自上而下进行，在证据满足的第一个等级处停止：
- critical（严重）：证据表明存在以下任一情况：
- 波及大多数用户的全局性故障；
- 多项不同的核心操作同时受阻；
- 拓扑中被其他实体依赖的某个受影响实体，其功能端到端受阻，已无法履行职责；
- 已确认的敏感数据泄露。
- high（高）：已确认受影响实体上的某项操作失败，出现调用被拒绝、超时、无法完成，或大范围性能下降。仅限于某个端点或查询路径的阻塞仍属于此级别，即使所有调用方都会遇到。
- medium（中）：证据显示存在已确认的轻微性能下降，或者始终无法确定操作确实失败，还是仅记录了错误。
- low（低）：证据确认已恢复，或确认没有影响。
已确认的故障，即使无法测量失败比例或影响范围，仍评为 high；只有证据尚无法确定操作是否失败时，才评为 medium。从 high 升至 critical，取决于故障阻塞了什么，例如多项核心操作，或其他实体依赖的某个实体。测得的失败比例以及故障是否仍在持续，决定证据满足哪个等级。当新信号加入尚未关闭的事件时，会结合新信号重新评估事件的严重程度。支付事件被评为 high。其 assessment_note 记录了原因：在已观测到的支付路径上，银行卡扣款失败，但未测量失败比例和更广泛的影响范围，也未确认其他核心操作受阻，因此没有依据将其提升至 critical。
置信度是一个独立评分，表示证据对 agent 解释的支持程度。它以规则触发模式变化的清晰程度为起点；当日志分别独立证实多个信号时，置信度会上升；存在证据缺口时则会下降，例如 agent 无法将某个原因明确关联到实体，或无法测量某项影响。重大事件的措辞会与置信度相匹配，因此，证据支持较弱的事件会被表述为一种可能的解释。支付事件的置信度为 0.65：两条查询相互印证了同一故障，但测得的数量是日志行数，而非尝试次数。
为什么告警生命周期由代码推导，而非模型决定
重大事件是第二版告警框架中的一条告警（ alert ），该框架也运行第二部分中的规则。它采用与框架中其他告警相同的状态：active（活动）、recovering（恢复中）和 inactive（非活动），这些状态由重大事件功能自身的代码设置：
- 当日志确认故障时，以 active 状态创建事件；只要其中任一信号仍显示故障，就保持活动状态。
- 当其背后的每条规则都再次显示正常时，转为 recovering。恢复中的事件仍然有效，因此故障再次出现时，同一事件会重新回到活动状态。
- 当运维人员关闭事件，或其背后的规则已不存在而导致事件关闭时，转为 inactive。此后故障再次发生，会创建一个新事件。
每次事件发生变化时，系统都会根据日志对各条规则所显示的情况推导状态。agent 无法设置状态，因此相同证据始终会得到相同状态。对于支付事件，两条规则都需要在后续产生检测结果，并在重新运行查询时确认故障已经消失。两者都恢复后，生命周期逻辑才会将事件转为恢复中。
哪些内容会写入告警，哪些会被拒绝
agent 提出重大事件，代码决定存储什么。在写入任何内容之前，写入流程会检查提案：
- 新事件必须包含一个由 agent 根据返回的日志行确认的信号，可以是规则所描述的故障，也可以是同时发现的具体错误；
- 延续事件时，必须指定一个已存在的事件；
- 如果新事件会与某个活动事件重复，就不会写入，而是返回该活动事件。
重大事件作为第二版告警框架中的告警存储在 .rule-events 索引中，与第二部分的规则写入同一索引。此后，它会获得该框架为所有告警提供的能力：
- 运维人员预期的字段，包括标题、状态和严重程度；
- 在所有展示告警的位置被列出和搜索；
- 通过操作策略路由至通知和工作流。
其生命周期状态由重大事件功能决定，如上一节所述，同时还会携带 agent 的推理说明。操作策略会像处理其他告警一样路由它。以下是发现流程写入 .rule-events 的事件，内容已节选：
`
1. {
2. "severity": "high",
3. "alert": { "status": "active" },
4. "data": {
5. "title": "Card charging — payment gRPC connection refused",
6. "symptom_hypothesis": "The payment service refuses the frontend's gRPC charge call on port 9999, preventing card charges from completing.",
7. "summary": "Frontend card-charge calls to the payment gRPC endpoint fail with code Unavailable and connection refused on port 9999. The failure is present now; 1,156 matching failure log lines were observed in the last hour, all from frontend. Attempt-level failure share and broader service or regional reach are not measured.",
8. "confidence": 0.65,
9. "signals": [
10. {
11. "type": "detection",
12. "description": "The frontend card-charge gRPC call logs code Unavailable with connection refused on port 9999 at the payment target. 1,156 matching failure log lines were observed in the last hour, from 1 affected service; total log lines and attempt-level failure share were not measured, and the latest sample still shows the failure.",
13. "metadata": { "rule_name": "Payment gRPC Connection Failure (frontend → payment)", "change_point_type": "spike" },
14. "evidence": { "result": "found", "esql_query": "FROM $.nightshift.sources.default.logs | WHERE @timestamp >= ?_tstart AND @timestamp < ?_tend | WHERE MATCH_PHRASE(body.text, \"failed to charge card\")" }
15. },
16. {
17. "type": "detection",
18. "metadata": { "rule_name": "Payment Service gRPC Unavailable (connection refused or i/o timeout)", "change_point_type": "spike" },
19. "evidence": { "result": "found", "esql_query": "FROM $.nightshift.sources.default.logs | WHERE @timestamp >= ?_tstart AND @timestamp < ?_tend | WHERE MATCH_PHRASE(body.text, \"rpc error: code = Unavailable\") AND (MATCH_PHRASE(body.text, \"connection refused\") OR MATCH_PHRASE(body.text, \"i/o timeout\"))" }
20. }
21. ],
22. "causal_features": [
23. { "feature_id": "payment", "type": "entity" },
24. { "feature_id": "frontend", "type": "entity" }
25. ],
26. "blast_radius": [
27. { "type": "dependency", "feature_id": "frontend-payment-grpc", "source": "frontend", "target": "payment", "protocol": "grpc" }
28. ]
29. }
30. }
`AI代码解释
存储的查询使用 FROM $.nightshift.sources.default.logs，即日志源的视图，而上文的讲解使用了数据流名称 logs.otel。causal_features 和 blast_radius 表示拓扑中受影响的实体：发生故障的实体，以及故障波及的服务、依赖关系和基础设施。每个信号都包含依据核实阶段的测量结果及生成这些结果的查询，因此告警中的每个数字都可以通过重新运行查询来验证。假设会指出具体机制，例如“支付服务在 9999 端口拒绝前端发起的 gRPC 扣款调用”，而 agent 只有在已确认的信号或拓扑包含某个实体时，才能将其列为受影响实体。
从计数变化到告警存储，整个流程的分工如下：
如何评估发现 agent
发现评估套件会重放故障场景快照，将每个场景产生的检测结果提供给 agent，并对其写入的内容评分。代码评估器会根据预期事件，检查分组、受影响实体、严重程度和状态。它们还会检查 agent 是否调用了流程要求的工具：确认查询、知识指标查找，以及现有告警搜索。延续评估套件会在多个周期中提供同一事件，检查 agent 是否持续更新同一条告警，并保持严重程度和拓扑稳定。大语言模型评判器则评估描述内容，以及置信度和严重程度是否与证据相符。
评估结果：七种故障场景下的重大事件发现
下表展示了评估套件的一次运行结果，每个评估器的评分范围为 0 到 1。由于仅运行一次，且样本量为 7（拓扑相关行的样本量为 2 或 3），无法估计方差，因此应将 1.00 分视为冒烟测试结果，而非基准测试结论。
方法说明：使用来自 Anthos 银行（ Bank of Anthos ）的七种故障场景。该示例应用独立于本系列其他
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
By Elastic Carlos Crespo
It was found that angent would re-run every ES|QL query that triggers the rule itself to retrieve the log, so that the alarm can point to services that cannot be accessed and attach a query to prove this.
Build and observe artificial intelligence where your data is located. Start with Elastic Agent Builder. You can also start free cloud trials immediately or on your local computer.
The ES|QL rules were triggered by the same payment failure, while the Significant Events function issued only one warning. In response to a query about the re-run of each rule, it counted 1,156 failed logs in the past hour and noted that it was payment services that could not be accessed. Every number in the alarm is accompanied by a search behind it, so you can check the evidence yourself. This is the last of the three series. Part I (Part 1) samples log sources and extracts knowledge indicators, i.e. evidence-supported facts about services, dependency and failure patterns. Part II (Part 2) transforms these knowledge indicators into validated ES|QL queries and the most powerful of them into rules. This paper describes what happens when these rules are triggered.
The second part concludes with a query: there is no match when the store is running properly, and 676 documents are matched when payment starts to fail. A second query on the same payment path can also be triggered. A single failure triggers two rules, which still give rise to two alarms, and neither of which can tell you whether the failure is real or whether it continues.
Change point detection will reveal a change in the behaviour of the rule. Subsequently, it was found that angent would look at each rule that changed the log to rerun its own ES|QL query, measure the scale of the failure, group the rules describing the same event into a group and assess the severity of the event. Clustering is the key to reducing noise, and that is why the two rules of payment eventually produce a warning rather than two.
The cross-cutting examples are the same as those for Part I and Part II: OpenTelemetry Demonstration Application (OpenTelemetry demo), running on Kubernetes, sending the container log to logs.otel. During the collection process, we changed the port at which the billing service was connected to the payment service to the wrong port and tripled the number of users of the load generator, resulting in each order failing at the payment chain. The results of the tests and the warnings below are from a replay of the scene.
Codes and models are divided in the manner described in Parts I and II. The code determines which rules have been changed, which test results are passed to angent, what state the alarm is in, and what is stored. The model determines whether the log confirms the problems detected by the rules, which rules should be grouped into a group and how serious the events are. The model is unable to edit a rule query and therefore cannot rewrite the query to fit its own assumptions. Each of its judgements must be traceable to the search results or knowledge indicators (Knowledge Indicator, KI), i.e. evidence-supported facts taken from the log sources in the first part.
Ask for knowledge KI how to translate Kibana 's alarm rules
Rule stage: Translating the reference knowledge indicators in Part II into Kibana alarm rules.
Each reference knowledge indicator in Part II is an ES|QL query with a rating of severity, the most powerful of which translates into rules. A matching query with a sufficiently high score will automatically be translated into Kibana alert rule, running within the framework of Kibana's second edition. The rest of the queries are still stored as knowledge indicators, and you can manually translate any of them into rules.
The converted rules run the query as planned and record the results of each discovery. It is these cumulative records that are monitored in the subsequent phases of the current line. In this collection, four queries were converted into rules: both were rated critical-level payment queries, and two to monitor failed mail delivery and Kubernetes restart, respectively.
How the mutation detects real change
Event stage: The mutation point test converts the trigger mode of the rule into the test result.
Over time, each rule results in a time series, and the question goes from “Does the rule match the content?” to “Did the conduct of the rule change?”
The source of the log may be recording the same intolerable error all day. The use of a fixed threshold would require the setting of a previously unknown baseline for each rule, while the queries generated in Part II would not be accompanied by a baseline. Thus, the test of workflow (workflow) compares each rule with its own recent history. It assigns the time series of each rule to Elasticsearch for change point aggregation, to determine whether or not the sequence has changed, such as surges, drops, steps or trend shifts, and to report on the extent of the change. workflow operates as planned, spreading the rules over multiple operations and ensuring that each rule is checked periodically without having to analyse the entire rule.
The level of detail in monitoring the rules depends on the severity rating given to it in Part Two. The key-level rule uses shorter-term windows and fine-particle time drums for analysis to capture brief variations. The remaining rules use longer time windows and thicker particlescale time drums in exchange for slower responses in exchange for fewer misstatements of less important rules.
When the behaviour of the rules changes, workflow records a test result: which rule changes, what type changes are, and how significant. If a rule remained in the same state throughout the failure period, only one test result would be produced, not one per inspection.
In this collection, the same changes occurred in two payment rules during the same detection operation:
`
1. [2. {3. "rule_name": "Payment gRPC Connection Failure (frontend → payment)",4. "change_point_type": "spike",5. "p_value": 0.0000127,6. "severity_score": 857. },8. {9. "rule_name": "Payment Service gRPC Unavailable (connection refused or i/o timeout)",10. "change_point_type": "spike",11. "p_value": 0.0000127,12. "severity_score": 8513. }14. ]
`AI code explanation![] (https://csdnimg.cn/release/blogv2/dist/pc/img/runCode/coon-arrowwhite.png)
The two queries matched the same 676 documents during the malfunction, so their time series surged simultaneously. The p value of about 1 in 80,000 means that such a dramatic increase is unlikely to be just noise.
The code determines which rules will produce the test results, so the model will only deal with rules that change behaviour. Most rules will not go this far:
The test results will enter the queue and wait for the discovery of angent treatment, with priority given to the most urgent results, depending on the severity and length of the waiting period. The results will only be marked as complete when the agent report is processed. If any part of the process fails, including the failure to write the alarm, the corresponding rule will remain in line for the next operation.
In this collection, the four queries in Part Two that were transformed into rules resulted in two test results, namely two payment rules. The mail and Kubernetes restart rules never match content and therefore never enter the process of an individual. The most accurate billing service start-up query in Part II was also not entered. It's split 25, never translated into rules, and therefore never triggered.
We found out how angent turned the results into a major incident report. Police
It was found that angent turned the results into a major incident alarm.
Discovery anent was found to be an Elastic anent builder. Its initial information is limited to the results of this series of tests, with each rule corresponding to a row containing the name of the rule, the source of the log, the type of change, the degree of visibility of the change, and the frequency with which the rule has changed in recent times.
The change points only indicate a change in the count. On the basis of the logs in the log sources, as well as the information built in Part I (Part 1) and Part II (Part 2), angent will determine whether the failure described in the rules is occurring, how serious and whether it continues.
For each test result, angent first looks for the search knowledge indicators behind the rule, and through it finds the characteristic knowledge indicators on which the query is based. These indicators identify the services, dependencies and infrastructure to be covered by the rules.
Then it reruns the rule query. Queries used for confirmation are those stored in knowledge indicators, with only the time-screening conditions attached and the requirement to return the latest records:
`
1. FROM logs.otel
2. | WHERE @timestamp >= ?_tstart AND @timestamp < ?_tend
3. | WHERE MATCH_PHRASE(body.text, "rpc error: code = Unavailable")
4. AND (MATCH_PHRASE(body.text, "connection refused") OR MATCH_PHRASE(body.text, "i/o timeout"))
5. | SORT @timestamp DESC
6. | LIMIT 5
`AI code explanation
As in Part II, the query displays the data stream name and runs a view of the log source, tstart and tend limiting the time window to 15 minutes before detection to the current. Angent may not modify WHERE, rename field or change FROM. The second part already validates the query in terms of log sources and ensures that each phrase has a data basis, so that it runs as it is, so that it can test the situation alleged in the rules.
Then angent will read all the lines back. Log lines are included only if they reflect the service, error and failure mechanisms described in the knowledge indicators. If a line, although containing these phrases, describes other circumstances, such as the success of retry or the transfer of different services, it is not the failure detected by the rule.
How to measure the extent, trends and duration of the impact of the malfunction
To decide whether or not to get up at 3 a.m. to deal with the problem, the crew needs to know how badly the problem is. Therefore, angent runs three more queries for every return log that does show a malfunction. Each query retains the FROM of the rule, sets its own time window and repeats two parts of the rule query:
- <scope> selects all log lines written by the operation, whether successful or unsuccessful, for example, every bank card debit call.
- <failure> is a condition for a log line to fail. For the payment rules, it is the MATCH PHRASE clause.
Scope of impact over the past hour: What is the percentage of failure of the operation and how many services are involved.
`
1. FROM logs.otel
2. | WHERE @timestamp >= ?_tstart AND @timestamp < ?_tend
3. | WHERE <scope>
4. | STATS affected = COUNT(*) WHERE <failure>, total = COUNT(*) BY service.name
5. | STATS failed = SUM(affected), total_lines = SUM(total),
6. services_hit = COUNT(*) WHERE affected > 0, services = COUNT(*),
7. top_failed = MAX(affected),
8. top_names = TOP(affected, 5, "desc", service.name) WHERE affected > 0
`AI code explanation
If the rules query does not contain a separate limit on the scope of the operation, such as the payment rules, then angent will take another form, filtering first on <failure> and then reporting the quantity, without proportionality:
`
1. FROM logs.otel
2. | WHERE @timestamp >= ?_tstart AND @timestamp < ?_tend
3. | WHERE <failure>
4. | STATS affected = COUNT(*) BY service.name
5. | STATS failed = SUM(affected), services_hit = COUNT(*),
6. top_failed = MAX(affected),
7. top_names = TOP(affected, 5, "desc", service.name)
`AI code explanation
Trends over the past 24 hours: How the number of failures changes, divided into up to 50 time barrels.
`
1. FROM logs.otel
2. | WHERE @timestamp >= ?_tstart AND @timestamp < ?_tend
3. | WHERE <scope>
4. | WHERE <failure>
5. | STATS count = COUNT(*) BY bucket = BUCKET(@timestamp, 50, ?_tstart, ?_tend)
6. | SORT bucket ASC
`AI code explanation
First and last occurrence within a maximum of 14 days: when the malfunction began and whether it was still occurring.
`
1. FROM logs.otel
2. | WHERE @timestamp >= ?_tstart AND @timestamp < ?_tend
3. | WHERE <scope>
4. | WHERE <failure>
5. | STATS earliest = MIN(@timestamp), latest = MAX(@timestamp)
`AI code explanation
All three queries are aggregate queries and therefore, regardless of the source of the log, only a small number of records are returned. Together they tell angent:
- Is the failure still happening. If the last occurred a minute ago and the last barrel still has a complete record of failure, the failure continues. If the latest occurred an hour ago, the failure was terminated, regardless of the change in the count.
- Since when. The earliest time of occurrence gave a basis for at least the duration of the failure, which had existed at least from that point in time.
- The size, where it affects. The proportion of failure and the number of services affected can be distinguished from whether a service produces a high level of noise or all callors suffer from malfunctions. If the rule query does not limit the operation separately, e.g. payment queries that match only the wrong text, angent reports failure counts. For payment rules, it counts 1,156 failure logs in the past hour, all of which are from the front end, and indicates the percentage of failure not measured. This number is different from the 676 in Part II, which comes from different running and time windows, so there are two different numbers.
These figures are used to determine severity and filter noise. If the failure occurs before angent view, the returned log line shows the success of the call, or the matching log line is only a backstage error that the system tolerates with a stable frequency, the results of the measurements reveal these circumstances and do not send a warning.
Agent will write these figures into the description of each signal in a fixed format, so that carriers can read them in the same structure each time. An example is given in the section “What to write in the alarm” below.
To determine the range of impacts based on the scale-up of log sources
The last step, based on verification, will determine where the impact of the failure goes beyond the rule query. Angent uses only the acquired characterization knowledge indicators to locate each identified malfunction into log source and to identify the affected entity: the entity where the failure occurred, as well as the services, dependencies and infrastructure affected by the failure. An entity will be listed only if confirmed log lines and knowledge indicators are supported by their own evidence.
The payment queries relate to frontend-payment-grpc, checkout-payment-grpc and grpc-js, all of which are stable identifiers in the first part, so today ' s pop-up uses the same physical identifiers as yesterday ' s alarm. In this collection, angent only listed Frontend-payment-grpc. All confirmed log lines are from the front end, and therefore the dependency of the closing service, although linked to the query, is not supported by confirmed evidence.
Reduce alarm noise: merge multiple signals into the same event
Upon verification on the basis, angent receives a signal for each rule that the rule has been checked and measured. The next step is to judge which signals describe the same event. Two signals can be attributed to the same event through three relationships:
- Tunnel matching: they affect the same entity, such as the same failure service, the same dependency relationship, or two callers for the same objective.
- Symptoms match: they reflect the same failure mechanisms, such as the same errors made against the same target, and the same effects are expressed.
- Cascading relationships: failure of an entity can explain the problems of other entities along the lines of dependency. Backend error and caller timeout are the same failures observed from both ends, even if the log lines differ in content.
The subgroups are transmissible, so if A and B are in group and B and C are in group, they are in group. Only two signals sharing service names will remain separate.
The impact of the event will be extended, so that before creating a new event, angent will first examine a significant event that is still active, active or recovering. If the corresponding rule for a signal is already a valid event, the failure is recorded as having resumed when it disappears; if the failure remains consistent with the same event, the event continues. If the signal comes from rules not yet contained in the event and the affected entity is present in the event, the failure is consistent with the event and will be added. The event was followed by new evidence, and the scope was widened. Only if the signals of any valid event cannot be sustained will they be grouped into new events.
The two payment rules in Part Two are compatible with the scale and symptoms. They all relate to frontend-payment-grpc, and both confirmed log lines also indicate that payment services are an unaccessible target, so they are merged into a major event. These two rules are two ways to capture the same failure: a failed to charge card log line that matches the front end, and a gRPC error that matches its bottom. Two rules were retained in Part Two because they were different inquiries; it was at the discovery stage that they were identified as belonging to the same event. If a rule to monitor the handling of the billing service orders starts in an hour, the event is added as a cascade failure because its failure point to payment services.
How to assess severity and credibility based on evidence degrees
Angent assesses the magnitude of the entire major event based on the measured impact of all the signals contained in the event. The severity of the rules determines the level of detail of the monitoring at the testing stage and its location in the queue, and is not assessed here. p Values, types of change and number of alerts are also excluded. The assessment is carried out on a top-down basis, according to the following levels, and ceases at the first level where the evidence is satisfied:
- Critical (serious): the evidence indicates either:
- Global failures affecting most users;
- Multiple different core operations are also hampered;
- An affected entity that has been relied upon by other entities and whose functions have been prevented from being able to perform its duties;
- Confirmed disclosure of sensitive data.
- High (high): an operation in the affected entity has been identified as having failed, with a call denied, timeout, unable to complete, or a wide reduction in performance. Blocks limited to a certain endpoint or query path remain at this level, even if all callers encounter them.
- Mediam (C): The evidence shows that there was a confirmed slight decline in performance or that it was always impossible to determine whether the operation had failed or only recorded errors.
- Low (low): the evidence confirms that it has recovered or that it has no impact.
The identified failure was rated as high even if it was not possible to measure the failure rate or the extent of the impact; it was only when the evidence was uncertain whether the operation had failed. From high to critical, depending on what is blocked by failure, such as multiple core operations or an entity on which other entities rely. The measured failure rate and whether the failure continues to determine which level the evidence meets. When a new signal joins an event that has not yet been closed, the severity of the event is reassessed in conjunction with the new signal. The payment event was rated high. Its access note records the reasons: on the observed payment path, the bank card debit failed, but failed to measure the failure rate and the wider impact, nor to confirm that other core operations were blocked, and therefore there was no basis for upgrading it to critical.
Confidence is an independent rating indicating the extent to which the evidence supports angent ' s interpretation. It starts with the clarity of the rule trigger mode change; confidence increases when the logs independently confirm multiple signals; when evidence gaps exist, for example, angent is unable to clearly link a cause to the entity or to measure an impact. The wording of a major event would match confidence, and therefore the evidence supporting a weaker event would be presented as a possible explanation. The confidence of the payment event is 0.65: the same failure is confirmed by two inquiries, but the number measured is the number of log lines, not the number of attempts.
Why the alarm life cycle is coded, not modeled.
The major event was an alert (alert) in the second edition of the police system, which also operates the rules in Part II. It uses the same states as other alarms in the framework: active (activity), recovering (recovering) and inactivity (inactive), which are set by the critical event function ' s own code:
- Create an event in an active state when the log confirms the failure; remain active as long as either signal still displays the failure.
- When every rule behind it shows normal again, it turns to recovering. The recovery is still in effect and the same event will return to active status when the failure recurs.
- Conversion to inactivated when the closure or the rules behind the closure have ceased to exist. Since then, failure has occurred again, creating a new event.
When each event changes, the system extrapolates the status of the rule according to the log. Angent cannot set the state, so the same evidence will always be given the same state. In the case of payment incidents, both rules require subsequent detection results and confirmation that the failure has disappeared at the time of re-run queries. When both are restored, the life cycle logic will shift events to recovery.
What's written in the alarm and what's rejected?
The code determines what to store. Before writing anything, writing processes check proposals:
- A new event must contain a signal confirmed by an individual on the basis of a returning log line, which may be a failure as described in the rules or a specific error simultaneously discovered;
- In order to continue an incident, an existing event must be designated;
- If a new incident duplicates an event, it will not be written, but will return to the event.
Major events are stored in the.rule-events index as a second edition of the alarm framework and are included in the same index as the rules of Part Two. Thereafter, it will acquire the capacity that the framework provides for all alerts:
- The anticipated field for transport personnel, including title, status and severity;
- Be listed and searched in all locations where the alarm is displayed;
- Operational strategy route to notification and workflow.
The state of life cycle is determined by the function of a major event, as described in the previous section, with the reasoning of an individual. The operational strategy will follow it as well as other whistleblowers. The following is an excerpt from the discovery of the process written to.rule-events:
`
1. {
2. "severity": "high",
3. "alert": { "status": "active" },
4. "data": {
5. "title": "Card charging — payment gRPC connection refused",
6. "symptom_hypothesis": "The payment service refuses the frontend's gRPC charge call on port 9999, preventing card charges from completing.",
7. "summary": "Frontend card-charge calls to the payment gRPC endpoint fail with code Unavailable and connection refused on port 9999. The failure is present now; 1,156 matching failure log lines were observed in the last hour, all from frontend. Attempt-level failure share and broader service or regional reach are not measured.",
8. "confidence": 0.65,
9. "signals": [
10. {
11. "type": "detection",
12. "description": "The frontend card-charge gRPC call logs code Unavailable with connection refused on port 9999 at the payment target. 1,156 matching failure log lines were observed in the last hour, from 1 affected service; total log lines and attempt-level failure share were not measured, and the latest sample still shows the failure.",
13. "metadata": { "rule_name": "Payment gRPC Connection Failure (frontend → payment)", "change_point_type": "spike" },
14. "evidence": { "result": "found", "esql_query": "FROM $.nightshift.sources.default.logs | WHERE @timestamp >= ?_tstart AND @timestamp < ?_tend | WHERE MATCH_PHRASE(body.text, \"failed to charge card\")" }
15. },
16. {
17. "type": "detection",
18. "metadata": { "rule_name": "Payment Service gRPC Unavailable (connection refused or i/o timeout)", "change_point_type": "spike" },
19. "evidence": { "result": "found", "esql_query": "FROM $.nightshift.sources.default.logs | WHERE @timestamp >= ?_tstart AND @timestamp < ?_tend | WHERE MATCH_PHRASE(body.text, \"rpc error: code = Unavailable\") AND (MATCH_PHRASE(body.text, \"connection refused\") OR MATCH_PHRASE(body.text, \"i/o timeout\"))" }
20. }
21. ],
22. "causal_features": [
23. { "feature_id": "payment", "type": "entity" },
24. { "feature_id": "frontend", "type": "entity" }
25. ],
26. "blast_radius": [
27. { "type": "dependency", "feature_id": "frontend-payment-grpc", "source": "frontend", "target": "payment", "protocol": "grpc" }
28. ]
29. }
30. }
`AI code explanation![] (https://csdnimg.cn/release/blogv2/dist/pc/img/runCode/coon-arrowwhite.png)
The stored query uses FROM $.nightship.sources.default.logs, a log source view, while the above description uses data stream names logs.otel. Causal features and blast radius indicate the entities affected by the expansion: the malfunctioning entity, as well as the services, dependency and infrastructure affected by the malfunction. Each signal contains a query based on the results of the measurements at the verification stage and the results generated, so each number in the alarm can be validated by reruning the query. Assumptions would indicate specific mechanisms, such as “payment services rejecting gRPC debit calls initiated at the front end of the 9999 port”, whereas angent could only be classified as an affected entity if the confirmed signal or zoom contained an entity.
The division of labour throughout the process, from counting to warning storage, is as follows:
How to evaluate the discovery
The evaluation package was found to replay the malfunction scene snapshot, providing the results from each scene to angent and scoring its contents. The code assessor will examine clusters, affected entities, severity and state in the light of anticipated events. They will also check whether angent has used the tools required by the process: confirmation of queries, search for knowledge indicators, and the existing alarm search. The continuous assessment package provides the same event over multiple cycles, checking whether angent continuously updates the same alarm, and maintains its severity and stability. A large-language model judge assesses the content of the description and whether confidence and severity are consistent with the evidence.
Assessment: Discovery of major incidents in seven scenarios of failure
The following table shows the results of the one-time operation of the evaluation package, with each evaluator rating ranging from 0 to 1. Since only one run and the sample volume is 7 (the sample volume of the pedestal-related line is 2 or 3), it is not possible to estimate the variance, and therefore the 1.00 score should be considered as the result of smoke test rather than the conclusion of the baseline test.
Methodological description: use seven scenarios from the Bank of Anthos Bank. The example should be used independently of the other in this series
Auto-fetched and translated by AI Atlas. All rights belong to the original author.