告警突然响起，错误率一路走高，接口延迟也在不断拉长。做运维的都清楚，难的从来不是发现异常，而是在海量指标、日志、调用链路和错综复杂的上下游依赖中，快速理清三个问题：问题发生在哪里？发生原因是什么？当前应该做什么？
火山引擎全栈可观测平台的深度调查能力，就是针对这类复杂排障场景设计的。用户可以直接从告警信息或者故障现象入手，平台会顺着线索自动开展排查，输出可复现验证的根因判断，附带对应的处置建议。整个排查路径、关键证据都完整留存，故障现象、排查过程、结论和证据还能整理成结构化资料，后续可以直接复用参考。
- 精准定位： 面对大量异常线索和复杂依赖，精准定位真正引发故障的对象与问题。
- 可回溯性： 调查路径、关键发现、判断依据和证实/排除结果全程可查看，结论可以逐步复核。
- 可复用性： 一次调查的场景、证据和结论可以沉淀下来，用于交接、复盘和后续同类故障排查。
01适合使用深度调查的三类典型排障痛点
结论不精准：
常规会话诊断时准时不准，难以稳定复现
常规会话诊断很吃大模型实时推理的效果，就算是同一个故障，换个时间、换一组上下文，哪怕上下文完全没变，模型给出的结论都有可能不一样。深度调查内置了一套完整的 RCA 校验框架，不再靠单次输出直接给答案，而是拆解成一套标准化、可核验的分步排查流程。系统会依次锁定排查范围、梳理异常表现、列出各类假设，再结合证据逐个确认或者排除猜想，把一次性的模型猜测，变成能够反复复现、核验的根因定位手段。
过程难回溯：
只有根因结论，看不清推理思路与依赖证据
大模型推理更像是一个“黑盒”：结论一旦出现偏差，运维人员很难了解模型查看的数据、中间判断过程，以及在哪一步脱离了实际情况。而深度调查则会完整记录调查对象、问题现象、假设推演以及证实或排除依据等过程，形成可以回看的完整排查链路，让每一条判断都有据可查，支持逐环节复核，方便快速找到分析出错的地方。
报告难复用：
成功案例停留在一次性排查，经验无法沉淀
如果问题、排查过程、结论和证据都只保存在单次对话里，就算这次顺利找到根因，经验也很难拆解留存和复用。若再次遇到同样问题，团队还得重新进行全流程分析。火山引擎全栈可观测平台的深度调查能力则能把业务场景、排查链路、关键证据、根因结论以及对应的处理建议，统一整理成结构化资料。经过验证的排查思路还能提炼成可直接运行的调查策略，方便团队复盘交接、沉淀排障知识，后续遇到同类问题，直接复用这套逻辑，接入自动化排查流程。
02使用深度调查，只需三步
第一步：从智能运维助手启动调查
登录火山引擎全栈可观测平台，打开智能运维助手。你可以直接基于现有告警启动分析，也可以手动描述故障现象，例如“网关请求耗时突然升高”或“服务错误率异常”。选择深度调查后即可开始排查。
在智能运维助手中使用
第二步：沿着调查进展查看关键发现
调查启动后，产品会围绕当前异常持续向下排查。你能清晰看到它正在分析哪些服务和上下游依赖、列出了哪些可疑原因、收集到了哪些佐证信息，还有哪些排查方向已经被证伪。全程不用来回跳转多个监控页面，也不用自己在一堆信息里拼凑线索。所有重点结论都会清晰说明：哪个对象出了问题、具体现象是什么、有哪些证据，以及这条排查思路是怎么推导出来的。
第三步：获得一份可复核的调查报告
调查完成后，平台会给出明确的故障对象、异常表现、关键证据和结论。值班人员可以沿着调查记录快速核对，也可以直接把结果交给研发、数据库或基础设施团队继续处置。
03真实场景：复杂依赖与干扰项并存
为了验证深度调查在真实条件下的表现，我们在测试集群中构造了一组典型微服务链路：入口 Gateway 依赖 Content Service、User Service，以及 MySQL、Redis 和一个长期存在异常的下游依赖。
测试分别在 Content Service → MySQL、User Service → Redis 链路上注入延迟或失败，同时保留另一条常态化异常链路作为干扰。产品需要做的不只是发现“哪里异常”，还要判断哪一处异常真正导致了本次告警，为什么这么判断。
04案例一：入口请求延迟升高
问题描述
2026-10-12 16:27（UTC+8），入口服务 opheus-playground-gateway 触发请求耗时升高告警。
- 问题对象： Gateway 入口服务。
- 问题现象： Gateway 请求延迟显著升高，pct99 约 563ms，发生告警。
- 发生时间： 2026-10-12 16:13–16:27（UTC+8）。
- 调查难点： 链路同时包含 Content Service、User Service、MySQL、Redis，以及长期存在异常的下游依赖。仅凭入口指标无法判断延迟来自入口服务、业务服务、数据库，还是常态干扰项。
启动调查
用户从告警进入智能运维助手并启动深度调查。深度调查会从入口服务开始查看调用关系，逐步聚焦异常路径；调查过程中同步展示当前关注对象、待验证方向和对应证据。
调查拓扑
假设与证据
调查结论
调查结果
深度调查沿 Gateway → Content Service → MySQL 调用链持续收敛，形成了可复核的根因结论：
- 根因概述： Content Service → MySQL 下游依赖异常，具体为 SELECT blog.articles 查询链路耗时飙升。
- 根因详情： 2026-10-12 16:27（UTC+8），Content Service 调用 MySQL 的延迟由约 1ms 基线升至约 506ms，直接导致 Content Service 响应达到 506–509ms，并逐级放大为 Gateway pct99 约 563ms、下游调用约 1.12s，最终触发入口请求耗时告警。
- 证据链： ① Gateway 指标显示 pct99 约 563ms；② Content Service 指标显示自身响应 506–509ms；③ Content Service → MySQL 指标显示平均/pct99 延迟约 506ms；④ Trace 中 SELECT blog.articles span 耗时约 542ms。四项证据发生在同一异常窗口，并沿调用方向完整解释延迟传播。
- 干扰项排除： User Service → Unknown 下游依赖属于长期常态异常，与本次告警的时间窗口、影响对象和传播路径不一致；MySQL CPU 约 0.75%，且无锁等待、慢查询或扫描突增，因此排除资源饱和、查询劣化、流量突变和变更引入等方向。
- 处置建议： 必要时先对 Content Service 限流、降级或切换实例；随后核查 VPC、路由、安全组、DNS 与连接策略，并为关键 SQL 建立延迟基线和告警。
05案例二：入口错误率升高
问题描述
2026-09-20 17:54（UTC+8），入口服务 opheus-playground-gateway 在短时间内出现错误率升高告警。
- 故障对象： Gateway 入口服务及其 Login 请求下游调用链。
- 发生时间： 2026-09-20 17:54（UTC+8），异常持续时间较短并很快恢复。
- 问题现象： Gateway 返回错误，User Service 的 Login 接口入口错误率升至 84%–100%；窗口内共出现 62 个 Login 错误，但请求流量没有同步增长。
- 调查难点： 短时异常容易在事后排查时丢失现场；同时链路中还存在 Content Service、数据库依赖和常态异常，必须区分本次新增故障与既有干扰项。
启动调查
用户直接以错误率告警启动深度调查。深度调查会围绕异常时间段查看服务指标与调用链，继续追踪到下游依赖，并把每个关键发现与证据放在同一条调查记录中。
调查拓扑
假设与证据
调查结果
深度调查沿 Gateway → User Service → Redis 调用链持续收敛，形成了可复核的根因结论：
- 根因概述： User Service → Redis 下游依赖失败，具体指向 Redis 实例 调用超时失败。
- 根因详情： User Service 处理 Login 请求时连接 Redis 发生约 5 秒 TCP 超时，导致 redis.dial 失败和 Login RPC 返回错误，随后向上传播为 Gateway HTTP 500 与入口错误率升高。
- 证据链： ① Gateway 错误 span 均落在 Login 调用 User Service 的链路；② User Service 入口错误率升至 84%–100%；③ 观测窗口内新增 62 个 Redis 错误 span，基线为 0；④ 62 个 Login 错误与 62 个 Redis 错误一一对应；⑤ Trace 显示 redis.dial 耗时约 5000ms 后超时失败。多项证据完整串联 Gateway → User Service → Redis 的失败传播路径。
- 干扰项排除： Gateway 与 User Service 均无流量突增、资源饱和或变更引入迹象；Content Service 下游异常与本次 Login 错误链路不一致，SQL 数据库故障也未获得 Trace 或指标支持，因此均被排除。
- 处置建议： 优先核查 Redis 实例可达性、故障转移状态，以及 User Service 所在 VPC、子网、安全组、网络 ACL 和 Redis 白名单；同时为 Redis 调用补充合理的超时、重试、降级与熔断策略，恢复后持续观察 Gateway 和 Login 错误率。
06效果对比：精准定位、可回溯、可复用
07让 AI 真正成为排障“可靠”搭档
对运维来说，靠谱的 AI 排障，并不是单纯更快输出一段文字。而是在故障突发、压力拉满、信息杂乱的时候，帮团队快速锁定真实问题，定位根因，给出可落地的处置建议，同时讲清楚这个结论是怎么推导出来的。深度调查把复杂故障分析，做成了一套可追溯、可复用的排查流程。从告警切入，顺着业务真实依赖逐层深挖，最终同时给到根因和对应的判断依据，让 AI 做排障，真正做到可信可靠。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
The alarm went off suddenly, the error rate went up and the interface was being lengthened. It is clear that the hard part is never to discover anomalies, but to quickly sort out three problems in big indicators, logs, telephone links and complex upstream and downstream dependence: where is the problem? What happened? What should we do now?
The in-depth survey capability of the Volcengine fully observable platform was designed to address such complex debris scenarios. Users can start directly with alarm messages or malfunctions, and the platform automatically follows a trail of searches, exporting the root causes of validation and accompanying disposal recommendations. The entire routing, key evidence, and failure, routing, conclusions and evidence are also structured and can be followed by direct reference.
- Precision: in the face of a large number of unusual leads and complex dependencies, precise targeting of objects and problems that actually cause trouble.
- Retroactivity: The path of the investigation, the key findings, the basis of judgement and the results of the validation/exclusion are available and the conclusions can be reviewed progressively.
- Reusable: The scenes, evidence and conclusions of an investigation can be deposited and used for handing over, double disking and subsequent type of failure screening.
01 for three types of typical disabling pain point for in-depth investigation
The conclusions are inaccurate:
It's hard to stabilize a regular session with a punctual diagnosis.
The normal session diagnosis is the result of the real-time reasoning of LLM, even if it is the same failure, and even if context remains intact, the model may give different conclusions. The in-depth survey contained a complete RCA verification framework, which no longer provided the answer directly by a single output, but was broken into a standardized, verifiable step-by-step process. In turn, the system locks in the scope of the check, combes the unusual performance, lists the assumptions, and, together with evidence-by-evidence confirmation or dispelling guessworks, turns the one-time model into a source locator that can be repeated and verified.
The process is difficult to trace:
It's only the root cause that doesn't understand the reasoning and the reliance on evidence.
LLM's reasoning is more like a “black box”: once the conclusions are biased, it is difficult for transport personnel to understand the data viewed by the model, the process of intermediate judgement, and the steps taken to depart from reality. The in-depth survey, on the other hand, records the process of the subject, problem phenomena, hypothetical evolution and proof or exclusion of the basis of the investigation in its entirety, creating a complete search chain that is reversible, allowing each judgement to be documented, supporting a chain-by-stage review, and facilitating the quick identification of errors in analysis.
The report is difficult to reuse:
The success story stays in one-off screening, and the experience doesn't sink.
If the problems, the process of vetting, the conclusions and the evidence are kept in a single conversation, even if the root causes are successfully identified this time, it is difficult to dismantle the retention and reuse of experience. If the same problem is encountered again, the team has to re-examine the full process. The in-depth survey capability of the Volcengine fully observable platform allows for the integration of operational scenarios, mapping links, key evidence, root cause findings and recommendations for response. The validated approach to screening also allows for the development of a directly operational investigative strategy that facilitates the team ' s ability to redisconnect and to develop knowledge of sedimentary barriers, and then to follow up on similar problems, using this logic directly and accessing automated screening processes.
02, use depth. Three steps.
Step one: Start an investigation from the smart travel assistant.
Login to the Volcengine All-Standable Platform to open the Smart Transport Assistant. You can start analysis directly on the basis of existing alarms, or you can manually describe malfunctions, such as “the sudden increase in gateway requests” or “aberration of service error rates”. Once an in-depth survey is selected, the search can start.
Use in smart mobility assistant
Step 2: Following the investigation to see key findings
After the investigation has been initiated, the product continues to be searched downwards around current anomalies. You can see clearly which services it is analysing, which reasons are suspicious, which supporting information is collected, and which other directions of scrutiny have been perjured. The whole process is not used to jump back and forth to multiple surveillance pages, or to collage clues with a bunch of information. All the key findings clearly indicate which target is in question, what the specific phenomenon is, what evidence is available and how this line of inquiry is derived.
Step 3: Access to a reviewable investigation report
Upon completion of the investigation, the platform will provide clear targets for failure, unusual performance, key evidence and conclusions. The watchman can quickly check along with the investigation records, or can direct the results to the R & D, database or infrastructure team for further disposal.
03 Real scene: complex dependence and interference side by side
In order to verify the performance of the in-depth survey under real conditions, we have constructed a typical microservice chain in the test cluster: the entrance Gateway relies on Content Service, User Service, and MySQL, Redis and a long-standing and abnormal downstream dependency.
The tests were delayed or failed by injections on the Content Service ' s MySQL, User Service ' s Redis chain, while retaining another regularized abnormal link as interference. What the product needs to do is not just find “where the anomaly is” but to determine which anomaly actually led to this alarm and why.
Case one, 04: Delayed increase in entry requests
Problem description
2026-10-12 16:27 (UTC+8), entrance service opheus-playground-gateway trigger request time-consuming alert.
- Subject: Gateway portal service.
- Problem phenomenon: Gateway's request was significantly raised, pct99 about 563 ms, and there was an alarm.
- Time of occurrence: 2026-10-12 16:13-16:27 (UTC+8).
- The difficulty of investigating: The link also includes the content Service, User Service, MySQL, Redis, and the persistence of abnormal downstream dependency. Access indicators alone do not determine whether delays come from access services, business services, databases or normal interference items.
Start the investigation.
Users move from alarms to intelligent mobility assistants and initiate in-depth investigations. The in-depth survey will look at calls from the entrance service and gradually focus on abnormal paths; during the investigation, the current subjects of concern, directions to be validated and corresponding evidence will be presented simultaneously.
I'm investigating Taku Po.
Assumptions and evidence
Findings
Findings
The in-depth investigation, which continued to shrink along the Gateway Contact Service, MySQL calls, led to a reviewable conclusion:
- A summary of the root cause: Content Service → MySQL downstream dependence anomaly, specifically SELECT blog.articles query links skyrocketing.
- For details: 2026-10-12 16:27 (UTC+8), the delay in calling MySQL was raised from approximately 1 ms baseline to about 506 ms, which directly led to the response of Content Service to 506-509 ms and was scaled up to Gateway pc 99 about 563 ms, or 1.12s, ultimately triggering the time-consuming request.
- Evidentiary chain: 1 Gateway indicator shows pc 99 approximately 563 ms; 2 Content Service indicator shows its own response 506 – 509 ms; 3 Content Service → MySQL indicator shows an average/pct99 delay of approximately 506 ms; 4 Trace Four evidences occurred in the same abnormal window and were fully explained in the direction of the call to delay dissemination.
- Disruption items are excluded: User Service → Unktown downstream dependence is a long-term normal anomaly that is inconsistent with the time window, impact object and transmission path of this alarm; MySQL CPU is approximately 0.75 per cent, and is subject to an unlocked waiting, slow searching or scanned surge, thus excluding the saturation of resources, poor queries, sudden flow changes and introduction of changes.
- Disposal proposal: first, if necessary, of examples of the degradation or switching of Content Service rate limit; then of the VPC, route, security unit, DNS and connection strategy, and the establishment of delayed baselines and alerts for key SQL.
Case two 05: Increased entry error rate
Problem description
2026-09-20 17:54 (UTC + 8), entrance service opheus-playground-gateway reports higher error rates within a short period of time.
- Bad target: Gateway portal service and its Login request downstream call chain.
- Time of occurrence: 2026-09-20 17:54 (UTC + 8), unusually short duration and recovery soon.
- Problem phenomenon: Gateway returned error, with an error rate of 84 per cent – 100 per cent for the Login interface of User Service; 62 Login errors occurred in the window, but the requested traffic did not increase simultaneously.
- The difficulty of investigating: it is very easy to lose the site in short periods of time during an ex post facto check; there are also links to Content Service, database dependence and normal abnormalities, and a distinction must be made between this new malfunction and the existing disturbance item.
Start the investigation.
Users have initiated in-depth investigations directly at the error rate. The in-depth survey will look at service indicators and call chains around unusual time periods, continue to track down downstream dependence and place each key finding in the same investigative record as the evidence.
I'm investigating Taku Po.
Assumptions and evidence
Findings
The in-depth investigation, which continued to accumulate along the Gateway →User Service →Redis call chain, led to the conclusion that the root causes for review were:
- Gene: User Service → Redis downstream dependence failed, specifically pointing to the Redis instance calling timeout failed.
- For details: User Service connects to Redis for approximately 5 seconds when processing Login requests, TCP timeout, resulting in the failure of redis.dial and the return error of Login RPC, and subsequent spread up to Gateway HTTP 500 with an increased error rate.
- Evidentiary chain: 1 Gateway error span falls on the link of Login calling User Service; 2 User Service entry error rate rises to 84 per cent - 100 per cent; 3 observation window adds 62 Redis errors span with a baseline of 0; 4 62 Login errors corresponded to 62 Redis errors; and 5 Trace shows timeout failed after redis.dial time-consuming approximately 5000ms. Multiple pieces of evidence are linked to the failed transmission path of the United States Service.
- Disruption item exclusion: Gateway and User Service have no sudden increase in traffic, saturation of resources or signs of introduction of changes; Content Service downstream anomalies are not consistent with this Login error link, and SQL database malfunctions are not supported by Trace or indicators and are therefore excluded.
- Disposal proposal: Prioritize verification of the availability of the Redis example, the failure state, and the VPC, subnet, safety group, network ACL and Redis white list where User Service is located; and call for Redis to supplement the reasonable timeout, retry, downgrading and smelting strategy to maintain continuous observation of Gateway and Login error rates.
Comparison of effects: precise location, traceability, reuse
07 makes AI a real "reliable" partner for the barrier.
In terms of mobility, a specter AI barrier is not simply a faster output. It is to help the team quickly lock up the real problems, locate the root causes, give recommendations for disposal that can be landed, and make it clear how this conclusion is derived from it. The in-depth survey resulted in complex failure analysis as a retroactive and reusable sorting process. From the alarm, from the bottom to the bottom following the real dependence of business, to the root cause and the corresponding basis of judgement, to the AI to be truly credible.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.