AI 能让代码写得更快，但在「不能出错」的分布式存储系统里，如何确保提效不变成风险？本文结合美团基础技术部在 AI 评审、问题排查和自动化开发中的真实实践，分享 AI 能帮上什么忙、容易在哪些地方犯错，以及哪些地方仍需要人来把关。比「多写代码」更重要的，是让代码和结论经得起验证。希望这些经验能为复杂系统中的 AI 应用提供参考，在享受效率提升的同时，守住质量与稳定性的底线。
1. 背景：为什么分布式存储需要更审慎地使用 AI Coding
今年年初以来，随着Agent的普及，AI Coding 在功能开发、测试生成、脚本编写和执行、问题排查这些场景里都已经用得上了：读代码更快，实现和测试可以先让它写一版，研发之外的一部分杂活也能交给它去跑。
效率上去了，质量上的账也要重新算。同样的人力，一个版本里能交付的功能更多，改动的代码也更多，就算 AI 写的代码千行缺陷率和人写的持平，一个版本累计引入的缺陷绝对数量照样会涨。所以 AI Coding 的目标不能只是更快地写代码，提速的同时，代码质量和验证充分性还得比常规研发要求得更高。
分布式存储把这个问题放大了。代码规模大，模块之间关系复杂，一致性、并发、持久化、故障恢复和性能每一项都碰得到。一处看上去很局部的改动，平时可能一切正常，等到节点重启、磁盘异常、网络抖动或者高并发的时候才暴露出来。出了问题，受影响的也不会只是某一个功能，而是上层业务的数据可用性和系统稳定性，严重时还会波及数据正确性。存储服务支撑着大量上层业务，故障影响面大，修复也困难，对缺陷的容忍度极低。
要满足这样的质量要求，光让 AI 生成代码不够，AI 评审（Code Review）、AI 测试与验证、AI Debug 和受控自动化也得一起建起来，AI 生成的代码才能放心地进入研发流程。本文结合存储团队在 AI 评审、AI Debug 和 AI 自动化上的实践，讨论在分布式存储这类复杂度高、容错低的系统里，怎样一边用 AI 提速，一边靠更严格的评审、测试、调试和自动化验证守住稳定性与数据正确性，希望对其他类似系统落地 AI Coding 也有参考价值。
2. AI 评审（Code Review）：有益补充，但不能替代人对设计背景的判断
2.1 AI 评审擅长的地方
1. 能注意到跨文件一致性
例：ENABLE_UNITTEST 在三处调用点控制测试 mock 变体的编译分支，但 CMake 迁移把这个名字换成了 WITH_UNITTEST，旧名字再也没被赋过值。单看任何一个文件都挑不出毛病，这个 bug 只存在于文件和文件之间的落差里。
2. 能构造可执行证据
例：AI 指出“这里没做边界检查”之后没有停下，接着构造了一个声明 1 GiB 数值、实际包体只有 5 字节的畸形包，打到真实的解码入口，把崩溃现场抓了出来。
3. 会承认自己证明不了的部分
例：一处疑似的并发竞态，AI 只能靠代码走读找到可疑路径。它明确说，要证实这段交错时序需要 ThreadSanitizer，而当前的工具链跑不了，于是把这条标成“未验证”，没有直接定性为缺陷。
4. AI 对一些常见竞态问题识别率高
例：一处“先读校验、再写推进”、中间没有互斥的更新，多次评审都稳定地识别了出来，给出的修复方向也一致（合并成单个原子操作）。这类常见的并发模式识别得很稳，可以固化成固定检查项。
5. AI 不会漏掉 PR 中被删掉的代码
例：一次重构把某个接口合并成单次 RPC，顺手删掉了成功后清理本地缓存的旧逻辑。AI 指出这会留下一个短暂的时间窗口，期间可能读到过期的旧数据。而人对被删掉的行，本来就最不敏感。
6. AI 会结合存量代码评审增量改动
例：评审之前，AI 会先分析存量代码。改动较大的 PR 里，新代码可能顺手修掉了一个原有问题，人工评审容易把这种顺带修复当成新问题反馈；AI 则会把“原本就有、这次已顺手修掉”的问题单独列出来，并声明不作为问题上报。
7. AI 评审不会局限于代码层面的变更
例：某个 PR 里，三个仓库的构建文件互相用 feature 分支名引用对方，没有固定版本，合入顺序不对或者分支被清理，主干构建就断了。这属于“合并编排”层面的问题，只看 PR 页面的人恰好最容易漏掉，而 AI 把构建文件也当成了评审对象。
2.2 AI 评审容易出错的地方
AI 的误判多数是因为看不到代码之外的东西，比如运维流程、设计决策和团队约定。我们的实践中也看到了AI 评审在修复建议、信噪比、定级和证据取舍上的问题。
1. 代码可达 ≠ 运维可达
例：报告把一处过期 epoch 的代码路径标成了现实风险。人工复核时没有质疑这段代码本身，质疑的是周围的系统到底会不会产生走到这个分支的输入："EPOCH_NOT_MATCH 只会在迁移分裂+1，分裂之后老的region就不存在了，迁移之后再迁移，src也不存在了，所以 EPOCH_NOT_MATCH 理论上走不到这个分支。"
同样的情况也出现在被报告标成竞态窗口的一天过期时间的问题上：“迁移是人工操作，线上不会存在迁移发起一天还没有成功的，需要人工介入，一天的过期时间就是解决这个问题的”。AI 能建模那个计时器，建模不了包在它外面的人工流程。
2. 把和一般实践不同的有意为之当成了缺陷
例：报告把在生产环境里直接调用 abort() 的一致性检查标成了正确性风险，建议改成记录 Error 日志，不要崩溃。实际我们选择了abort，因为存储系统是正确性高于可用性的，在特定场景下，我们宁可 abort，也不能让这个请求继续而产生非常难处理的脏数据。
3. AI 评审附带的修复建议，本身也需要评审
例：AI 针对一个 32 位溢出给出的修复，是校验声明的 size 和实际收到的字节数是否一致。复核时发现 get_size() 返回的是 int，size 落在 2³¹ 到 2³² 之间会被截断成负数，这条校验本身就会静默失效。个例修好了，同一类问题并没有修好。
4. 廉价的“请确认”条目挤占了真正的信号
例：AI 评审把线程池默认值、重命名的配置项、被移除的二级缓存刷新开关都标成了“待确认”。每一条最后得到的回复，依据的都是团队自己压测得出的历史数据，和代码缺陷无关："根据压测调整的，后台线程数调低是为了降低对前台的影响。"
5. AI 评审的问题定级失真
AI 缺少一道界限明确的质量门槛：同一类理论上可能的问题，这次被标成最高级，下次又可能被标成不重要。拿到一份 AI 报告，定级没法完全信，为了保险只能把所有问题挨个审一遍，反而很难聚焦到真正高等级的问题上。
6. 把注释、头文件断言和设计文档当作证据
例：两份 AI 评审用的是同一个“场景 × 约束”框架，却给出了相反的结论。一份直接复述头文件里的断言判定实现是对的；另一份去核对了具体的实现，判定是错的。另一个例子是简单依赖设计文档，即迁移设计文档中要求的版本连续性校验，在代码里却没有任何引用。
两份评审看的代码没有区别，区别在于有没有去验证这些“说明”本身。
2.3 让评审结论可执行：构建测试、集成测试与运行时日志
静态评审只能说明“代码里存在这条路径”。这条路径是否真的会出错、已有的保护能不能拦住、线上有没有发生过，还需要另外三类证据来回答。
2.3.1 构建与单测：把推理变成红绿
实验：AI对一个项目全仓评审的 117 个疑似问题，逐条写 gtest 验证，测试在 master 上失败即视为确认。结果：36 条红测试确认，3 条被证伪（误报），41 条只能靠源码确认，37 条未验证。
- 判决确定。变红的测试集中在三类：输入触发 abort、错误被吞掉后仍然报成功、解码越界。严重度按失败现象来定就行，不用再看 AI 的措辞。
- 能证伪。有 3 条发现被测试推翻了，这是静态评审很难给出的结论。
- 能暴露 CI 自身的问题。比如测试报告里硬编码了 failures="0"，失败用例也不进汇总，所以要让 AI 自己构建、看原始输出，不能只信 CI 的绿灯。
- 边界。竞态和 UAF 需要 sanitizer，时序问题需要调度控制，故障注入点、受限主机、时钟偏移这些条件也还有缺口，“未验证”那份清单其实就是测试基础设施要补的课。变红的测试也只能证明代码会出错，证明不了线上会产生那个输入：有一个数据迁移场景，测试确认会有数据过期问题，人工复核后发现，在人工操作的流程里，这个过期恰恰是有意设计的。
2.3.2 贴近运行时的集成测试：主要价值是降误报
比如我们有一个项目的类实现做了特殊考虑，使得我们可以在单进程里就拉起真实的 Raft 多节点集群，全仓验证所有需要“生产集群”的场景。
- 让已有保护参与验证。一个 AI 发现的问题称 leader 切换会导致删除丢数据，而已有的一条集成测试恰好在删除窗口切主，结果证明是安全的。
- 裁决分歧。比如“快照能否捕获 split 中间态”，两次 AI 评审的意见相反，走真实的 change_peer 入口、加同步点注入写一个测试，就能直接判定。
fake/mock 的边界就是验证的边界。fake 要走真实入口、用生产配置，并写明替换了什么。
例：分布式 KV 的 hang 告警 Bug 没被测出来，是因为测试用的设置和实际生产发生的不同；网络阶段的丢弃保护在单测里被手工喂了时间戳，线上路径却从来不设置。
2.3.3 运行时日志：从“可能发生”到“已经发生”
实验：把一台存储节点 11 天的 WARN 日志（67.8 万行、34 个调用点）交给 AI 映射回源码，找到 5 个缺陷，对应的同学人工确认全部属实，没有误报。
- 发现静态审查不会怀疑的问题。例：“fail times 1”11 天只出现一次，说明计数器卡住了，hang 告警永远不会触发。
- 证伪错误假设。例：一个长达 19 分钟的响应，各阶段耗时之和与总耗时的比值恒为 1.0，由此排除了时钟错误，进而定位到发送阶段缺少丢弃保护。
- 让定级有依据。例：实测阶段耗时已经达到 1,156 秒，离存储这个耗时的 int32 溢出已经不到 2 倍。
其中两个缺陷并不是日志直接触发的，而是 AI 为了解释日志去读相邻代码时发现的，同样属实，日志提供的真实配置和数量级，可能也让相邻代码的推理更准。这只是一个样本，不能说明总能零误报，但足以支持一个做法：能拿到运行时日志时，评审和 Debug 都应附上。日志的局限是只覆盖执行过的路径，行号也可能与部署版本不符（需按消息文本定位）。
2.3.4 小结
这四层证据是互相补充的关系：静态评审找候选，测试判真伪，日志看现实中有没有发生。
2.4 使用 AI 评审的建议
- 结论按证据等级分四档输出，不要混排：测试确认（附一个在当前主干上失败的测试）、测试证伪、源码追踪确认（写明为什么没法写成测试）、未验证（写明缺的具体条件，不要笼统地写“需要环境”）。各次评审的“未验证”条目汇总起来，就是测试基础设施的待办清单。报告里还要分清每条结论来自测试、运行时日志还是读代码；日志按消息文本和函数名定位，不要用行号。
- “无问题”的结论要列出已检查的路径：可以写成“以下 N 条路径已检查，逐条给出行号，以及反例为何不成立；其余未覆盖”。同一个场景框架，较弱的模型配置给出了“无正确性问题”，较强的配置却推出了相反的因果链。“无问题”会让后续排查就此停下，是评审里代价最高的一句话。
- 审的是 checkout 出来的代码，不是 PR 网页：任何关于仓库状态的断言，比如什么被纳管了、什么已提交、有没有冲突，都要附上在真实分支上实际执行的命令和输出作为依据。例如一个 PR 的网页 diff 只显示了约 2% 的真实改动，服务端的合并预览会丢掉分支上的大部分内容，所以评审之前，先用三方 diff 核验评审对象本身。
- 对 AI 给出的类型或边界类修复建议，先把调用链上的类型列全：采纳前把这个值经过的每个函数签名列出来，位宽不匹配往往出在往下游数两个函数的位置（案例见 2.2 第 3 条）。
- 上下文：给稳定的不变式，而不是全部文档：前面好几条误判，根子都在 AI 缺少背景：哪些不变式是靠有意的 abort 来守护的，哪些计时器背后其实是人工流程，哪些设计早已论证定案。但这类细节多，实现又常变，过时的信息反而会误导 AI（我们试过把项目文档喂给 AI、把项目历史文档作为知识库，都出现过因为内容过时而被带偏的情况）。更有效的做法是只提供稳定的、高层的不变式和关键设计约束，并且和代码放在一起维护。针对本次 PR 的设计文档，配合 prompt 使用效果比较好；缺陷知识库（包括 accept/reject 以及暂缓处理的缺陷）能防止重复误报（要警惕它让回归问题检不出来）。每轮评审之后要补的，是“哪些上下文缺失让 AI 理解错了”，不是简单地把发现的问题堆进 agents.md/claude.md。知识库的效果还高度依赖模型能力，SOTA 模型能分辨冗余信息，能力较弱的模型则容易被误导。
- 不要用多 Agent 投票代替正确性判断：多个 Agent 读的是同一份 diff，缺的是同一份背景，盲区是共享的，会在同一个地方错得一致。实践中，合并报告就曾把多个 Agent 都给出的错误结论当成了正确结论，直接标成最高优先级。票数反映的是重合度，代表不了正确度，交叉评审的结论还是要逐条验证。
- 要求 AI 声明执行过程是否发生降级：AI 评审前通常会先做代码索引之类的前置步骤，任何一步失败，它都可能直接降级成 grep 之类的策略接着评审，正确率会明显下降，报告里却未必会提。所以要让报告写明实际用了哪些工具和步骤、哪些环节失败或降级，方便判断结论的可信度。
3. AI Debug：加速定位，但不能直接作为根因结论
AI Debug 最直接的价值，是缩短从“看到异常”到“找到值得验证的方向”的这段路。它能很快把日志、监控、调用栈、代码和配置关联起来，沿多个方向提出候选，还能生成诊断脚本、最小复现和对照实验。碰到跨模块、跨仓库的问题，信息整理和重复排查的时间能省下不少。
但 AI 给出的解释，不管看上去多完整、多具体，都不能直接当作根因结论。分布式存储的故障往往取决于并发时序、节点角色、持久化状态、恢复过程、外部依赖和运维操作，AI 拿到的通常只是其中一部分。它很容易找到一条“能解释现象”的代码路径，却没有证明线上当时真的走过这条路径，也没有证明别的原因产生不了同样的结果。
所以更合适的定位，是把 AI 当作调试过程中的假设生成器和验证助手：它负责扩大搜索范围、压缩排查空间、执行可控验证，人负责补充系统语义、判断证据强度、确认根因和审查修复。
3.1 AI Debug 擅长的地方
1. 快速汇总分散线索
AI 可以把不同节点的日志、监控指标、调用栈、配置和代码变更对齐到同一条时间线上，再顺着去找同一个状态的生产者和消费者。跨文件检索、正常路径和异常路径的比对、版本差异整理这类机械活，交给它最合适。
例：创建成功，随后操作失败，稍后重试恢复。 AI 串联三个代码仓库，发现报错来自“没有查到刚创建的资源”。人工补充“数据库已有记录、请求去向正常、稍后重试成功”后，排查收敛到缓存：创建之前，某台机器缓存了“资源不存在”；创建之后，这条缓存没有及时更新，导致后续操作失败。定期刷新后，缓存才与数据库一致，重试随之成功。
2. 并行提出可以被验证的候选
同一个延迟、错误码或状态异常，往往有好几种解释。AI 可以从缓存、队列、并发、网络、持久化、重放和资源竞争等方向同时展开，并给每个候选列出必要条件和缺失的证据。这份候选集合是用来帮工程师决定下一步查什么的，不是让 AI 从里面直接挑一个“最像根因”的答案。
例：旧数据一直没有清理，占用空间持续增长。 AI 梳理了“清理任务没运行、数据仍被使用、删除失败”等候选，并指出对应的日志线索。人工据此发现：节点已经开始工作，但清理模块的“启动完成”标志没有设置，因此每轮清理都被跳过。AI 提供了可逐项检查的排查清单，人工用现场日志确认实际命中的原因。
3. 把局部机制推导成可执行实验
AI 可以把代码里的条件、阈值和执行顺序转成明确的实验预期，再结合实际的输入和配置，检查这些条件能不能同时成立。实验要走真实入口，让系统自己产生待验证的状态，并设置能区分不同解释的对照和断言。这样“看起来可能发生”的事情，就变成了“能复现或者能被证伪”的判断。
例：把“空格在哪里丢失”的推测变成完整链路实验。 请求校验失败时，AI 用单点实验确认服务端会删除首尾空格，最初据此归因于服务端。人工要求继续检查客户端后，AI 将问题转成可检验的对照：记录客户端计算签名时的内容、实际发送的内容，以及服务端收到并用于重算签名的内容，逐段比较第一次差异出现在哪里。如果内容在发送前就已改变，就能否定“服务端首次改写”的解释。实验发现，客户端先按带空格的内容计算签名，发送前又删除空格，使签名与实际发送内容不一致。AI 因而修正了原先的因果定位。
4. 根据新证据持续缩小问题空间
AI 可以结合分段观测和对照实验，一步步确认哪些因素参与了问题，并据此调整排查的优先级。每次干预之后，都要把结果和原始异常、正常基线比一比，看清哪些已经解释了，还剩多少偏差。指标没有回到基线就接着查，别把局部改善当成排查的终点。
例：参见本节第1点。 “数据库已有记录”排除了创建未写入数据的解释，“请求去向正常”排除了请求被发往错误位置的解释，“稍后重试成功”则进一步指向会随时间变化的缓存状态。每条新证据都用来调整候选范围，再结合缓存更新逻辑解释整个过程。
3.2 AI Debug 容易出错的地方
1. 输入不完整、缺少证据，却给出完整而确定的结论
代码和日志通常装不下全部的线上状态、设计决策和运维约束。AI 容易拿未经核实的前提去补信息缺口，把局部推导写成完整的因果链，甚至在没有日志、代码或现场证据的情况下直接定性。要让它明确列出依赖的假设和缺失的信息，每个关键判断都附上能核查的证据，并说明这条证据支持的是哪一步，没有证据的只能标成假设或待验证。分析写得流畅、带着代码行号，只说明它好复核，不说明结论正确。
例1：局部查询漏查，不等于编号会重复。 AI 发现，创建对象前的一处查询只返回正常对象，不返回正在删除的对象。它把这处查询当成防止重复创建的关键检查，推断“旧对象查不到，就可能允许用相同编号再次创建”。人工指出，新对象编号由统一分配器保证唯一，这处查询并不负责编号分配；“相同编号再次进入创建流程”这一推导前提没有成立，因此撤销了风险结论。
例2：并行测试失败，原因并非最初猜测的名称碰撞。 测试改为同时运行后出现失败，AI 最初归因于随机名称碰撞。人工要求补充日志证据后，分析才确认：一个测试清理资源时，把另一个测试的资源也删掉了。名称碰撞虽然听起来合理，却不能替代实际删除行为的证据。
2. 把单点事实当成了因果位置
单点实验能证明某个组件有某种行为，证明不了本次异常最早出在这个组件。要沿实际的处理链路，把各环节的输入、输出和状态对齐，找到第一次偏离预期的位置，再验证后面的环节是怎么传播或者掩盖这个偏差的。上下游的证据补齐了，才分得清异常是在哪里产生的、又是在哪里被看到的。
例：参见 3.1 第3点的空格校验问题。 “服务端会删除首尾空格”确实有单点实验支持，但它不能证明本次请求的空格是在服务端首次丢失。完整链路实验发现差异在发送前就已产生，说明证据真实也不等于因果位置判断正确。
3. 把相关因素、缓解效果和根因混在一起
一次操作之后现象变好了，先只能说明两者相关，这个操作可能同时改变了多个条件，也可能把现场清掉了。产生初始错误的原因、让已有问题显现出来的条件、扩大影响的因素、能恢复服务的措施，这几样要分开识别，靠控制变量和操作前后的证据去确认各自的作用，不能把“有效缓解”直接升级成“已找到根因”。
例：参见 3.1 第1点的创建后操作失败问题。 这次请求恢复，是因为缓存刷新了；但创建资源时没有及时更新旧缓存的缺陷仍然存在，相同操作顺序下还可能再次失败。解释清楚“这次为什么恢复”，还不等于解决了“为什么会发生”。
4. 静态可达或测试绿色，被误当成线上已发生
静态分析发现一条危险路径，只能算一个风险候选。要判断它会不会造成实际问题，得检查触发条件在真实运行中能不能同时满足，以及系统已有的保护措施能不能拦住错误或者限制影响。要进一步认定它就是本次故障的原因，还需要现场证据证明当时确实走过这条路径，并且观察到的异常是由它造成的。测试通过也只证明给定的环境、输入和断言所覆盖的行为成立，据此既不能说生产中已经发生了同样的问题，也不能说系统整体是正确的。静态推导、实验验证和现场证据各自能证明到哪一步，要分别标清楚。
例：本节第1点的编号冲突误报。AI 从局部漏查推导出重复创建风险，却没有先证明同一编号能够再次被分配。全局编号规则使这一前提不成立，因此该推导既不能证明实际风险可达，更不能被写成已经发生的故障。这一案例说明的是静态推导的证明边界。
5. 围绕触发 case 做局部最优修补，不能彻底解决类似问题
AI 往往会在当前失败的那一行附近，给出最快能让用例变绿的改动，却不去检查同一个约束在其他场景下是不是还被破坏着，资源共享、并发操作，以及数据的创建、使用、恢复和清理过程尤其容易被漏掉。
例：局部方案还要覆盖并发、副本和恢复。 在一项数据拆分方案评审中，AI 默认各步骤使用的数据版本保持不变，也一度依赖当前机器的缓存完成后续初始化。人工追问后台任务是否会改变版本、其他副本或重启后如何取得相同信息，方案才补充版本保护，并把必要信息写入可恢复的日志，缓存只用于加速。这是设计评审中被纠正的方案缺口，不是已发生的线上故障。
3.3 使用 AI Debug 的建议：推荐工作方式
核心原则是先保存事实，再提出假设；用实验验证因果，由人复核根因和修复；还没解释清楚的现象，接着往下查。
1. 固定现场事实，并记录状态采集边界
开始分析之前，先由人和 AI 一起建立一份不掺推测的事实底稿，包括故障表现和影响、发生经过、运行环境和操作历史，再按需补上数据、缓存、外部依赖和配置生效的情况。原始日志和监控、业务方的转述、静态的代码推导，要分开记录。每份状态都要注明采集时间、来源实例和读取方式，没法确认几份证据是否属于同一时刻或同一个请求时，就明确标出来，不要把它们拼成一张“同时发生”的现场图。重启、重试、刷新缓存这些操作会改变现场，动手之前先把证据存下来。
2. 让 AI 输出候选集合，不接受单一根因
让 AI 给每个候选写清楚触发条件、代码路径、状态变化、能解释和解释不了的现象、当前的证据以及证伪的方法；人再补上全局不变量、历史设计决策和运维约束，例如 ID 由谁分配、角色怎么切换、哪些状态会持久化、哪些失败允许重试。
排查过程中统一使用下面这些名称，免得结论在措辞里被悄悄升级：
- 观察事实：日志、监控、现场状态、原始字节或测试结果直接支持的内容；
- 候选：能够解释部分现象，但必要条件或因果链尚未闭合；
- 已验证因素：实验已证明它参与问题，但还不能据此认定为根因；
- 根因：第一处错误、传播过程、保护失效和最终用户结果已经闭合，主要替代解释已被排除；
- 放大因素：会扩大影响或延长故障，但不产生最初异常；
- 缓解方案：能够止损或恢复服务，但不等于消除了根因；
- 未验证风险：静态上可能成立，尚未证明线上可达或已造成影响。
这份候选集合和分类，就是一张持续更新的排查地图：每拿到新证据就重新分类，被反证的候选及时排除并留下依据，只能解释部分现象的因素，不要升级成根因。
3. 用测试复现，但限定它能证明什么
单元测试适合验证局部算法、边界条件和某个最小时序，但证明不了生产请求会走到这条路径；测试环境里的端到端复现，能验证多个组件之间的传播链，不过只有版本、配置、数据状态和入口都和生产一致的那部分，才能往生产上外推。
并发、超时、部分失败和恢复类的问题，优先用故障注入来控制触发时点，同时观察业务结果和中间状态；关键验证按第 5 节的要求完成红绿闭环。
4. 沿完整链路寻找第一次错误
对每个高优先级的候选，沿着“生产—传播—持久化—提交—消费—恢复—清理”记录实际状态，找到第一次和正确路径分叉的位置，再接着解释它是怎么一步步变成最终用户看到的结果的。
5. 由人确认根因，并从全局不变量出发审查修复
AI 的结论写进事故报告、或者据此修复之前，要由熟悉这个模块的工程师复核：现场有没有证明必要条件成立，从第一处错误到用户结果的链路是否闭合，保护机制为什么失效，主要的替代解释有没有排除。AI 的“高置信”代替不了这道门禁。
修复评审要从系统的关键设计和全局不变量出发，解决的是这一类问题，不能只是让触发 case 变绿：沿相关调用链检查锁与引用、角色和 epoch、持久化与重放、错误传播，以及正常、异常、恢复和清理这几条路径，并评估兼容性、性能和发布风险。生产代码和测试代码的改动都要审。
4. AI 自动化：擅长可验证任务，但关键操作必须受控
团队尝试用 AI 做自动化，让它连续承担环境检查、构建修改、失败定位和测试回归这些繁琐，但是模式相对固定的工作。团队使用 AI 发起了两个此类任务：1） 分布式 KV 的构建系统升级。该项目原来用 Make + Autotools 构建，脚本维护起来费劲，因此需要迁到 CMake；2） 分布式 KV 服务端适配 ARM 服务器。服务端要运行在华为鲲鹏服务器上以降低成本，需要做好 ARM 适配。这两项工作的实施路径和验证方法比较清楚，但涉及的细节多、工作量大，人工处理时需要反复修改、执行和检查结果。实践下来，AI 确实省去了不少重复劳动，能够根据构建和测试的反馈持续推进。但连续执行时间一长，也可能沿着错误方向排查、扩大修改范围，甚至通过调整测试条件得到“通过”的结果。方向、边界和验收标准，仍要由人把关。
4.1 AI 自动化擅长的地方
1. 完成跨文件、规则明确的批量改造
构建迁移和平台适配会牵出依赖声明、编译选项、宏定义、测试目录和 CI 脚本等一大批关联修改，AI 可以把这些关系理清楚，并连续完成修改和验证。
例：CMake 改造覆盖了 31 个构建文件和配套脚本；ARM的适配则从依赖安装、测试编译、进程启动一路打通到结果文件对接。
2. 根据执行反馈，持续推进多轮修复
编译报错、链接失败和测试异常都能给 AI 明确的反馈，它可以按“执行—定位—修复—复测”的节奏连续干下去。
例：ARM 适配中，AI 陆续处理了依赖缺失、接口桥接、链接冲突和测试框架兼容这些问题，省掉了人工反复敲命令、看结果的成本。
3. 通过回归测试暴露隐含假设和存量问题
换一种构建方式或运行平台，原来环境里没显现的问题就会冒出来。
例：CMake 改造的测试执行暴露出路径硬编码、测试状态污染和内存管理缺陷；
例：ARM 适配也暴露了测试中的越界、未初始化和释放后使用问题。这些失败 AI 可以一直跟下去，修复后的全量回归和重复运行验证也由它来做。
4.2 AI 自动化容易出错的地方
1. 容易采用深度优先的排查方式，沿一个假设越挖越深
AI 找到一个看似合理的解释之后，容易顺着这条路径一直追下去，不断补推导、做实验、改代码，却没有先比较其他可能的原因，也没有回头检查最初的假设是否成立。最后可能花了大量时间分析很深的机制，真正的问题只是编译参数、依赖路径或者测试配置对不上。排查的步骤多、解释得详细，并不说明方向是对的。
2. 容易在缺少证据时，把问题归因于基础库或操作系统
碰到崩溃、内存异常或者跨平台差异，AI 可能过早怀疑内存分配器、系统库、编译器或操作系统的行为，反而忽略了项目自己的调用错误、链接配置和测试缺陷。
例：ARM 适配的过程中，就需要人明确要求它收回对依赖库和系统库的泛化怀疑，同时补充已有的静态库、本地编译方式等信息。基础组件可以列为排查候选，但要有具体证据支持，不能成了解释未知问题的默认答案。
3. 把消除报错或测试变绿，当成问题已经解决
例：CMake 改造工作的修复记录里，AI 用过 _exit 跳过全局析构来避免退出崩溃，也在引入一个始终成功的鉴权替身之后调整过断言；ARM适配改造的 CI 则跳过了三个依赖真实 KMS 的用例。这些处理在特定的测试环境里可能说得过去，但它们改变了验证的范围，不能据此认为原问题已经解决，或者真实的依赖链路已经验证过。
4. 为解决局部问题，不必要地扩大修改范围
AI 可能去改任务范围之外的源码、调整公共的构建逻辑，或者靠放宽检查绕过当前的失败。
例：ARM 适配中出现过需要回退的源码和格式修改，也用过允许重复符号定义的全局链接选项。当前的问题是绕过去了，却可能带来兼容性影响，或者把后面的错误盖住。
4.3 使用 AI 自动化的建议
1. 先明确目标、边界和验收标准，再让 AI 连续执行
要说清楚允许改什么、必须保留什么、依赖从哪里取，以及通过哪些验证才算完成。在既定约束内的构建
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
AI allows the code to be written faster, but in a distributed storage system that "do not make mistakes", how can it be ensured that efficiency does not become a risk? This paper combines the real practices of the American Foundation Technology Department in AI evaluation, problem mapping and automated development, sharing what AI can do to help, where it can easily make mistakes, and where it still needs people to do it. It's more important than "multiplier" to make codes and conclusions verifiable. It is hoped that these experiences will inform AI applications in complex systems, while enjoying increased efficiency while maintaining the bottom line of quality and stability.
Background: Why distributed storage needs to be used more carefully
Since the beginning of this year, with the spread of Agent, AI Coding has been used in functional development, test generation, script preparation and implementation, problem mapping: Read the code faster, so that it can write a version of it first, and that part of the chores other than R & D can run.
Efficiency goes up and the quality count goes back. The same manpower, with more functionality delivered in one version and more code changes, increases the absolute number of deficiencies introduced cumulatively in one version, even if the codes written by AI have a thousand-line deficiency rate and one-man-man. So the goal of AI Coding is not just to write code more quickly, but to speed it up, the quality of the code and the adequacy of verification are higher than conventional R&D.
Distributive storage magnifies the problem. The size of the code and the complexity of the relationship between the modules are met by consistency, condensation, durability, failure recovery and performance. A seemingly partial modification, which may normally be normal, will not be revealed until the node is restarted, the disk abnormal, the network vibrates or the hair is high. Problems arise and will not affect only one function, but rather data availability and system stability in the upper business, which will also affect data accuracy in serious cases. Storage services support a large number of top-level operations, with a high level of failure and difficulties in repairing, and a very low level of tolerance for deficiencies.
To meet this quality requirement, AI generation code is not enough, and AI review, AI testing and validation, AI Debug and controlled automation have to be built together, so that the AI generation code can enter the R&D process safely. This paper, combined with the practices of the storage team in AI evaluation, AI Debug and AI automation, discusses how, in a system such as distributed storage, which is highly complex and has a low tolerance, it can be used to speed up AI while maintaining stability and data validity through more rigorous evaluation, testing, debugging and automated validation, hoping to be of value to AI Coding for other similar systems.
2. AI Review: A useful addition, but not a substitute for judgement on the design background
2.1 AI Evaluation expertise
1. Cross-document coherence noted
Example: ENABLE UNISTEST has three call points to control the compilation branch of the mock variant, but CMake migrates to replace this name with WITH UNTEST, the old name has never been given value. There's nothing wrong with looking at any file alone, and this bug exists only between documents and files.
2. Capable of constructing enforceable evidence
Example: AI stated that it did not stop after “there was no border check” and then constructed a deformation package stating 1 GiB, with only 5 byte in the actual package, to reach the real decoded entrance and capture the crash site.
3. The part that you can't prove.
Example: In a suspicious and competitive situation, AI can only find a suspicious path by using a code. It made it clear that the staggered sequence required ThreadSanitizer, and that the current tool chain could not run, so it was marked as “unverified” and did not directly qualify as defective.
4. AI Identification rate for some common competitive issues High
For example, a “read-and-read-and-write” and no mutually exclusive updates have been made. Several reviews have identified them in a stable manner and have given the same direction for repair (consolidated into a single atomic operation). Such common concurring patterns are well identified and can be consolidated into fixed inspection items.
5. AI will not miss the code removed from PR
Example: One re-construction merges an interface into a single RPC and removes the old logic of cleaning up local cache after success. AI indicated that this would leave a short window of time during which older data could be read out. People are the least sensitive to what is being deleted.
AI will evaluate incremental changes in relation to the inventory code
Example: Before the evaluation, AI analyses the inventory code. In the larger PR, the new code may have replaced the original problem by hand, and the manual review would have made it easier to treat this retrofit as a feedback on the new problem; AI would have singled out the issue of “originally, and this time has been replaced”, and would have reported inaction.
AI Evaluation is not limited to code-level changes
Example: In one of the PRs, the construction files of the three warehouses are referred to each other by their feauty branch name, without a fixed version, in the wrong order or the branch is cleared, and the backbone construction is broken. This is a problem at the level of "consolidation" where people who look at the PR page only happen to be the easiest to miss, and AI considers the construction file to be the object of an evaluation.
2.2 AI Evaluation is prone to error
AI's miscalculation is mostly due to the fact that it does not see anything beyond the code, such as traffic flow, design decision-making and team engagement. We have also seen in practice the problems of AI evaluation with respect to recommendations for restoration, the noise ratio, ranking and the taking of evidence.
1. Codes to be reached
Example: The report identifies an expired epoch code path as a real risk. The manual review did not question the code itself, but the question was whether the system around it would produce input to the branch: "EPOCH NOT MATCH would simply move to division +1, the old region would not exist after division, the migration would not exist, and the src would not exist, so the EPOCH NOT MATCH would not theoretically be able to reach the branch."
The same applies to the issue of the expiration of the day when the report is marked as a competitive window: “the migration is manual, there will be no online migration launch day that has not been successful and will require manual intervention, and the expiration of the day will solve the problem”. AI can model the timer, not the manual process outside it.
2. Deficiencies between intent and general practice
Example: The report labels the consistency check directly calling abort() in the production environment as a correctness risk and recommends that it be changed to record the Error log and not collapse. We actually chose abort, because storage systems are more accurate than available, and in a given scenario we prefer abort, not to allow this request to continue to produce very difficult dirty data.
3. AI will also need to evaluate the accompanying restoration proposal itself.
Example: AI gave a repair for a 32-bit spill to verify whether the declared size and the number of byte actually received were consistent. Upon review, it was found that getting size() returns at int, size between 231 and 232 would be cut to negative, and the verification itself would be silent. The case has been fixed and the same problem has not been fixed.
Cheap "Recognition" entries crowd out real signals.
Example: AI has marked the thread pool defaults, renamed configurations, removed secondary cache refreshing switches as “to be confirmed”. Every last response is based on historical data measured by the team itself, not related to code defects: "According to pressure, the lower number of lines in the back desk is intended to reduce the impact on the front desk."
5. AI evaluated issues for distortion of rating
AI lacks a well-defined quality threshold: the same type of theoretically possible problem, this time marked at the highest level, and the next time it may not be important. Getting an AI report, you can't really believe it. You can only examine all the questions one by one for insurance, but it's hard to focus on really high-level issues.
6. Use of notes, headline document assertions and design documents as evidence
Example: Two AI reviews used the same "scenario x bound" framework, but reached the opposite conclusion. The assertion in one of the direct recapitulations was correct; the other was wrong to check the actual realization. Another example is the simple reliance on the design document, i.e. the continuous validation of the version required in the migration design document, without any reference in the code.
There is no difference between the codes examined by the two reviews, but the difference is whether the “notes” themselves are validated.
2.3 Enable evaluation findings to be implemented: build test, integrate test and run time log
Static evaluation can only indicate " this path exists in the code ". Whether this path is really wrong, whether existing protections can be stopped, whether or not it has occurred on the line requires three additional types of evidence to answer.
2.3.1 Build and measure: transform reasoning into Red Green
Experiment: A total of 117 suspect questions assessed by AI for a project, article by article gtest verification, and failure of the test on master is considered confirmation. Result: 36 red tests confirmed, 3 were perjured (misreported), 41 were confirmed only by source code and 37 were not verified.
- Judgement confirmed. The red-red test is concentrated in three categories: input trigger, error is reported as successful when swallowed, decoded across borders. The severity will be determined by failure, no more AI language.
- Perjury. Three articles were found to have been overturned by the tests, a conclusion that was difficult to draw from the static evaluation.
- to expose C.I.'s own problems. For example, the test report has a hard-coded failures = "0" and failed, so let AI build and see the original output, not just the CI green light.
- Boundary. Competing state and UAF require sanitizer, time-series issues require movement control, failure injection points, restricted hosts, clock deviations, and the list of “unverified” is actually a course to be filled in to test the infrastructure. The red-red test can only prove that the code is wrong and that the input does not occur online: there is a data migration scene, the tests confirm that the data will be out of date, and a manual review reveals that in the manual process, the expiration was deliberately designed.
2.3.2 Integrated testing close to running time: the primary value is understated
For example, we have one project category where special consideration has been given so that we can pull up a real Raft multi-point cluster in a single process and check all the scenes that require “production clusters”.
- Involve existing protections in validation. An AI-discovered problem is that a leader switch leads to the deletion of lost data, and an existing integration test happens to remove the main part of the window and the result proves to be safe.
- Difference of decision. For example, "Can a snapshot capture the spit mesotype?" Two AI reviews, contrary to the opinion, can be determined directly by taking the real change peer portal, adding sync points to write a test.
The boundary of fake/mock is the border of certification. Fake is going to take the real entrance, use the configuration of the production and indicate what has been replaced.
Example: The distribution KV's hang whistle-blower Bug was not detected because of the different settings used for testing and the actual production; the network phase's disposal protection was manually fed time stamped in a single test, but the online path was never set.
Run-time logs: from "possible to occur" to "have occurred"
Experiment: Give an AI map back to the source code for 11 days of a storage node (678,000 lines, 34 call points) and find five defects, all of which are true and have not been misreported.
- Discovering issues that static reviews would not suspect. Example: "fail times 1" only happens once in 11 days, which means the counter is stuck and the Hang alarm will never trigger.
- False hypotheses. Example: A 19-minute response with a constant ratio of 1.0 for the sum of the stages to the total time, which precludes a clock error and leads to a lack of discard protection at the dispatch stage.
- Make the ratings sound. Example: The measurement phase has already taken up to 1,156 seconds, less than two times the time-consuming int32 spill.
Two of these deficiencies are not directly triggered by the log, but are found by AI in an attempt to explain how the log is going to read the adjacent code. It is equally true that the journal provides the true configuration and magnitude that may also allow the adjacent code to be more accurate. This is merely a sample, which does not indicate that there is always a zero-sum error, but it is sufficient to support the practice that the evaluation and Debug should be attached when the run-time log is available. Logs are limited to covering only executed paths, and line numbers may also be inconsistent with the deployment version (which needs to be positioned according to message text).
Summary
The four tiers of evidence are complementary: static reviews for candidates, tests for authenticity, and logs to see if there is any reality.
2.4 Recommendations for the use of AI
- The conclusions are produced in four stages according to the level of evidence and are not mixed: test confirmation (with a failed test on the current backbone), test testimony false, source tracking confirmation (identify why it cannot be written as a test), unverified (identify the specific conditions missing, not “need the environment” in general). The “unverified” entries from the reviews are aggregated and represent the list of to-dos for testing infrastructure. The report also indicates whether each conclusion comes from the test, running time log or the read code; the log is located by message text and function name and does not use line numbers.
- The conclusion “no problem” should set out the path examined: it could be written as “the following N path has been checked, a line number given by article, and why otherwise, the rest is not covered”. In the same scenario framework, the weaker model configuration gives "incorrectness problems" and the stronger configuration introduces the opposite causal chain. “No problem” would stop the follow-up, the most costly sentence in the review.
- Reviewed the code from checkout, not the PR web page: any determination of the condition of the warehouse, such as what is in the hands of the bank, what has been submitted and whether there is any conflict, must be supported by an order and an output actually executed in the real branch. For example, a PR web page diff shows only about 2% of real changes, and the service-end merger preview loses much of the branch, so the evaluation is preceded by a tripartite diff examination of the subject itself.
- For the type or boundary class fixation suggestions given by AI, first the type of the call chain is listed in the list of each function through which the value passed before adoption, and the width is often not matched by the location of two functions downstream (see Article 3 in case 2.2).
- context: To stabilize the constant, not the entire document: previous miscalculations, with root in AI lacking context: which ones are deliberately guarded by abort, which timers are actually manual processes behind them, and which designs are already validated. But this type of detail is numerous and often variable, and outdated information is misleading to AI (we have tried to feed project files to AI and to use project history documents as a knowledge base, all of which have been biased because they are outdated). It would be more effective to provide only stable, high-level static and critical design constraints and to maintain them together with codes. For this PR design document, co-opt works better; a flawed knowledge base (including accept/reject and suspension deficiencies) prevents repetition of misreporting (sware that it does not detect a return problem). Each round of reviews is supplemented by “which context is missing to allow AI to get it wrong” and is not simply stacked with problems found.md/claude.md. The effectiveness of the knowledge base is also highly dependent on modelling capabilities, with SOTA models able to distinguish redundant information and less capable models vulnerable to misinformation.
- Don't use multiple Agent votes instead of correctness judgements: multiple Agents read the same diff, missing the same background, blind areas are shared, and can be wrong in the same place. In practice, the combined reports have taken as the correct conclusions the many erroneous conclusions given by Agent, directly marking the highest priority. The number of votes reflected the degree of overlap, which did not represent the correctness, and the conclusions of the cross-examination were subject to article-by-article validation.
- To require AI to state whether there has been a downgrading of the implementation process: AI usually does pre-editing steps such as code indexing before evaluation, and any failure of any step may be directly downgraded to strategies such as grep, and then evaluates the correct rate, which is significantly lower, but not necessarily mentioned in the report. It would therefore be useful for the report to describe the tools and steps actually used, the links that failed or were downgraded in order to assess the credibility of the conclusions.
AI Debug: Accelerated positioning, but not directly attributable to conclusion
AI Debug's most direct value is to shorten the road from "seeing anomalies" to "finding directions worth proving." It can quickly link logs, monitoring, call stacks, codes and configurations, present candidates in multiple directions, and generate diagnostic scripts, minimum recurrences and control experiments. In the case of cross-modules and cross-warehouses, the time spent on information collation and double-checking could be saved.
However, the explanation given by AI, however complete and specific it may seem, cannot be taken directly as the root cause of the conclusion. Distributional storage failures often depend on a symmetric sequence, nodal role, durability, recovery process, external dependence and transport operations, of which AI usually receives only part. It is easy to find a code path that “explains phenomena”, but it does not prove that the path was actually followed on the line or that other reasons did not produce the same result.
So the more appropriate position is to use AI as a hypothetical generator and authentication assistant during debugging: It is responsible for expanding the scope of the search, compressing the search space, performing controllable validation, and for supplementing the semantics of the system, judging the strength of the evidence, identifying the root causes and reviewing the restoration.
3.1 AI Debug's specialty
1. Quick aggregation of decentralized trails
AI can align the logs, monitor indicators, call stacks, configurations and code changes of different nodes to the same time line and then follow the same situation of producers and consumers. Mechanical work such as cross-file retrieval, regular and abnormal routing, copy discrepancy collation, is most suitable for it.
Example: Created successfully, subsequent operation failed and later retry resumed. AI In a series of three code warehouses, errors were detected from “no newly created resources were identified”. After manual updating of “the database has been recorded, requested to go to normal, later successful retry”, the screening dwindled to cache: Prior to its creation, a machine cache “does not exist”; after its creation, this cache was not updated in a timely manner, resulting in the failure of subsequent operations. After regular updating, cache was consistent with the database and retry succeeded.
2. Parallel presentation of verifiable candidates
There are often several explanations for the same delay, error code or abnormal state. AI could be carried out simultaneously in the direction of cache, queue, parallel distribution, networks, durability, reallocation and competition for resources, and each candidate could be given the necessary conditions and missing evidence. This pool of candidates is used to help engineers decide what next step is looking for, not to get AI to pick out the answer from inside.
Example: Old data have not been cleaned up and occupancy of space continues to grow. AI combed the candidate list “The clean-up mission is not running, the data is still being used, deletion failed” and indicated the corresponding log trail. On this basis, it was artificially discovered that nodes had started to work, but that the “start-up-complete” sign for the clean-up module had not been set and therefore each round of clean-up had been skipped. AI provided an item-by-item checklist to manually use the field log to confirm the actual hit.
3. Inducing local mechanisms into enforceable experiments
AI can translate the conditions, thresholds and sequence of implementation in the code into clear experimental expectations, then check whether these conditions can be established in conjunction with actual input and configuration. The experiment is to take the real portal, allow the system to create itself in a state to be verified, and set up a comparison and assertion that distinguishes between different interpretations. So the “possible” thing becomes a “repeated or perjury” judgement.
Example: The theory of where the space is lost is turned into a complete link experiment. When the validation is requested to fail, AI uses a single-point experiment to confirm that the service end will remove the first-end spaces, which are initially attributed to the service end. A manual request continues to check the client, AI turns the problem into a verifiable comparison: records the content of the client ' s calculation of the signature, the content actually sent, and the content received by the service and used to recalculate the signature, comparing the first-time difference by paragraph. If the content changes before sending, the interpretation of “service-end first rewriting” is rejected. The experiment found that the client calculates the signature by using the contents with spaces and deletes the spaces before sending them, making the signature inconsistent with the actual content sent. AI thus modified the original causal position.
4. Continuous reduction of problem space based on new evidence
AI could identify, step by step, the factors involved in the problem in the context of segment observations and control experiments, and adjust the priority of the inventory accordingly. After each intervention, a comparison is made between the results and the original anomalies, the normal baseline, to see which have been explained and how many deviations remain. Without returning to the baseline, the indicator should continue to look at the partial improvement as the end point of the inventory.
Example: See point 1 of this section. “Database Recorded” excludes an explanation for creating unwritten data, “Request to go normally” precludes an explanation that the request is sent to the wrong location, and “Lately retry Success” further points to a cache state that changes over time. Each new evidence was used to adjust the range of candidates and to update the logic of the process in conjunction with cache.
3.2 AI Debug is prone to error
1. Incomplete, lacking evidence but giving complete and conclusive conclusions
Codes and logs usually do not contain the full online state, design decision-making and operational constraints. AI can easily fill information gaps with unverified premises, and write local extrapolations into complete causal chains, even when there are no logs, codes or live evidence. To enable it to clearly list dependent assumptions and missing information, each key judgement is accompanied by verifiable evidence, indicating the steps that this evidence supports, and those without evidence can only be labelled as hypothetical or subject to validation. The analysis is well-written, with code line numbers, and only indicates that it is good for review and does not indicate that the conclusions are correct.
Example 1: A local checkout does not duplicate a number. AI found that a query before creating an object returns only normal objects and does not return objects that are being deleted. It treats this query as a key check to prevent re-creation, assuming that “unable old objects, may be allowed to be re-created with the same number”. It is artificially stated that the new object number is the only one guaranteed by the Unified Partitioner and that this query is not responsible for the numbering; the corollary of "the same number again enters the creation process" is not established and therefore the risk conclusion is reversed.
Example 2: Parallel tests failed, not because of initial guess of a name collision. The test was changed to fail after running simultaneously and AI was initially attributed to random name collisions. When manual requests for additional log evidence were made, the analysis confirmed that one test of clean-up resources had also eliminated the other. The name collision, while sounding reasonable, was no substitute for evidence of actual deletion.
2. Treat single-point facts as causal positions
A single-point experiment can prove that a component is acting in a manner that does not prove that this anomaly was first found in this component. Follow the actual processing chain, align the input, output and state of the chain, find the first deviation from the intended location, and verify how the subsequent link spreads or conceals the deviation. Upstream and downstream evidence is completed to determine where the anomaly arose and where it was seen.
Example: See the space verification question at point 3. The "service endspace will be deleted" is indeed supported by a single-point experiment, but it cannot prove that the space in this request was lost for the first time at the service end. The full link experiment found that the difference had occurred before it was sent, suggesting that the evidence was not true and that the causal position was correct.
3. Combining relevant factors, mitigation effects and root causes
The phenomenon has improved after one operation, which can only be explained by the two, which may change several conditions at the same time and may also clear the site. The causes of the initial error, the conditions under which the existing problem becomes apparent, the factors that broaden the impact, the measures that restore the service are to be identified separately, the control of variables and pre- and post-operational evidence to identify the respective roles, and the “effective mitigation” cannot be directly upgraded to “found root causes”.
Example: See the problem of post-establishment failure in point 1. The request was reinstated because cache had been refreshed; however, deficiencies in the creation of the resource that did not update the old cache in a timely manner remained and could again fail in the same order of operation. Explaining “why this recovery” is not the same as solving “why it happened”.
4. Static reach or test green as if it had occurred online
Static analysis found a dangerous path and only one risk candidate. To judge whether it poses a real problem, it is necessary to check whether the trigger conditions are met in real time and whether the protective measures already in place in the system can stop errors or limit the impact. To further determine that it was the cause of this failure, there was also a need for on-site evidence that the path had indeed been followed and that the anomalies observed had been caused by it. By proving only that the conduct covered by the given environment, input and assertion is valid, the test cannot say that the same problem has occurred in production or that the system as a whole is correct. The steps that can be demonstrated by static extrapolation, laboratory validation and on-site evidence are clearly identified.
Example: The numbering of point 1 of this section is misreported. AI extrapolates the risk of duplicate creation from local leakages without first proving that the same number can be allocated again. The global numbering rule does not establish this premise, so that the extrapolation does not prove that the actual risk is achievable, let alone written down as a failure that has occurred. This case illustrates the boundary evidenced by static extrapolation.
5. Partially best patched around trigger case, which does not fully solve similar problems
AI tends to be close to the current line of failure, giving the quickest changes to make the case green, without checking whether the same constraint is still being undermined in other settings, and the sharing of resources, their operation, and the creation, use, recovery and clean-up of data are particularly vulnerable.
Example: Local programmes will also cover simultaneous distribution, copying and restoration. In an evaluation of the split programme, AI defaulted that the version of the data used for each step would remain unchanged and that subsequent initialization would at one time rely on the cache of the current machine. The program supplements version protection by manually asking whether the back-office job will change the version, other copies or how to obtain the same information after restarting, and writes the necessary information into the recoverable log. This is a corrected programme gap in the design evaluation and is not an online failure that has occurred.
3.3 Recommendation for the use of AI Debug: Recommended working methods
The core principles are the preservation of facts before assumptions are made; the testing of causes and consequences, the review of causes and repairs by human beings; and the failure to explain the phenomenon clearly before going down.
1. Fixed on-site facts and recording of status collection boundaries
Before starting the analysis, a non-predictative background of facts, including performance and effects, history of occurrence, operating environment and operational history, is created by people and AI, followed by data as required, cache, external dependence and configuration effective. Original logs and controls, recitation of business parties, static code extrapolation are to be recorded separately. Each state indicates the time of collection, the example of the source and the manner in which it was read, and it is not possible to establish whether several of the evidence belonged to the same moment or to the same request, so that it is clearly marked and is not composing them as a “situational occurrence”. Relaunching, retry, and updating cache will change the scene and save the evidence before doing so.
2. Bringing together AI output candidates and not accepting single root causes
Let AI spell out to each candidate the trigger conditions, code paths, state changes, phenomena that can be explained and not explained, current evidence, and methods of perjury; add to the overall non-variant, historical design decision-making and operational constraints such as who is assigned the ID, how roles are changed, which states will last, which failures allow retry.
The following names are used consistently during the screening process, so that the conclusions are not quietly elevated:
- Observation of the facts: logs, surveillance, state of the site, elements directly supported by original byte or test results;
- Candidates: Some phenomena can be explained, but the necessary conditions or causal chains are not closed;
- Verified factors: the experiment proved that it was involved in the problem but could not be identified as root causes;
- Root cause: the first error, the dissemination process, the lapse of protection and the end-user result have been closed and the main alternative interpretation has been excluded;
- amplification factor: increases impact or prolongs failure, but does not cause initial anomalies;
- Mitigation options: to be able to stop or restore services, but not to eliminate root causes;
- Non-validation of risk: static may be established, not proven to have reached or to have caused effects on line.
This candidate pool and classification is an updated inventory map: When new evidence is obtained, it is reclassified and the counter-provoked candidates are excluded in a timely manner and leave a basis for explaining only a partial element of the phenomenon and not escalating to its root cause.
3. Recapturing with tests, but limiting what it can prove
The module tests are suitable for the validation of local algorithms, boundary conditions and a minimum time sequence, but do not demonstrate that the production request will follow that path; the end-to-end recurrence of the test environment can verify the transmission chain between multiple components, but only the part of the version, configuration, data status and entry point that is consistent with production can be pushed out of production.
The problem of simultaneous, timeout, partial failure and recovery was prioritized to control the trigger point with a failure injection, while observing the operational results and the intermediate state; critical validation completed the red and green rings as required by section 5.
4. Search for the first error along the complete chain
For each high-priority candidate, the actual state of production-transmission-persistence-submission-consumption-recovering-cleaning is recorded along the lines of “production-dissemination-continuation-consumer” and the location of the first and correct routings is identified, followed by an explanation of how it gradually became the result seen by end-users.
5. Identification of root causes and review of repairs from a global perspective
The conclusions of AI are to be reviewed by engineers familiar with the module prior to the incident report or its restoration: whether the necessary conditions have been established at the site, whether the link from the first error to the user ' s result is closed, why the protective mechanism is not working, and whether the main alternative explanation is excluded. AI's “high faith” cannot replace the door.
The restoration evaluation is based on the key design of the system and on a global, non-variant approach. It addresses this type of problem and cannot simply turn the trigger case green: the chain of call-checking locks and references, role and epoch, persistence and re-laying, error transmission, and normal, unusual, recovery and clean-up of these routes, as well as assessing compatibility, performance and publish risks. Changes in production and test codes are reviewed.
4. AI Automation: good at verifiable tasks, but critical operations must be controlled
The team tried to automate it with AI so that it could continue to perform the cumbersome tasks of environmental inspection, build modifications, fail to locate and test back, but the pattern is relatively fixed. Team uses AI to initiate two such tasks: 1) the build system upgrade of distributed KV. The project, originally built with Make + Autotools, has been hard to maintain and therefore needs to be moved to the CMake; 2) distributed KV service end matching ARM server. The service needs to fit the ARM in order to run on the Wigand server to reduce costs. The implementation path and the validation methodology for both exercises are clear, but the details are numerous and heavy, and manual processing requires repeated revision, implementation and inspection. In practice, AI did save a lot of duplication of effort and was able to continue on the basis of feedback from construction and testing. However, for a long period of consecutive execution, it may be possible to check in the wrong direction, expand the scope of the modification, or even “pass” by adjusting the test conditions. Directions, borders and acceptance criteria remain open.
4.1 AI places of excellence in automation
1. Completion of cross-documentation, clear-cut batch modifications
Building migrations and platform approximation leads to a large number of associated changes, such as relying on declarations, compilation options, macro definitions, testing directories and CI scripts, which AI can clarify and complete continuously.
Example: The CMake adaptation covered 31 build files and supporting scripts; the ARM appliance went from relying on installation, testing compilation, process initiation to the outcome document interface.
2. Continued promotion of multiple rounds of restoration based on implementation feedback
Compiler error, link failure and test anomaly provide clear feedback to AI, which can continue at the “execution-position-rehabilitation-result” rhythm.
Example: ARM fit, AI continues to address the issues of dependence on missing links, interfaces, link conflicts and test framework compatibility, saving the cost of hand-knocking orders and results.
3. Implied exposure assumptions and stock issues through regression tests
In other words, building or running a platform, problems that were not present in the environment would emerge.
Example: Tests for CMake modification have exposed hard path codes, test state contamination and memory management deficiencies;
Example: ARM appliance also exposes transboundary, non-initiation and post-release uses in testing. These failures AI can follow, and full return and duplicate validation after repair is done by it.
4.2 AI Automation is prone to error
1. Easier depth priority screening and excavation along a hypothesis Deep
AI, after finding a seemingly reasonable explanation, can easily follow this path, continuously extrapolating, experimenting, changing codes without first comparing other possible reasons, and without looking back at the validity of the original assumptions. Finally, it may take a great deal of time to analyse the very deep mechanism, and the real problem is simply to compile the parameters, rely on the path or test configuration. The number of steps to be examined and the detailed explanations given do not indicate the right direction.
2. It is easy to attribute problems to the base or operating system in the absence of evidence
In case of crashes, memory anomalies or cross-platform differences, AI may be too early to suspect the memory distributor, library, compiler or operating system, and instead ignore the project ' s own call error, link configuration and test defects.
For example, in the adaptation process, ARM needs to be explicitly asked to withdraw generalized doubts about relying on libraries and reservoirs, while complementing existing static libraries, local compilations, etc. Basic components can be listed as candidates for screening, but to be supported by specific evidence, they cannot be the default answer to the unknown.
3. Greening elimination of errors or testing as a problem
Example: In the restoration record of the CMake retrofit, AI used exit to avoid exit from the crash and adjusted the assertion after introducing a consistently successful scaffolding; the ARM-adaptation CI passed three examples relying on real KMS. These treatments may be relevant in a given testing environment, but they change the scope of the certification and cannot be considered as having solved the original problem, or as having been demonstrated by a genuine reliance chain.
4. Unnecessary expansion of the scope of the modification to address local issues
AI may change the source code outside the mandate, adjust the public construction logic, or bypass current failures by relaxing inspections.
Example: A source code and format change that requires rollback has appeared in the ARM appliance, as well as the global link option that allows the symbol definition to be repeated. The current problem is to bypass, but it may have compatibility effects, or to cover the errors behind it.
4.3 Recommendations for the use of AI automation
1. Identify objectives, boundaries and acceptance criteria, and allow AI to implement them continuously
It was clear what was allowed to change, what had to be retained, where to rely on, and what certification was to be completed. Build within an established limit
Auto-fetched and translated by AI Atlas. All rights belong to the original author.