思想领袖
AI Agents 罕见崩溃。 他们静静地失败了研究显示为什么
在 Google 上的首选来源中添加 Unite.AI
大型语言模型agents现在回答支持票，查询数据库和计划旅行。 当它们出错时，它们通常会返回一个自信的答案和成功的代码。 越来越多的研究解释了为什么这些失败是很难看到的，以及工程团队是如何学习测量这些失败的。
似乎成功的失败
传统软件往往会大声失败。 一个函数丢出一个例外，一个服务器返回一个错误代码，一个监测仪表板变成红色。 AI agent行为不同。 这是一个循环：一个语言模型读取一个提示，决定调用搜索API或数据库等工具，读取结果，并决定下一步做什么。 每一步都可能会出错，而链条中的每个网络请求仍然成功。
考虑一个普通案件。 天气工具失败，返回正常响应中的“ 无法提供的服务 ” 文本。 该模型阅读了该文本，以礼貌方式道歉，甚至更糟的是，该模型从其培训数据中撰写了可信的预测。 用户收到流利的答案。 服务器日志以正常的空闲度显示成功请求。 常规监控库里没有任何记录显示出事了
模式提供者在自己的文件中描述了同样的空白。 兹Q 1Z的Gemini API指导指出，该模型"不执行函数本身". 它只提出函数名称和参数，开发者的应用程序运行调用。 因此，该应用是系统唯一看到真实结果的部分。 Anthropic的建造agents工程指南提出了相关的观点，即agents需要在每一个步骤，如工具结果，从环境出发，来判断自己的进度。 当一个工具将失败报告为普通文本时，这种地面真理是错误的，agent没有办法注意到。
这不是一个新问题，只是旧问题更尖锐的版本。 2015年，由D. Sculley领导的Google工程师描述了机器学习系统中的"隐藏技术债务". 他们认为，生产ML系统中的大多数风险都位于模型代码之外，在数据依赖性，配置和胶体代码中。 Agents在该列表中新增一层。 提示和工具描述是可以改变系统行为而不改变其代码的配置。
小编辑，大效果
微小的迅速改变能有多重要？在ICLR 2024的论文中，华盛顿大学的Melanie Sclar和同事只改变了几发提示的格式：分隔符，间隔符和大写符。 意义不变。 同一项任务的性能在LLaMA-2-13B上仍有76个精确点的差异。 灵敏度并没有随着更大的模型，更多的实例或指令调音而消失。
作者们建议报告各种可能迅速格式的绩效，而不是一个数字。 对于工程团队来说，实用的教训比较简单。 即时编辑就是代码更改。 它需要一个版本历史，需要测试才能到达用户。
这同样适用于工具描述，模型读作是其提示的一部分。 Anthropic的指南指出，工具定义应该像主要迅速一样“迅速的工程注意”。 它描述了一个用相对文件路径不断犯错的编码agent. 当工具被更改为需要绝对路径时，模型使用时没有出错。 模型没有改变。 只有它读到的描述
一个成功证明小
第二行研究研究的是一致性。 基准测试由 shunyu Yao 和同事在 Sierra 发布，并在 ICLR 2025 发布，模拟客户服务对话，其中 agent 必须使用工具并遵循业务规则。 作者推出一个名为pass^k的度量：agent在所有 k 尝试中正确解决相同任务的概率。
结果令人清醒。 GPT-4o agent在一次尝试中解决了约61%的零售任务。 当它不得不连续8次成功完成同样的任务时，这个比例下降到了25%左右。 一次称的演示是关于一个系统如何在千载运行中表现的
多Agent系统中断的地方
许多小组现在把几个agents连在一起，有一个计划，另一个执行，第三个检查工作。 UC Berkeley的一项2025年研究“为什么多Agent LLM系统失败？” 分析了7个受欢迎的多agent框架的1 600多个附加说明的处决痕迹。 作者在梅尔特·塞姆里(Mert Cemri)的带领下，构建了一个名为MAST的分类学，有14个故障模式，分为三个组：系统设计问题，agents之间的错配，以及任务验证。
他们的核心发现是，许多失败来自系统的设计方式，而不仅仅是基础模型的极限。 Agents忽略了他们指定的角色，失去了对话的轨迹，或者在不检查结果的情况下宣布任务完成。 这些失败没有留下任何错误。 只有有人能读懂每个agent实际做的事，它们才显眼。 兹Q 2Z对多agent悖论的覆盖使架构产生了一个相关的观点：添加agents可以放大一个早期错误而不是取消它。
使无形可见
工程反应借用了分布式系统，许多服务长期以来都遵循这些系统的要求，采用一种称为追踪的技术。 应用到agents，每个用户请求都成为追踪，每个模式调用或工具调用都成为其中的"span". 一个跨度记录其输入，输出，使用的符号以及花费的时间。 这项工作的行业标准机构OpenTeleometry正在起草关于基因AI跨度的共享公约。 截至2026年，这些公约的标识仍然与制定时相同，因此属性名称仍然可能改变。
一些开源项目如Langfuse,Arize Phoenix和AcruxCore已经应用了这个想法。 他们的源代码是公开的，因此团队可以检查一个追踪是如何存储的，并在自己的服务器上运行软件。 他们将模型呼叫和工具呼叫记录为单一跟踪的跨度，让团队给他们附加质量分数。 工具的选择不只涉及一个细节：一个工具跨度必须记录工具实际返回的内容，而不仅仅是网络请求是否成功。 没有这些，上述气象工具的故障就看不见。
工业调查表明，现在追查是常见的，但检测落后于追查。 LangChain的Agent工程状况报告在2025年末调查了1,340名从业人员。 质量是将agents投入生产时被引用最多的障碍，由32%的受访者命名。 89%的组织报告了对agents的某种可观察性，62%的组织详细追踪了个别步骤和工具呼叫。 只有52.4%的测试机进行离线评价，37.3%的测试机进行了现场流量评价。 大多数球队可以看到他们的agents做了什么。 少检查是否正确。
按比例排列的分级回答
追踪显示发生了什么。 他们没有说答案是否很好。 每一次手动阅读谈话都不进行规模化，因此球队越来越多地使用第二个模型作为分级员，这个方法被称为LLM-as-a-judge. 黎明郑和同事的一篇NeurIPS 2023论文发现，GPT-4等强力模型在超过80%的时间里同意了人类的偏好，大致与人类彼此同意的速度相同。
同一份文件记录了该方法的弱点。 裁判模型更喜欢先显示的答案，更喜欢更长的答案，并且可以偏好自己写的文字。 因此，法官评分是估计数。 它们对于体积是有用的，但需要对照一小组人类标注的例子来检查，对答案顺序随机排列。
从实践到政策
这一衡量工作也正成为治理方面的期望。 美国国家标准和技术研究所于2023年1月公布了AI风险管理框架。 其四项核心职能之一，即“计量”，要求各组织在一段时间内量化、跟踪和记录AI风险。 对agent来说，追踪和评价分数是函数要求的证据。
团队现在可以做什么
这与在这个平台上提出的可靠性是对代理AI的真正考验的说法相呼应。 新增上方的研究这个论点下面的测量细节：可靠性不是一个你曾经观察到的属性，它是一个你必须不断测量的范围。
研究指出了一套短暂的习惯，主要的模型提供者推荐了许多相同的习惯。 OpenAI建造agents的实用指南告诉各团队在优化前"设置evals以建立性能基线"，当agent超过一系列失败尝试或面临高风险行动时，将控制权交给人类。 Anthropic建议在agents自行行动之前“在沙箱环境中进行广泛试验”。
在实践中，这意味着几个具体步骤。 将提示和工具描述置于版本控制之下，作为代码。 对照固定数据集测试每个快速变化，并期望得出一系列结果而不是一个数字。 运行每个测试例多次，因为一个通过会隐藏不一致之处。 记录工具输出在痕迹内，所以一个失败的工具不能隐藏在成功代码后面。 将自动裁判的分数当作估计，然后对照人来检查。
这些习惯都不需要模型能力的突破。 它们要求将AI agent视为什么：一种必须测量的概率系统，因为它很少会宣布自己的失败。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Thought Leaders
AI Agents Rarely Crash. They Fail Quietly, and Research Shows Why
Add Unite.AI to your preferred sources on Google
Large language model agents now answer support tickets, query databases and plan trips. When they go wrong, they usually return a confident answer and a success code. A growing body of research explains why these failures are hard to see, and how engineering teams are learning to measure them.
A Failure That Looks Like Success
Traditional software tends to fail loudly. A function throws an exception, a server returns an error code, and a monitoring dashboard turns red. An AI agent behaves differently. It is a loop: a language model reads a prompt, decides to call a tool such as a search API or a database, reads the result, and decides what to do next. Each step can go wrong while every network request in the chain still succeeds.
Consider a common case. A weather tool fails and returns the text “service unavailable” inside an otherwise normal response. The model reads that text, apologises politely, or worse, writes a plausible forecast from its training data. The user receives a fluent answer. The server logs show a successful request with normal latency. Nothing in a conventional monitoring stack records that anything went wrong.
Model providers describe the same gap in their own documentation. Google’s Gemini API guide notes that the model “doesn’t execute the function itself”. It only proposes a function name and arguments, and the developer’s application runs the call. The application is therefore the only part of the system that sees the real result. Anthropic’s engineering guide on building agents makes the related point that agents need “ground truth” from the environment at each step, such as tool results, to judge their own progress. When a tool reports a failure as ordinary text, that ground truth is wrong, and the agent has no way to notice.
This is not a new kind of problem, only a sharper version of an old one. In 2015, Google engineers led by D. Sculley described “hidden technical debt” in machine learning systems. They argued that most of the risk in a production ML system sits outside the model code, in data dependencies, configuration and glue code. Agents add a new layer to that list. Prompts and tool descriptions are configuration that can change a system’s behaviour without any change to its code.
Small Edits, Large Effects
How much can a small prompt change matter? In a paper presented at ICLR 2024, Melanie Sclar of the University of Washington and colleagues changed only the formatting of few-shot prompts: separators, spacing and capitalisation. The meaning stayed the same. Performance on the same task still varied by up to 76 accuracy points on LLaMA-2-13B. The sensitivity did not disappear with larger models, more examples or instruction tuning.
The authors recommend reporting a range of performance across plausible prompt formats rather than one number. For engineering teams, the practical lesson is simpler. A prompt edit is a code change. It needs a version history, and it needs a test before it reaches users.
The same applies to tool descriptions, which the model reads as part of its prompt. Anthropic’s guide advises that tool definitions deserve “just as much prompt engineering attention” as the main prompt. It describes a coding agent that kept making mistakes with relative file paths. When the tool was changed to require absolute paths, the model used it without errors. The model did not change. Only the description it read did.
One Success Proves Little
A second line of research looks at consistency. The τ-bench benchmark, published by Shunyu Yao and colleagues at Sierra and presented at ICLR 2025, simulates customer-service conversations in which an agent must use tools and follow business rules. The authors introduced a metric called pass^k: the probability that an agent solves the same task correctly in all k attempts.
The results were sobering. A GPT-4o agent solved about 61% of retail tasks on a single attempt. When it had to succeed on the same task eight times in a row, the rate fell to about 25%. A demo that works once says little about how a system behaves on the thousandth run.
Where Multi-Agent Systems Break
Many teams now chain several agents together, with one planning, another executing and a third checking the work. A 2025 study from UC Berkeley, “Why Do Multi-Agent LLM Systems Fail?”, analysed more than 1,600 annotated execution traces across seven popular multi-agent frameworks. The authors, led by Mert Cemri, built a taxonomy called MAST with 14 failure modes in three groups: system design issues, misalignment between agents, and task verification.
Their central finding is that many failures come from how the system is designed, not only from the limits of the underlying model. Agents ignore their assigned role, lose track of the conversation, or declare a task finished without checking the result. These failures leave no error behind. They are visible only if someone can read what each agent actually did. Unite.AI’s coverage of the multi-agent paradox makes a related point from architecture: adding agents can amplify an early error rather than cancel it out.
Making the Invisible Visible
The engineering response borrows from distributed systems, where requests have long been followed across many services with a technique called tracing. Applied to agents, each user request becomes a trace, and each model call or tool call becomes a “span” inside it. A span records its input, its output, the tokens used and how long it took. The industry standards body for this work, OpenTelemetry, is drafting shared conventions for generative AI spans. As of 2026, those conventions are still marked as in development, so attribute names may still change.
Several open-source projects such as Langfuse, Arize Phoenix and AcruxCore already apply this idea. Their source code is public, so teams can inspect how a trace is stored and run the software on their own servers. They record model calls and tool calls as spans of a single trace and let teams attach quality scores to them. The choice of tool matters less than one detail: a tool span must record what the tool actually returned, not only whether the network request succeeded. Without that, the weather-tool failure described above stays invisible.
Industry surveys suggest that tracing is now common, but testing lags behind it. LangChain’s State of Agent Engineering report surveyed 1,340 practitioners in late 2025. Quality was the most cited barrier to putting agents into production, named by 32% of respondents. 89% of organisations reported some form of observability for their agents, and 62% had detailed tracing of individual steps and tool calls. Only 52.4% ran offline evaluations on test sets, and 37.3% evaluated live traffic. Most teams can see what their agents did. Fewer check whether it was correct.
Grading Answers at Scale
Traces show what happened. They do not say whether the answer was good. Reading every conversation by hand does not scale, so teams increasingly use a second model as a grader, an approach known as LLM-as-a-judge. A NeurIPS 2023 paper by Lianmin Zheng and colleagues found that strong models such as GPT-4 agreed with human preferences more than 80% of the time, about the same rate at which humans agree with each other.
The same paper documented the method’s weaknesses. Judge models prefer the answer shown first, prefer longer answers, and can favour text written by themselves. Judge scores are therefore estimates. They are useful for volume, but they need to be checked against a small set of human-labelled examples, with answer order randomised.
From Practice to Policy
This measurement work is also becoming a governance expectation. The US National Institute of Standards and Technology published its AI Risk Management Framework in January 2023. One of its four core functions, “Measure”, asks organisations to quantify, track and document AI risks over time. For an agent, traces and evaluation scores are the evidence that function asks for.
What Teams Can Do Now
This echoes the argument made on this platform that reliability is the real test of agentic AI. The research above adds the measurement detail underneath that argument: reliability is not a property you observe once, it is a range you have to keep measuring.
The research points to a short set of habits, and the major model providers recommend many of the same ones. OpenAI’s practical guide to building agents tells teams to “set up evals to establish a performance baseline” before optimising, and to hand control to a human when an agent exceeds a set number of failed attempts or faces a high-risk action. Anthropic recommends “extensive testing in sandboxed environments” before agents act on their own.
In practice, that means a few concrete steps. Keep prompts and tool descriptions under version control, as code. Test each prompt change against a fixed dataset, and expect a range of results rather than one number. Run each test case several times, because one pass hides inconsistency. Record tool outputs inside traces, so a failed tool cannot hide behind a success code. Treat automated judge scores as estimates, and check them against people.
None of these habits require a breakthrough in model capability. They require treating an AI agent as what it is: a probabilistic system that must be measured, because it will rarely announce its own failures.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.