OCT. 8, 2026 (英语).
您的 agent 返回 HTTP 200，保留在它的延迟预算范围内，没有工具将融资称为错误。 然而，它仍然记录着3A座席的确认情况，而从未问过座席选择的agent子座席是免费的，还是向旅行者推荐一个Florentine牛排屋，其简介为称素食者。
我们和agent合作的每个团队都试图回答同样的三个问题：我的agent怎么样了？它的损失模式是什么，它失败的方式不断回来？我该怎么爬：改变一些东西，知道它有帮助，并避免它滑回去。
将 agent 提升到最初的 80%，在那里它处理你预期的测试案例，今天可以相对直截了当和快速。 一个评价数据集，一个编码的agent，以及一个紧凑的内环的运行，分级，固定，比较，通常会让你在数日内完成已知情景上的成功的任务。 然后，你发布，质量曲线平坦或退化。
图1. Agent质量在发布之前迅速攀升，然后在生产中波动。
剩下的20%更困难的是地面在发布之后移动。 当用户发现agent实际能做什么时，使用量会变换，现场流量会停止与您初始的eval集类似。 与此同时，下面的系统也发生了变化：当你升级一个模型，更新你的驾驭，或者改变一个工具或技能时，一切仍然运行，健康检查都保持绿色(图2)，但对话质量和任务成功可能会以标准部署管道的方式转变，而你永远不知道警告。
图2. 每一次基础设施健康检查都保持绿色。 失败是沉默的，发生在谈话内部。
当一个会话确实出错时，找出为什么意味着要分离几层，这些层看起来与外界几乎完全相同。 例如：
- 模型可能已经幻觉了一种争论或放弃了前三个转弯的限制。
- 管弦乐带可能经过错误的子agent或转弯之间失去状态。
- 一项工具合同可能拒绝输入，因为其有效值从未在计划中记录。
- 指示或技能可能只是缺少一条规则，没有人想写下来。
这些失败模式可能有不同的主人和不同的修正。 原始的轨迹记录了发生在什么旁边的事而不是导致什么分辨这些层需要读取失败的轨迹以及产生它们的确切代码修订。 在线评价仪表板告诉你，一个分数移动了，一个编码的agent可以在你知道在哪里查看后检查一个痕量，但是在生产量方面，没有人可以亲手读取每一个对话。
图3. AQuA工作生产外环，并将发现回馈到你的内环。
在我们的6月帖子上，我们走过发布前的内环末端以旅行接待结束。 这个帖子覆盖了外环，发布在开放时作为可堆肥的构件，你可以在自己的项目中运行，适应你的堆肥，在我们探索团队如何运行连续的agent生产质量时帮助我们塑造。
AQuA (Amient Quality Quality Agent) 在您的 Google 云项目中，在您的 agent 边上运行，从 Cloud Trace, Cloud Loging，或 BigQuery 按时间表扫描生产轨迹，每次部署后，或是在您的笔记本电脑关闭时按需运行。 原始会话记录、 源头快照和大查询表都留在您的工程边界内，AQuA 从来不坐落在请求路径中，也不会写回您的 agent。
图4. AQuA在您的 Google 云项目中运行在您的 agent 侧面，读取轨迹并在请求路径外部署时间源快照。
每一次运行都阅读了最近会议的样本，并通过一个五阶段的管道：
图5 五阶段扫荡管道将原始生产过程转化为经核实的洞察力，对已部署的源头快照进行根源性诊断。
- 样本。 从遥测(Cloud Trace / Cloud Looking 或 BigQuery)中抽取最多1000个最近会话的随机样本(封顶以保持运行成本的可预测性).
- 检讨。 每场课按一个九点常见分解清单(程序、工具选择和参数、基础、任务完成等)分级，查看agent的指示、工具和可选目标。 当会话失败时，它会写一个结构化的实际/预期的发现。
- 组分组会议结论与候选问题分组相同。
- 核实一个单独的模型对照最多3份完整记录誊本检查每个候选群组，并抛弃证据不支持的群组。
-追踪将幸存的集群与未见的14天后在BigQuery(NEW, RREURRING
把它当作一个初级质量的工程师来做第一个传球：阅读对话，过滤噪音，为轮换的任何人准备有证据的档案。
为了引导阶段2向您的领域发展，您会写一个纯英语开发者目标(object.md)，该目标附在每一次审查的提示上；确定性Python自定义的度量衡(eval config.yaml)与法官一起运行，以趋势通过率。 默认情况下，AQuA会使用单通过会话审查法官，在每次会话的一次模式调用中根据该核对表评价对话(保持预定的扫码并产生驱动集群的实际/预期的diffs). 您也可以选择使用 Gemini 平台管理的轨迹 AutoRaters(任务成功，工具使用质量，轨迹质量)，该轨迹运行着每个量级的专用评价员(例如，如果您想要对照标准化的箱外路标来评分，或者将公尺与离线评价对齐).
当一个洞察力值得调查时，你触发根 - 因为分析从仪表盘聊天或agents - cli 水运行。 AQuA 读取失败的轨迹和部署时捕获的不可变的源快照：当缺陷出现在您的回转中时，它引用<path>: < start>-<end>，并提议一个锁定在快照线上的编辑；当错误位于您的代码之外( 上游依赖、 交接或检索的有效载荷)，它将故障归结为轨迹中的该步骤，而无需提出代码 diff。 它从不自行应用编辑或打开拉动请求。
它坐落在你已经拥有的环路之间：离线 evals 级一个候选人建立针对已知的测试案例，在线 evals 监视过速率生产趋势，编码 agents 或专业优化器编辑提示和代码。 AQuA将原始生产流量转化为诊断，密码化的洞察力和存档的故障记录片，播下你的内环。
让我们在旅行便利上行走一跑(google/adk-recipes)，由旅行者通过子agents(灵感 agent，地点 agent,poi agent，规划 agent，飞行搜索 agent，飞行座位选择 agent，预订 agent，以及前行/中行/后行)，通过记忆(键，值)将工作旅行存储在会话状态(旅行 concierge/tools/memory.py).
将AQuA附在旅行接待项目上需要三个agents-cli命令：
agents-cli 扩展加“${AQUA CHECKUT}”
agents-cli 以下单项目 - project="${GOOGLE CLOUD PROJECT}"——应用
agents-cli 部署 - project="${GOOGLE CLOUD PROJECT}" - 区域 - 东部 1
agents-cli部署除了AQuA的跑者，BigQuery数据集和Cloud Run仪表板(后身份-Aware Proxy)之外，还写了一张不可改变的快照，其中由部署修订(Review 1)按键，将旅行-Concierge的源树刻到Cloud Survey.
在仪表板的配置页面上，我们保存了一个开发者目标(goal.md)，以引导审查向高层次的产品变化性并抑制风格噪声(同时仍然要求每个发现都引用agent或子agent违反其指令或工具状态的混凝土转折):
目标：帮助旅行者从旅行灵感转向具体行程和经确认的预订，跨越我们的agents分机，每个经确认的航班、酒店、座位和推荐都以工具结果和旅行者概况为依据。
无法确保覆盖的模式：
- 中交变化(发布测试前的一个弱点)：如果用户在初始计划后更新目的地，日期，或飞行/酒店选择，确保后续的子agent工具呼叫和状态更新反映变化。
- 发送或授权通过灵感时上下文已删除 agent、规划 agent和预订 agent(如旅行者配置偏好或先前的选择)。
忽略：语气，问候，小谈，或用户浏览选项和离开而不预订的会话。
纯文本
复制
如果您还想要在 eval config 中指定 Python 值。 随时间推移，你可以用agents-cli auqua 度量衡来发布。
通过32场多场的agent扫射(32次重播4次脚本旅行者行程，产生1,583个横跨旅行 Concierge及其子agents的OpenTeleometry跨度),5场会程干净通过，27场会程产生42个结构化的发现，每个会程逐个范围到特定子agent的孤立指令和工具声明。 以下是关于规划 agent的一个发现：
实际：当用户在同一转弯中选择出航的UA204航班并请求座位3A和3B时，规划 agent将这些座位号码直接保存到会话状态，而不呼叫航班座位 seat agent检查是否有3A和3B.
需要：规划 agent 应调用 flight seat selection agent 在将座位号码保存到会话状态或返回到会话状态之前验证座位的可用性和定价。
将42项调查结果分为9个候选群组。 校验者(Gemini 3.7 Flash)对照每组最多3个完整记录和每个子agent的定义检查，并拒绝3个假阳性分组：在2个中，旅行者明确要求搭乘第一次返回航班或直接跳槽预订；在3个中，分组将两个不同子agents(飞行搜索 agent和灵感 agent)的不相关的即时工具错配合并。 这就留下了6个经核实的问题，主要是：
- 用户自愿获得座位(15场，规划-agent)时通过座位可用性检查：当旅行者在飞行选择(或在中途换乘目的地之后)的同时命名座位时，规划 agent将座位直接写入会话状态(返回HTTP 200，工具错误为零)，而不呼叫飞行座位 selection agent，以检查座位是否存在或是否开放。
——跨子agent边界放下饮食限制(7会，灵感 agent place agent)：灵感agent的提示包括旅行者的描述(food preference: vegan)，但place agent被包裹为孤立的工具，其提示从未收到过该描述。 当灵感 agent在请求中不通过"vegan"而询问烹饪旅行想法的地方 agent时，place agent推荐Carbonara和Bistecca alla Fiorentina(steak).
- Poi agent(5届，poi agent)中的快速到工具集矛盾：兴趣点子agent的提示需要从Google地图定位Lite中校验图像，映射，并放置ID字段，但agent是以空工具列表(工具=[])声明的，因此它编织了占位符 https://example.com/... URL.
图6. 按受影响会议排列的旅行接待网的核实见解。
在仪表盘中打开顶端的洞察力可以显示该组的全部细分情况：匹配会话的次数(15个痕迹)，核查摘要，以及15个关联会话的痕迹与引发它的确切结果：
图7. 仪表板上的透视详细视图，显示故障摘要和15个链接的会话痕迹。
点击“从洞察卡发布”中调查根因诊断agent(Gemini 3.8 Flash)与订正1的33文件源快照。 交叉引用寄存器树上的15个失败的痕迹，agent 追踪错误到旅行 concierge/sub agents/ planning/prompt.py: fli
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
OCT. 8, 2026
Your agent returns HTTP 200, stays inside its latency budget, and no tool call raises an error. Yet it still records seat 3A as confirmed without ever asking the seat-selection sub-agent whether 3A is free, or recommends a Florentine steakhouse to a traveler whose profile says vegan.
Every team we work with that runs an agent in production is trying to answer the same three questions: How is my agent doing? What are its loss patterns, the ways it fails that keep coming back? And how do I climb: change something, know it helped, and keep it from sliding back?
Getting an agent to the first 80%, where it handles the test cases you anticipated, can be relatively straightforward and fast today. An evaluation dataset, a coding agent, and a tight inner loop of run, grade, fix, and compare usually get you to task success on known scenarios in days. Then you launch, and the quality curve flattens or degrades.
Figure 1. Agent quality climbs quickly before launch, then fluctuates in production.
What makes the remaining 20% harder is that the ground moves after launch. Usage shifts as users discover what the agent can actually do, and live traffic stops resembling your initial eval set. At the same time, the system underneath also changes: when you upgrade a model, update your harness, or change a tool or skill, everything still runs and health checks stay green (Figure 2), but conversation quality and task success may shift in ways a standard deploy pipeline never warns you about.
Figure 2. Every infrastructure health check stays green. The failure is silent and happens inside the conversation.
And when a session does go wrong, working out why means separating several layers that can look nearly identical from the outside. For example:
- The model may have hallucinated an argument or dropped a constraint from three turns earlier.
- The orchestration harness may have routed to the wrong sub-agent or lost state between turns.
- A tool contract may have rejected an input because its valid values were never documented in the schema.
- The instructions or skills may simply be missing a rule nobody thought to write down.
Those failure modes can have different owners and different fixes. A raw trajectory records what happened next to what, not what caused what. Telling those layers apart requires reading the failing trajectories alongside the exact code revision that produced them. Online evaluation dashboards tell you that a score moved, and a coding agent can inspect a trace once you know where to look, but at production volume nobody can read every conversation by hand.
Figure 3. AQuA works the production outer loop and feeds what it finds back into your inner loop.
In our June post, we walked that pre-launch inner loop end to end on travel-concierge. This post covers the outer loop, released in the open as composable building blocks you can run in your own project, adapt to your stack, and help us shape as we explore how teams run continuous agent quality in production.
AQuA (Ambient Quality Agent) runs unattended beside your agent in your Google Cloud project, sweeping production trajectories from Cloud Trace, Cloud Logging, or BigQuery on a schedule, after each deployment, or on demand while your laptop is closed. Raw session transcripts, source snapshots, and BigQuery tables stay inside your project boundary, and AQuA never sits in the request path or writes back to your agent.
Figure 4. AQuA runs beside your agent in your Google Cloud project, reading trajectories and deploy-time source snapshots outside the request path.
Each run reads a sample of recent sessions and puts it through a five-stage pipeline:
Figure 5. The five-stage sweep pipeline turns raw production sessions into verified insights, with root-cause diagnosis running on demand against the deployed source snapshot.
- Sample. Pulls a random sample of up to 1,000 recent sessions (capped to keep run cost predictable) from your telemetry (Cloud Trace / Cloud Logging or BigQuery).
- Review. Grades each session against a nine-point checklist of common breakdowns (procedure, tool selection and arguments, grounding, task completion, and others) with the agent's instructions, tools, and optional goal.md in view. When a session fails, it writes a structured actual / expected finding.
- Cluster. Groups session findings that share the same failure mechanism into candidate issue clusters.
- Verify. A separate model checks each candidate cluster against up to three full transcripts and discards clusters the evidence does not support.
- Track. Matches surviving clusters against open insights (verified issues tracked across runs) in BigQuery as NEW, RECURRING, or auto-RESOLVED after 14 days unseen.
Think of it as a junior quality engineer doing the first pass: reading conversations, filtering out noise, and preparing the case file with evidence sessions for whoever is on rotation.
To steer Stage 2 toward your domain, you write a plain-English developer goal (goal.md) that is appended to every review prompt; deterministic Python custom metrics (eval_config.yaml) run alongside the judge to trend pass rates. By default, AQuA uses the single-pass session_review judge, which evaluates the conversation against that checklist in one model call per session (keeping scheduled sweeps economical and producing the actual / expected diffs that drive clustering). You can also opt in to the Gemini platform's managed trajectory AutoRaters (task_success, tool_use_quality, trajectory_quality), which run dedicated per-metric evaluators (for example, if you want to score sessions against standardized out-of-the-box rubrics or align metrics with offline evaluations).
When an insight is worth investigating, you trigger root-cause analysis from the dashboard Chat or agents-cli aqua run. AQuA reads the failing trajectories alongside the immutable source snapshot captured at deploy time: when the defect is in your repo, it cites <path>:<start>-<end> and proposes an edit anchored to the snapshot lines; when the fault lies outside your code (an upstream dependency, handoff, or retrieved payload), it attributes the failure to that step in the trajectory without proposing a code diff. It never applies an edit or opens a pull request on its own.
It sits between the loops you already have: offline evals grade a candidate build against known test cases, online evals monitor pass-rate trends in production, and coding agents or specialized optimizers edit prompts and code. AQuA turns raw production traffic into diagnosed, code-anchored insights and archived failure transcripts that seed your inner loop.
Let's walk a run on travel-concierge (google/adk-recipes), which routes a traveler across sub-agents (inspiration_agent, place_agent, poi_agent, planning_agent, flight_search_agent, flight_seat_selection_agent, booking_agent, and pre_trip / in_trip / post_trip) and stores the working trip in session state via memorize(key, value) (travel_concierge/tools/memory.py).
Attaching AQuA to the travel-concierge project takes three agents-cli commands:
agents-cli extension add "${AQUA_CHECKOUT}"
agents-cli infra single-project --project="${GOOGLE_CLOUD_PROJECT}" --apply
agents-cli deploy --project="${GOOGLE_CLOUD_PROJECT}" --region us-east1
Alongside AQuA's runner, BigQuery dataset, and Cloud Run dashboard (behind Identity-Aware Proxy), agents-cli deploy writes an immutable snapshot of travel-concierge's source tree to Cloud Storage keyed by the deployment revision (Revision 1).
On the dashboard's Configuration page, we save a developer goal (goal.md) to steer the review toward high-level product invariants and suppress stylistic noise (while still requiring every finding to cite a concrete turn where an agent or sub-agent violated its instructions or tool state):
Goal: Help travelers move from trip inspiration to a concrete itinerary and confirmed bookings across our sub-agents, with every confirmed flight, hotel, seat, and recommendation grounded in tool results and the traveler's profile.
Failure modes to make sure we cover:
- Mid-conversation changes (a weak spot in pre-launch testing): if a user updates destination, dates, or flight/hotel choices after an initial plan, make sure subsequent sub-agent tool calls and state updates reflect the change.
- Dropped context when handing off or delegating across inspiration_agent, planning_agent, and booking_agent (such as traveler profile preferences or prior selections).
Ignore: tone, greetings, small talk, or sessions where the user browses options and leaves without booking.
Plain text
Copied
If you also want a deterministic Python rubric in eval_config.yaml to trend pass rates over time, you can publish it alongside the sweep with agents-cli aqua metrics publish.
Across a 32-session multi-agent sweep (32 replays of four scripted traveler journeys, producing 1,583 OpenTelemetry spans across travel_concierge and its sub-agents), 5 sessions pass cleanly and 27 produce 42 structured findings, each scoped by span to the specific sub-agent's isolated instructions and tool declarations. Here is one finding on planning_agent:
actual: When the user selected outbound flight UA204 and requested seats 3A and 3B in the same turn, planning_agent saved those seat numbers directly to session state without calling flight_seat_selection_agent to check whether 3A and 3B were available.
expected: planning_agent should call flight_seat_selection_agent to verify seat availability and pricing before saving outbound or return seat numbers to session state.
Clustering groups those 42 findings into 9 candidate clusters. The verifier (Gemini 3.7 Flash) checks each cluster against up to three full transcripts and each sub-agent's definition, and rejects 3 false-positive clusters: in two, the traveler had explicitly asked to take the first return flight or jump straight to booking; in the third, clustering had merged unrelated prompt-tool mismatches from two different sub-agents (flight_search_agent and inspiration_agent). That leaves 6 verified issues, led by:
- Bypassing seat availability checks when a user volunteers a seat (15 sessions, planning_agent): When a traveler names a seat alongside their flight choice (or right after switching destinations mid-trip), planning_agent writes the seat straight into session state (returning HTTP 200 with zero tool errors) without calling flight_seat_selection_agent to check whether the seat exists or is open.
- Dropping dietary constraints across sub-agent boundaries (7 sessions, inspiration_agent → place_agent): The inspiration agent's prompt includes the traveler's profile (food_preference: vegan), but place_agent is wrapped as an isolated tool whose prompt never receives that profile. When inspiration_agent asks place_agent for culinary trip ideas without passing "vegan" in the request, place_agent recommends Carbonara and Bistecca alla Fiorentina (steak).
- Prompt-to-toolset contradiction in poi_agent (5 sessions, poi_agent): The point-of-interest sub-agent's prompt requires verified image, map, and place ID fields from Google Maps Grounding Lite, but the agent is declared with an empty tool list ( tools=[] ), so it fabricates placeholder https://example.com/... URLs.
Figure 6. Verified insights from the travel-concierge sweep, ranked by affected sessions.
Opening the top insight in the dashboard surfaces the full breakdown for the cluster: the number of matching sessions (15 traces), verification summary, and the 15 linked session traces with the exact findings that triggered it:
Figure 7. Insight detail view in the dashboard, showing the failure summary and the 15 linked session traces.
Clicking Investigate from the insight card launches the root-cause diagnosis agent (Gemini 3.8 Flash) against Revision 1's 33-file source snapshot. Cross-referencing the 15 failing traces against the repository tree, the agent traces the bug to line 93 of travel_concierge/sub_agents/planning/prompt.py: the fli
Auto-fetched and translated by AI Atlas. All rights belong to the original author.