OCT. 15, 2026 (英语).
您的agent可以循环，燃烧符文，或者给出一个完全幻觉的答案，仍然看起来很好。 气温检查不会赶上的校对：Soup 以下是里面的内容：
当团队开始建造AI agent时，测试通常看起来是一样的。 你用电线连接一个模型，交给它一些工具，在终端里运行三到四个提示，并监视输出。 如果答案看起来合理，你把它运送。
在实践中，这种情况迅速崩溃。 您调整了一个系统，以在一个工作流程中修正边框，突然 agent 停止在另一个工作流程中调用所需的工具。
更糟糕的是，agents失败与标准软件不同。 agent可以卡在一个重试循环中，发射多个工具的呼叫，然而仍然返回一个礼貌的，似是而非的最终答案。
在生产中，这种隐性失败的代价很高：
- 它悄悄地刺激了你的帐单和象征性的使用。
- 耗尽下游API的税率限制和配额。
- 它让用户盯着一个装货旋转器。
- 它可以在外部系统中触发重复的副作用，同时在表面看到"罚款".
——侵蚀用户信任，第一印象难取回。
将一个可靠的评价套件纳入到您的开发生命周期中，似乎在初始的原型制作过程中会增加一些间接费用，直到修改您的系统时，关键是认识到评价是一个渐进的过程：任何可重复的评价系统(无论多么简单)都比依靠临时抽查或根本没有系统要好。 你不需要一个复杂的自动分级架构在0日。 相反，从简单，可重复的检查开始，随着你agent的能力和生产需求的增长，你的评价发展成为一个更复杂，自动化的框架。
通过将这些决定纳入评价框架，建立可重复的评价系统大为容易。 过渡超越简单的主观检查取决于两个核心支柱：人的监督和目标明确的工具。 在本篇文章中，我们探索如何使用开源 agent-eval工具包建立评价框架。 我们将展示手动审查的痕迹与自动核查的结合如何帮助发现静态回归早期的开发和无缝的缩放作为您的 agent 向生产过渡。
在写测试代码之前，你需要一个共同的愿景，了解一个好的结果是什么样子。 在关键用户行程(CUJs)中锁定您的测试，而不是抽象的基准测试。
CUJ定义了用户期望系统实现的目标
“作为业务分析员，我期望agent 接收一个对价交易差异电子邮件，找出有争议的交易，核实贸易经济学与我们的内部分类账
该语句中的每个条款都成为评价点：agent是否解决了正确的交易ID？它在起草前有没有检查过分类账？它是否将内部的ID映射回外部名称？
CUJ的每一条款都成为评价点。
界定这些期望使产品利益攸关方、下游用户和工程师商定如何衡量“成功”。
您不需要6个月的生产日志来构建您的初始黄金数据集。 相反，将每个 CUJ 细分为具体情景(既包括快乐路径，也包括边缘案例)，并将其输入一个用户模拟器LLM(我们喜欢ADK用户模拟). 以人物和行为约束为主(例如，不耐烦的分析师给出不完整的票号，或开发者询问多段调试问题)，模拟器反射真实用户的混乱措辞，打字，和多回合复杂度。
ADK用户模拟的示例情景。
通过对照合成用户和现实情景测试你的agent，可以在用户互动前记录全面的执行痕迹(包括工具日志). 这使得您能够根据您的发布标准确定并解决任何性能差距。 随着直播流量的开始流动，你可以逐渐用真正的生产痕迹来取代这些合成装置。
单项上线得分永远不会告诉你为什么agent失败。 为了分离根源，分层进行四类检查：单位测试、确定性测量、LLM-as-a-judge rumics和综合测试。
四层检查，从单位检验到综合检验。
制定评价衡量标准涉及试验和错误，在运行之间删除一个吵闹的测试或交换样本查询是诱人的。 不过，每次加入或删除测试时，都会使基准无效：由于agent变好或者测试套件移动，无法判断一个分数是否改变。 在您调制一个特性的同时，保持您的金色数据集的冻结，一次改变一个变量，只有在您确定基本要求已经改变时，才退出测试。
删除测试会移动基线，而不是agent.
LLMs在定性1比5的分级上臭名昭著地不一致；一个"3"和一个"4"之间的边界模糊不清，经常从运行转向运行。
只要有可能，就把你的法官名词变成布尔(pass/fail)问题，而不是开放式的评分表。 与其问"这个答案从1到5有什么用？"，不如问"agent在拨打退款工具前是否核实过账户ID? (是/否))”。 (我们喜欢适应性Rubrics)
如果通过每一次质量检查的回复需要40秒，发出12次冗余工具呼叫，每次查询花费60美分，则仍然是生产失败。
总是评价整个系统，而不仅仅是最后文本。 轨迹耐久性、象征性消费(即期、推理和完成标志)、API成本和工具调用计数，以及答题质量。
切换管弦乐框架往往迫使团队拆卸并改写其整个测试套件。 虽然大多数框架会发出OpenTeleometry的痕迹，将您在Open Information等开放规格上的痕迹标准化，通过将评价与您的管弦框架脱钩来解决了这个问题。 agent-eval工具箱等工具在本土上支持这个工具，最小设置。
不需要从头开始建立可观察平台。 使用agent-eval工具箱等开源公用设备，在Gemini平台上将OpenTeleometry微量采集与Agent评价服务(原为Vertex AI Expression)搭配起来，可以在几小时内站立评价管道。 但建立你的实际评价策略需要更多的思考。
开始小并保持接近原始数据：
- 涵盖主要CUJ的假设草案，并通过你的agent进行模拟。
- 忽略汇总分数，从上到下读取实际执行痕迹：提示、中间推理步骤、工具参数和最后输出。
- 用红色笔对痕迹进行分级，标记agent犹豫不决，幻觉一个参数，选择错误的工具，或闪烁的每一个地点。
一个agent追踪，用红笔分级。
为什么红笔很重要：与自动生成图表相比，手动读取20个痕迹感觉很慢，但却节省了几天的浪费精力。 而不是猜测假想的失败模式，和写通用的标语错过标记，一个小时的红笔显示您的系统如何在现实中打破。 汇总这些指出让你得到一份准确的确定性断言的高信号列表和布尔法官检查您需要建立下一个。
一旦有了初步的检查清单，agent-eval工具包等工具将接管举重并规范执行。 帮助你的东西，例如：
- 脚手架与执行：脚手架将您的黄金数据集与您的 agent 相比，执行多回合方案，覆盖 HTTP(- base-url) 或离线。
- 框架不可知痕量摄入：通过 OpenInformation 常规获得结构化的 OpenTelection 痕量，无论是用 Google ADK, LangGraph，或自定义运行时间。
- Agent评价整合：将这些痕迹传送到Agent评价服务在Gemini平台上，以运行你的确定性检查和布尔判官名词，而无需讲解API代码。
用你的输油管做一次定期的运行：
- 运行每个更改：每当调整一个提示，交换一个模型，或者编辑一个工具docstring，执行完全相同的测试电池，并比较三角洲与基线。
agent-eval HTML报告中的量度随时间推移而变化。
永远不要盲目相信一个新的LLM 将自动分数与你手定的分数比较如果法官给一个你用红色墨水标记的痕迹打过分收紧分数，直到自动信号符合你的工程判断
一旦您的 Day 1 框架被激活，优化就成为一个有纪律的机械循环：
一个agent样本上的调制环的一转。
在实际部署中，达到生产质量往往需要50至60个小迭代。 您不是在每天早上快速重写整个系统；您正在逐渐转换单个节点。 一个微调澄清了一个工具参数描述。 接下来的新增是一个消极的限制。 接下来会调整JSON输出方案。
例如，每次评价结束后，agent-eval框架都会构建一个互动的、全一的HTML报告。 本报告将通过/失败的布尔度量度与关键操作度量度(包括延迟度、信号使用、API支出、缓存使用和详细的工具调用执行路径)结合起来。 此外，它包含了自动的 Gemini 根因子分析失败的对口转折，确定导致您下个调整失败的准确的即时指令或参数设置。
这些数据取代了预感。 在人工抽查时，瞬间的变化可能感觉要聪明得多，只有测试电池才会显示耐久性翻了一番，工具召唤的幻觉滴答15%。
团队建设agents的课：小的，量度的变化每一次都会击败大改写。
一旦你相信从你的Eval管道发出的信号，就把它变成一个永久回归安全网：
- CI/CD 牵引请求门，带有无头的CLI：在您的构建管道(如Cloud Build或GitHub Actions)中执行像 agent-eval 无头的工具。 在每次拉动请求时，它都会运行你的黄金数据集，执行两项合并规则：
- 全球通过阈值：套房必须符合最低通过条(例如95%的布尔成功率)。
- 核心评价指标的零回归：即使总得分保持稳定，如果以前通过的任何衡量指标倒退，积分也会失败。 这样可以防止开发者意外地为微小的边框改进而交易关键能力。
- 共享的仪表板和集中存储：配置你的eval进程，以导出运行的文物——包括原始追踪的JSONL，即时散列，模型版本，即时剖面图和记分卡摘要——输入大查询或云存储等中央数据仓库。 向共享团队的仪表板发布这些指标，让每个人在发布的准确性，耐久百分位数和象征性预算方面都能看到。
发布之后，真正的生产痕迹流回金色数据集。
链接到 Youtube 视频(仅在 JS 被禁用时才可见)
观看我们的同伴录像，AI Agent：超越Vibe检查
- 抓取代码：在GitHub上克隆开源 agent-eval工具包，用于脚手架数据集，收集痕迹，并生成诊断HTML报告出箱。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
OCT. 15, 2026
Your agent can loop, burn tokens, or give a completely hallucinated answer and still look fine. A vibe check won't catch that. This post is the playbook behind our AI Agent Clinic episode How to Actually Evaluate (Any) AI Agent: Beyond the Vibe Check, where we build an eval pipeline for a LangGraph agent in 60 minutes. Here's what's inside:
When teams start building an AI agent, testing usually looks the same. You wire up a model, hand it a few tools, run three or four prompts in a terminal, and eyeball the output. If the answer looks reasonable, you ship it.
In practice, this breaks down fast. You tweak a system prompt to fix an edge case in one workflow, and suddenly the agent stops calling a required tool in another.
Worse, agents fail differently than standard software. An agent can get stuck in a retry loop, fire off multiple tool calls, yet still return a polite, plausible-sounding final answer.
In production, that hidden failure is costly:
- It quietly spikes your cloud bill and token usage.
- It exhausts downstream API rate limits and quotas.
- It leaves users staring at a loading spinner.
- It can trigger duplicate side effects in external systems, all while looking "fine" on the surface.
- It erodes user trust, and first impressions are hard to win back.
Integrating a reliable evaluation suite into your development lifecycle can seem like extra overhead during initial prototyping, right up until modifying your system prompt causes unexpected breaking changes. The key is recognizing that evaluation is a progressive journey: any repeatable evaluation system (no matter how simple) is better than relying on ad-hoc spot checks or no system at all. You don't need an elaborate automated grading architecture on Day 0. Instead, start with simple, repeatable checks and evolve your evaluation into a more complex, automated framework as your agent's capabilities and production demands grow.
Establishing a repeatable evaluation system is made substantially easier by integrating these decisions into an evaluation framework. Transitioning beyond simple subjective checks relies on two core pillars: human oversight and targeted tools. In this article, we explore how to set up an evaluation framework using the open-source agent-eval toolkit. We'll demonstrate how combining manually reviewed traces with automated verification helps detect silent regressions early in development and scales seamlessly as your agent transitions to production.
Before writing test code, you need a shared vision of what a good outcome actually looks like. Anchor your tests in critical user journeys (CUJs) rather than abstract benchmarks.
A CUJ defines what a user expects the system to achieve, structured around their persona, goals, constraints, and verification conditions:
"As an operations analyst, I expect the agent to ingest a counterparty trade discrepancy email, identify the disputed transaction, verify the trade economics against our internal ledger within 30 seconds, and draft a response reverse-translating our internal trade IDs back into the broker's nomenclature without human intervention."
Every clause in that statement becomes an evaluation point: Did the agent resolve the correct transaction ID? Did it check the ledger before drafting? Did it map internal IDs back to the external nomenclature?
Every clause of a CUJ becomes an evaluation point.
Defining these expectations upfront forces product stakeholders, downstream users, and engineers to agree on defining how “success” is measured.
You do not need six months of production logs to build your initial golden dataset. Instead, break each CUJ down into concrete scenarios (both happy paths and edge cases) and feed them into a user simulator LLM (we like ADK user simulations). Primed with a persona and behavioral constraints (for example, an impatient analyst giving incomplete ticket numbers, or a developer asking a multi-part debugging question), the simulator mirrors the messy phrasing, typos, and multi-turn complexity of real users.
Example scenarios for ADK user simulations.
By testing your agent against a synthetic user and realistic scenarios, you can record comprehensive execution traces (including tool logs) prior to user interaction. This enables you to pinpoint and resolve any performance gaps against your release standards. As live traffic begins to flow, you can progressively replace these synthetic setups with real production traces.
A single top-line score will never tell you why an agent failed. To isolate root causes, layer four types of checks: unit tests, deterministic metrics, LLM-as-a-judge rubrics, and integration tests.
Four layers of checks, from unit tests to integration tests.
Setting up evaluation metrics involves trial and error, and it is tempting to delete a noisy test or swap out sample queries between runs. Every time you add or remove a test, though, you invalidate your baseline: you cannot tell if a score changed because the agent got better or because the test suite moved. Keep your golden dataset frozen while you tune a feature, change one variable at a time, and only retire a test when you are certain the underlying requirement has changed.
Deleting tests moves the baseline, not the agent.
LLMs are notoriously inconsistent at grading qualitative 1-to-5 scales; the boundary between a "3" and a "4" is fuzzy and often shifts from run to run.
Whenever possible, turn your judge rubrics into boolean (pass/fail) questions rather than open-ended rating scales. Rather than asking "How helpful was this answer from 1 to 5?", ask "Did the agent verify the account ID before calling the refund tool? (Yes/No)". (We like Adaptive Rubrics)
A response that passes every quality check is still a production failure if it took 40 seconds, made twelve redundant tool calls, and cost sixty cents per query.
Always evaluate the whole system rather than just the final text. Track latency, token consumption (prompt, reasoning, and completion tokens), API cost, and tool-call counts right alongside answer quality.
Switching orchestration frameworks often forces teams to scrap and rewrite their entire testing suite. While most frameworks emit OpenTelemetry traces, they don’t use the same semantic conventions for tool calls, steps, and tokens. Standardizing your traces around open specifications like OpenInference solves this by decoupling evaluation from your orchestration framework. Tools like the agent-eval toolkit support this natively, with minimal setup.
You do not need to build an observability platform from scratch. You can stand up the evaluation pipeline in a couple of hours using open-source utilities like the agent-eval toolkit which pairs OpenTelemetry trace capture with the Agent Evaluation service on the Gemini platform (formerly Vertex AI Evaluation). But building your actual evaluation strategy requires more thought.
Start small and stay close to the raw data:
- Draft scenarios covering your main CUJs and run simulations through your agent.
- Ignore aggregate scores and read the actual execution traces from top to bottom: the prompts, intermediate reasoning steps, tool arguments, and final outputs.
- Grade the traces with a red pen, marking every spot where the agent hesitated, hallucinated a parameter, picked the wrong tool, or rambled.
One agent trace, graded with a red pen.
Why the red pen matters: Manually reading twenty traces feels slow compared to auto-generating a chart, but it saves days of wasted effort. Instead of guessing at hypothetical failure modes and writing generic rubrics that miss the mark, an hour with a red pen shows you how your system actually breaks in reality. Aggregating those notes gives you a high-signal list of the exact deterministic assertions and boolean judge checks you need to build next.
Once you have that initial list of checks, tools like agent-eval toolkit takes over the heavy lifting and standardizes execution. Helping you with things like:
- Scaffolding and execution: Scaffold your golden dataset and execute multi-turn scenarios against your agent over HTTP (--base-url) or offline.
- Framework-agnostic trace ingestion: Get structured OpenTelemetry traces with OpenInference conventions, whether you build with Google ADK, LangGraph, or a custom runtime.
- Agent Evaluation integration: Routes those traces into the Agent Evaluation service on Gemini platform to run your deterministic checks and boolean judge rubrics without bespoke API code.
Establish a regular cadence of runs with your eval pipeline:
- Run on every change: Whenever you adjust a prompt, swap a model, or edit a tool docstring, execute the exact same test battery and compare the delta against your baseline.
Metrics over time in the agent-eval HTML report.
- Validate the judge against your traces: Never trust a new LLM-as-a-judge metric blindly. Compare the automated scores against the traces you graded by hand on Day 0. If the judge gives a passing score to a trace you marked up with red ink, tighten the grading prompt until the automated signal matches your engineering judgment.
Once your Day 1 framework is active, optimization becomes a disciplined, mechanical loop:
One turn of the tuning loop on a sample agent.
In a real deployment, reaching production quality often takes 50 to 60 small iterations. You are not rewriting the entire system prompt every morning; you are gradually turning individual knobs. One tweak clarifies a tool parameter description. The next adds a negative constraint. The next adjusts the JSON output schema.
For instance, upon completion of each evaluation run, the agent-eval framework constructs an interactive, all-in-one HTML report. This report aligns pass/fail boolean metrics alongside key operational metrics, including latency, token usage, API expenditure, cache utilization, and detailed tool-call execution paths. Additionally, it incorporates automated Gemini root-cause analyses for failed conversational turns, pinpointing the exact prompt directive or parameter setting that triggered the failure to guide your next adjustment.
That data replaces hunches. A prompt change might feel much smarter in manual spot-checking, only for the test battery to reveal that latency doubled and tool-call hallucinations ticked up by 15%.
The lesson for teams building agents: small, measured changes beat big rewrites every time.
Once you trust the signal coming out of your eval pipeline, turn it into a permanent regression safety net:
- CI/CD pull request gates with headless CLI: Execute tools like agent-eval headlessly in your build pipeline (such as Cloud Build or GitHub Actions). On every pull request, it runs your golden dataset and enforces two merge rules:
- A global pass threshold: The suite must meet a minimum pass bar (for example, 95% boolean success).
- Zero regressions on core evaluation metrics: Even if the overall score holds steady, the build fails if any previously passing metric regresses. This prevents developers from accidentally trading a critical capability for a minor edge-case improvement.
- Shared dashboards and centralized storage: Configure your eval processes to export run artifacts—including raw trace JSONL, prompt hashes, model versions, latency profiles, and scorecard summaries—into a central data warehouse like BigQuery or Cloud Storage. Publishing these metrics to a shared team dashboard gives everyone visibility into accuracy, latency percentiles, and token budgets across releases.
After launch, real production traces flow back into the golden dataset.
Link to Youtube Video (visible only when JS is disabled)
- Watch the walkthrough: Check out our companion video, How to Actually Evaluate (Any) AI Agent: Beyond the Vibe Check, where we build this whole pipeline live on a real LangGraph agent, in 60 minutes.
- Grab the code: Clone the open-source agent-eval toolkit on GitHub to scaffold datasets, collect traces, and generate diagnostic HTML reports out of the box.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.