OCT. 28, 2026 (英语).
Agents CLI eval 环路的实时行走，从第一次运行到回归跟踪，模型后端实验和自动提醒
构建在演示上工作的agent与构建在模型升级，工具变更，以及新对话形状之间保持工作的agent是一个不同的问题。 第一个是周末；第二个是循环。 一个团队写测试案例，运行，分级，读取失败之处，更改agent，再运行案例。 如果分数上升，变化是好的。 如果它下降，返回。 数周后，agent向天花板走去模型，工具，和迅速可以一起做的。 质量工程名称是攀登山峰。
Agents CLI eval namespace是围绕这个环构建的。 指令小而框架不可知：eval生成运行agent与一个数据集对齐，eval等级得分生成的痕迹，eval比较 diffs两个运行结果并列，eval将集群失败分析成主题，eval优化运行一个自动的丘陵爬行传递到 agent的提示。
下面：基线回归模型互换调度和提醒。
整个使用的脚手架是LangChain模板，其舰载Agents CLI v1.8.0. 设置为单项命令：
agents-cli 创建我的agent\
-agent google/agents-cli/扩展/langchain/template@v1.8.0\
- 运行云
cd 我的agent agents-cli安装
纯文本
复制
在测试/eval/datasets/basic-dataset.json上装有eval数据集的脚手架船，在测试/eval/eval config上装有配置。 yaml，以及测试/eval/response squality.py时的定制的LLM-作为判断标准。 配置宣布要运行的度量：
# 测试/ eval/ eval config. yaml
参数运行：
- 自定义响应质量
自定义( M):
- 名称：自定义响应质量
自定义函数文件：回复质量.py
以gemini-3.8-flash作为法官，按1至5的尺度对每份答复进行准确性、相关性和清晰度的衡量。 默认数据集有一个单一的无足轻重的大小写，这不足以让山坡爬行。 我们要做的第一件事就是把它扩展为四个探索不同行为的案例：
{
"eval_cases": [
{"姓名":"欢呼"
"agent data":{"转":{{"事件":{"作者":"用户","内容":{"部分":{{"文本":"你好"}}}}}}}{{
{"名称":"sf weather"
"agent data":{"转":{{"事件":{"作者":"用户","内容":{"部分":{{"文字":"旧金山的天气是什么？"}}}}}}}}}}{{}
{"名称":"rome weather"
"agent data":{"转身":{{"事件":{"作者":"用户","内容":{"部分":{{"文字":"今日罗马天气如何？"}}}}}}}}}{{
{"名称":"off topic"
"agent data": {"转弯":{{"事件":{"作者":"用户","内容":{"部分":{{{"文字":"谁赢得2024年世界大赛？}}}}}}}}}{{{
]
}
四个案例：一个问候，一个模拟工具用硬码的雾化反应处理的天气问题，一个工具默认在的城市的天气问题，以及一个非主题的问题，agent必须决定如何处理。 这是一个小的数据集，通过设计；在数据集小到可以运行在每个变化上时，丘陵攀登的环路会得到回报。
随着数据集的建立，我们运行 eval 生成驱动 agent 对每一个案例，然后 eval 等级得分由此产生的痕迹：
agents-cli eval生成 -- dataset tests/eval/datasets/basic-dataset.json \
-o 测试/评估/产出/
agents- cli eval 等级 - 跟踪测试/ eval/ 输出/ eval dataset\\\. json\
-- 配置测试/ eval/ eval config. yaml
基线运行产生：
评价摘要
自定义( S):
数字 case 总计：4
数字 case 验证：4
数字 case 错误：0
中点数：4. 0000
积分：1.4142
将全部结果保存到文物/等级-成果/成果 20261003 224516.json
HTML结果保存到文物/等级结果/结果 20261003 224516.html
4个案件中5个中的4.0个平均数，标准差1.41。 这个stdev立即告诉我们，这些案例不是统一的评分，这是比整个5.0平方英尺更健康的起点。 平面5.0要么意味着agent是完美的(不太可能)，要么数据集太容易显示变异。 真正蔓延的4.0是一个要攀登的山丘。
为了证明这个循环，我们引入了一个现实的回归。 一个在几个agent项目上工作的开发者不慎将错误的系统粘贴到我们的天气中 agent：一个厨师人物，意为不同的代码库。 在app/agent.py中：
{\fn方正粗倩简体\fs12\an8\1cHFFFF00\b0}之前
root agent：编译状态图形=创建 agent(
model=llm，工具=[get weather], system prompt="你是一个有帮助的助手"
)
{\fn方正粗倩简体\fs12\an8\1cHFFFF00\b0}之后
root agent：编译状态图形=创建 agent(
模型=llm，工具=[get weather]
系统即时= ()
"你是一个专业的厨师。 帮助使用者掌握食谱、烹饪 "
"技术，和成分替代。 总是建议一种食谱 "
"在每个回应的结尾"
),
)
变化编译，agent仍然运行，在预eval输油管(lint，单位测试，型号检查)中没有任何东西捕获。 重运行相同的 eval 生成加 eval 级序列生成：
评价摘要
自定义( S):
数字 case 总计：4
数字 case 验证：4
数字 case 错误：0
中点：2.7500
积分：0.9574
将全部结果保存到文物/等级-结果/结果 20261003 225158.json
贾森
平均2.75，从4.0下降。 法官同意我们的直觉天气下的厨师agent是比通用辅助产品更糟糕的产品 Stdev也从1.41收紧到0.96，这与厨师迅速将大多数病例推向同一平庸范围一致。
1至5级的1.25点降幅相当大，当diff看似小("一个字符串字元变化")时，那种可以滑过代码审查的回归。 在agent达到生产之前，环路完全表面呈现了变化，并附有特定的编号。
平均分数为起始数；每例diff是丘陵攀登的发生地。 eval比较读取两个结果文件并生成结构化的JSON diff:
agents- cli eval 比较\\
文物/等级成果/成果 20261003 224516.json\
文物/等级成果/成果 20261003 225158.json
输出列出哪些密钥已更改，哪些则保持不变：
{
"碱基键":["agent info","eval case results","评价数据集"
"元数据","摘要度量衡","胜率"]
"candidate keys":["agent info","eval case results","评价数据集"
"元数据","摘要度量衡","胜率"]
"更改的键": ["eval case results", "metadata.create timestamp"] (中文(简体) ).
"未更改的 keys":["agent info","评价 dataset","元数据"
"总结度量衡","胜率"]
"分歧":{... 每例得分 diffs...}
}
差异对象与基线和候选值并列，按个案分列。 对于一个丘陵攀爬的工作流程，我们通常通过jq或一个小的Python脚本来管道输出以提取信号；原始的JSON是完整的证据。
浏览两个结果文件
厨师促导的基线候选人
问候5.0 发布进入"你好"之后的无意菜谱投注
sf weather 4.0 2.0 忽略了天气工具，幻觉状态
rome 天候2.02.0 · 在基线上已经产生幻觉；厨师没有使情况恶化
5.0 3.0回答道奇们的提问
3个案件撤诉，1个被扣押。 其中最重要的是sf weather：厨师人物将模型的注意力从注册的get weather工具中拉开，因此agent停止调用，开始编造天气值。 单位测试无法捕捉到这一点：代码运行，工具仍然被注册，但模型围绕它的行为已经转变。 只有一个端到端的 eval 在上下文中评分。
平均下降本身(−1.25)是一个清晰的信号，但每个案例的分解是告诉开发者在哪里寻找的。 一个单行系统即时变化通过三个不同案例以三种不同的方式传播：欢迎在基调上失去一点，sf weather在工具使用上失去两个，在范围爬行上失去两个。
一个常见的攀登山峰的问题是，一个更便宜的或更快的模型是否能够令人接受地完成这项工作。 对于LangChain模板，该模型在app/agent.py中定义：
LLM = "gemini-3.8-闪电"
llm = Chat GoogleGenerativeAI(型号=LLM，温度=0)
我们把gemini-3.8-flash换成gemini-2.5-flash(恢复有用的系统提示第一)，重新运行eval生成加eval等级，比较两个结果文件。 平均从4.00下降到2.75，与厨师-即时回归的大小相同，但每例的崩溃讲述了一个完全不同的故事：
案件3.8-闪光2.5-闪光
问候 5.0 5.0
sf 天气为4.0 3.0
rome_weather 2.0 → 2.0 ·
离主题5.0 1.0
更便宜的型号在问候和rome weather案例上正好匹配3.8-flash，并在sf weather上损失了1分(略少抛光的用词). 整个可见的痕迹投向了离题专题，更便宜的模型拒绝用套期
教训是，汇总数字具有误导性。 你只看那个恶毒的，你会叫gemini-2.5-flash 和厨师的提示一样糟糕。 看看每例破案情况，你称这为agent大部分工作量的合理交换，一个案件(处理近期事件外的问题)暴露出一个真正的限制。 如果用户提问的制作分布与前三例比第四例更匹配，则交换值得考虑。 如果非主题类问题很常见，则不然。
关于这个实验有两个警告。 首先，模型互换与快报交互：快速调制更强大的模型在移植到弱报时往往会泄露能力，因此正确的比较是"弱报模式与快报调制"与"强报模式与自己调制的快报"相对。 其次，Vertex AI Eval Service调用一个法官模型来对痕迹进行分级，法官在整个比较中恒定。 更换法官是一种不同的试验，应当单独进行。
Agents CLI不载运调度器，但eval命令被设计为降入标准基础设施。 我们可以使用的模式是 cron 工作或 Cloud 调度器触发器，运行 eval 生成加 eval 等级于 cadence 上，并将结果写到持续的位置：
# / usr/bin/env bash # 音乐会的音乐会
# 脚本/ 排程 eval.sh —— 由 cron 或 Cloud 排程器在晚上引用
设置 - euo 管道故障
cd "$(dirname "$0")/.."
日期=美元(日期=%Y%m%d)
Out=艺术/ 计划运行/ $DATE
毫迪尔 - p $OUT
agents-cli eval生成 -- -- 数据集测试/eval/数据集/递归。 json - o $UT/
agents- cli val 级 - traces $OUT/ eval dataset #%.json \
-- 配置测试/ eval/ eval config. yaml \
产出
晚间运行在稳定的回归数据集捕获从模型更新中漂移，agent工具的数据变化，以及agent脚下移动的依赖性。 对于一个制作的agent，计划运行也捕捉模型供应商的静态升级，这可以移动agent的行为而无需我们方面有任何改变。 计划运行的数据集通常大于内环中使用的数据集(几十到数百个案例而不是少数)，因为它每天运行一次。
计划运行每天生成一个结果文件，但一个没人看的文件不是一个提醒。 这里的图案可以是将每个新运行的平均值与基线(前一次运行，或承诺的已知善)进行比较的包装脚本，当下降超过阈值时退出非0. 非零退出是触发来自周围调度器的警报的原因。
以下是一个例子：
#!/usr/bin/env 蟒蛇3
""如果一个eval结果在基线下倾斜超过-阈值，则退出非零。"
导入参数、 json、 ys
def 平均值(路径):
以打开( 路径) 作为 f:
返回 json. load(f) ['摘要度量衡' [0]['意为分数']
p=rg. 参数Parser ()
p.add 参数('candidate')
p.add argument ('- basine', required=True) (英语).
p.add 参数('-门槛'，类型=浮动)
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
OCT. 28, 2026
A live walkthrough of the Agents CLI eval loop, from a first run to regression tracking, model-backend experiments, and automated alerting
Building an agent that works on a demo is a different problem from building an agent that stays working across model upgrades, tool changes, and new conversation shapes. The first takes a weekend; the second takes a loop. A team writes test cases, runs them, grades them, reads what failed, changes the agent, and runs the cases again. If the score goes up, the change was good. If it goes down, revert. Over weeks of this, the agent inches toward the ceiling of what the model, tools, and prompt can do together. The quality-engineering name for this is hill-climbing.
The Agents CLI eval namespace was built around this loop. The commands are small and framework-agnostic: eval generate runs the agent against a dataset of cases, eval grade scores the generated traces, eval compare diffs two run results side by side, eval analyze clusters failures into themes, and eval optimize runs an automated hill-climbing pass on the agent's prompt.
Below: baseline → regression → model swap → scheduling and alerting.
The scaffold used throughout is the LangChain template that ships with Agents CLI v1.8.0. The setup is a single command:
agents-cli create my-agent \
--agent google/agents-cli/extensions/langchain/template@v1.8.0 \
-d cloud_run
cd my-agent && agents-cli install
Plain text
Copied
The scaffold ships with an eval dataset at tests/eval/datasets/basic-dataset.json, a config at tests/eval/eval_config.yaml, and a custom LLM-as-judge metric at tests/eval/response_quality.py. The config declares the metric to run:
# tests/eval/eval_config.yaml
metrics_to_run:
- custom_response_quality
custom_metrics:
- name: custom_response_quality
custom_function_file: response_quality.py
The metric grades each response on a 1-to-5 scale for accuracy, relevance, and clarity, using gemini-3.8-flash as the judge. The default dataset has a single trivial case, which is not enough for a hill-climb. The first thing we do is expand it to four cases that probe different behaviors:
{
"eval_cases": [
{"name": "greeting",
"agent_data": {"turns": [{"events": [{"author": "user", "content": {"parts": [{"text": "hello"}]}}]}]}},
{"name": "sf_weather",
"agent_data": {"turns": [{"events": [{"author": "user", "content": {"parts": [{"text": "What is the weather in San Francisco?"}]}}]}]}},
{"name": "rome_weather",
"agent_data": {"turns": [{"events": [{"author": "user", "content": {"parts": [{"text": "How is the weather in Rome today?"}]}}]}]}},
{"name": "off_topic",
"agent_data": {"turns": [{"events": [{"author": "user", "content": {"parts": [{"text": "Who won the 2024 World Series?"}]}}]}]}}
]
}
Four cases: a greeting, a weather question the mock tool handles with the hardcoded foggy response, a weather question for a city the tool defaults on, and an off-topic question that the agent has to decide how to handle. This is a small dataset by design; the hill-climbing loop pays off when the dataset is small enough to run on every change.
With the dataset in place, we run eval generate to drive the agent against each case, then eval grade to score the resulting traces:
agents-cli eval generate --dataset tests/eval/datasets/basic-dataset.json \
-o tests/eval/output/
agents-cli eval grade --traces tests/eval/output/eval_dataset_*.json \
--config tests/eval/eval_config.yaml
The baseline run produces:
Evaluation Summary
custom_response_quality:
num_cases_total: 4
num_cases_valid: 4
num_cases_error: 0
mean_score: 4.0000
stdev_score: 1.4142
Saved full results to artifacts/grade_results/results_20261003_224516.json
Saved HTML results to artifacts/grade_results/results_20261003_224516.html
Mean 4.0 out of 5 across four cases, with a standard deviation of 1.41. That stdev tells us immediately that the cases are not scoring uniformly, which is a healthier starting point than a flat 5.0 across the board. A flat 5.0 either means the agent is perfect (unlikely) or the dataset is too easy to show variation. 4.0 with real spread is a hill to climb.
To demonstrate the loop, we introduce a realistic regression. A developer working across several agent projects accidentally pastes the wrong system prompt into our weather agent: a chef persona meant for a different codebase. In app/agent.py:
# Before
root_agent: CompiledStateGraph = create_agent(
model=llm, tools=[get_weather], system_prompt="You are a helpful assistant"
)
# After
root_agent: CompiledStateGraph = create_agent(
model=llm, tools=[get_weather],
system_prompt=(
"You are a professional chef. Help users with recipes, cooking "
"techniques, and ingredient substitutions. Always suggest a recipe "
"at the end of every response."
),
)
The change compiles, the agent still runs, and nothing in the pre-eval pipeline (lint, unit tests, type checks) catches it. Re-running the same eval generate plus eval grade sequence produces:
Evaluation Summary
custom_response_quality:
num_cases_total: 4
num_cases_valid: 4
num_cases_error: 0
mean_score: 2.7500
stdev_score: 0.9574
Saved full results to artifacts/grade_results/results_20261003_225158.json
JSON
Mean 2.75, down from 4.0. The judge agrees with our intuition that a chef persona on a weather agent is a worse product than a general-purpose helpful assistant. The stdev also tightened from 1.41 to 0.96, consistent with the chef prompt pushing most cases toward the same mediocre range.
A 1.25-point drop on a 1-to-5 scale is substantial, and the kind of regression that can slip through code review when the diff looks small ("one string literal changed"). The loop surfaced the change at all, with a specific number attached to it, before the agent reached production.
Mean scores are a start; the per-case diff is where the hill-climb happens. eval compare reads two result files and produces a structured JSON diff:
agents-cli eval compare \
artifacts/grade_results/results_20261003_224516.json \
artifacts/grade_results/results_20261003_225158.json
The output lists which keys changed and which stayed the same:
{
"baseline_keys": ["agent_info", "eval_case_results", "evaluation_dataset",
"metadata", "summary_metrics", "win_rates"],
"candidate_keys": ["agent_info", "eval_case_results", "evaluation_dataset",
"metadata", "summary_metrics", "win_rates"],
"changed_keys": ["eval_case_results", "metadata.creation_timestamp"],
"unchanged_keys": ["agent_info", "evaluation_dataset", "metadata",
"summary_metrics", "win_rates"],
"differences": { ... per-case score diffs ... }
}
The differences object has the full per-case breakdown with the baseline and candidate values side by side. For a hill-climbing workflow, we typically pipe the output through jq or a small Python script to extract the signal; the raw JSON is the complete evidence.
Walking the two result files and lining up the scores per case shows where the regression landed:
case baseline → candidate what the chef prompt did
greeting 5.0 → 4.0 ↓ launched into an unsolicited recipe pitch after "hello"
sf_weather 4.0 → 2.0 ↓ ignored the weather tool, hallucinated conditions
rome_weather 2.0 → 2.0 · already hallucinated at baseline; chef persona did not make it worse
off_topic 5.0 → 3.0 ↓ answered the Dodgers question but tacked on a recipe
Three cases dropped, one held. The most important of the three is sf_weather: the chef persona pulled the model's attention away from the registered get_weather tool, so the agent stopped calling it and started making up weather values. Unit tests can't catch this: the code runs, the tool is still registered, but the model's behavior around it has shifted. Only an end-to-end eval that scores the response in context surfaces this.
The mean drop by itself (−1.25) is a clear signal, but the per-case breakdown is what tells the developer where to look. A one-line system-prompt change propagated through three different cases in three different ways: greeting lost a point on tone, sf_weather lost two on tool usage, off_topic lost two on scope creep.
A common hill-climbing question is whether a cheaper or faster model would do the job acceptably. For the LangChain template, the model is defined in app/agent.py:
LLM = "gemini-3.8-flash"
llm = ChatGoogleGenerativeAI(model=LLM, temperature=0)
We swap gemini-3.8-flash for gemini-2.5-flash (restoring the helpful system prompt first), re-run eval generate plus eval grade, and compare the two result files. The mean drops from 4.00 to 2.75, the same magnitude as the chef-prompt regression, but the per-case breakdown tells a completely different story:
case 3.8-flash → 2.5-flash
greeting 5.0 → 5.0 ·
sf_weather 4.0 → 3.0 ↓
rome_weather 2.0 → 2.0 ·
off_topic 5.0 → 1.0 ↓↓
The cheaper model matched 3.8-flash exactly on the greeting and rome_weather cases and lost a point on sf_weather (slightly less polished phrasing). The whole visible drop traces to off_topic, where the cheaper model refused with a hedge about the event "not having happened yet" (a training-cutoff artifact), and the judge marked it harshly.
The lesson is that the summary number is misleading. Look only at the mean and you'd call gemini-2.5-flash as bad as the chef prompt. Look at the per-case breakdown and you'd call it a reasonable swap for most of the agent's workload, with one case (handling out-of-scope questions about recent events) exposing a real limitation. If the production distribution of user questions matches the first three cases more than the fourth, the swap is worth considering. If off_topic-style questions are common, it is not.
Two caveats on this experiment. First, model swaps interact with the prompt: a prompt tuned for a stronger model often leaks capability when ported to a weaker one, so the right comparison is "weaker model with a prompt tuned for it" versus "stronger model with its own tuned prompt." Second, the Vertex AI Eval Service calls a judge model to grade the traces, and the judge is held constant across the comparison. Changing the judge is a different experiment and should be done separately.
Agents CLI does not ship a scheduler, but the eval commands are designed to drop into standard infrastructure. The pattern we can use here is a cron job or Cloud Scheduler trigger that runs eval generate plus eval grade on a cadence and writes the results to a persistent location:
#!/usr/bin/env bash
# scripts/scheduled_eval.sh — invoked nightly by cron or Cloud Scheduler
set -euo pipefail
cd "$(dirname "$0")/.."
DATE=$(date +%Y%m%d)
OUT=artifacts/scheduled_runs/$DATE
mkdir -p $OUT
agents-cli eval generate --dataset tests/eval/datasets/regression.json -o $OUT/
agents-cli eval grade --traces $OUT/eval_dataset_*.json \
--config tests/eval/eval_config.yaml \
--output $OUT/
A nightly run against a stable regression dataset catches drift from model updates, data changes in the agent's tools, and dependencies that shift under the agent's feet. For a production agent, the scheduled run also captures the model vendor's quiet upgrades, which can move the agent's behavior without any change on our side. The dataset for the scheduled run is typically larger than the one used in the inner loop (dozens to hundreds of cases rather than a few), since it only runs once a day.
The scheduled run produces a result file every day, but a file nobody looks at is not an alert. The pattern here can be a wrapper script that compares each new run's mean against a baseline (the previous run, or a committed known-good) and exits non-zero when the drop exceeds a threshold. The non-zero exit is what triggers the alert from the surrounding scheduler.
Here is an example:
#!/usr/bin/env python3
"""Exit non-zero if an eval result dips below baseline by more than --threshold."""
import argparse, json, sys
def mean(path):
with open(path) as f:
return json.load(f)['summary_metrics'][0]['mean_score']
p = argparse.ArgumentParser()
p.add_argument('candidate')
p.add_argument('--baseline', required=True)
p.add_argument('--threshold', type=float,
Auto-fetched and translated by AI Atlas. All rights belong to the original author.