消费者Agent测试工具(CATT)测试助理如何使用你的服务。 运行一个脚本或模型驱动的agent对抗MCP服务器，MCP应用程序，一个网站或HTTP API. 检查一下服务做了些什么，检查了证据，并比较了整个建筑的成果。
Muse和Instinct 被支持的替代模拟目标。 Muse 仍然是默认值。 两者都使用共享连接器测试合同，包括路由，内存，支付，浏览器/HTTP任务和调度。 其来源和限度见"永定研究". 驱动审判的模型与被模仿的助手是分开的。
{\fn黑体\fs22\bord1\shad0\3aHBE\4aH00\fscx67\fscy66\2cHFFFFFF\3cH808080}提供"安全" {\fn黑体\fs22\bord1\shad0\3aHBE\4aH00\fscx67\fscy66\2cHFFFFFF\3cH808080}MIT许可证
试宿者。 CATT是一个代孕者：它大致是一个助手的
主办方，而不是私人执行。 考试不合格
与 Muse 或任何主机助理兼容。 每个Muse特定的行为
有记录依据：有记录的、非正式的、假定的或未知的。
此项目不隶属于或认可Meta,Instinct,OpenAI或Anthropic.
连接道记录每个Muse特定行为的基础。 工具类
并遵循有文件记载的准则。 缪斯-克利投影
模板客户端遵循非官方镜像；本地投影是假设
通用基线。 中性cli投影支持两个目标作为CATT
治疗。 Instinct 运行有一个单独的源记录并标签为
共享主机算法作为假设。 本地/CLI MCP是CATT的测试适配器；
没有找到公开的Instinct MCP整合合同。
在 Instinct 下运行本地预定示例，无需模型或账户：
uv运行消费者-agent-测试工具-连接器-运行实例/连接/intinct-booking.yaml \
-- driver 脚本 -- spect example/ connectionor/ fixture-booking.calls.json \
-- start "python - m client agent testing tool. connector fixture - port {port} - 变量好 - 无"
设定仿真目标：在您的情景或通过中的本能
- 模拟-目标直觉。 两个目标都使用本地或cli投影和通用的
MCP客户端。 默认为纯文本回复，而不运行 Muse 准则
检查。 它的本地钱包固定覆盖支付行为；实时支付 rails
信任的网络协议、 断层和文件仍然位于
共享合同。
三个快速启动。 连接器道和流行的MCP服务器快速启动在无账户的回转上运行；覆盖猫沙盒快速启动称为主机沙盒。 测试您自己的服务将它们变成一个工作流程，文档索引列出每个页面。
您需要 Python 3. 12 或更新( Linux 测试 3. 12, macOS 测试 3. 14)
和20号节点或新节点
已安装的软件包声明 QQ20；测试为 25. 9。 记录的释放
证据涵盖macOS。 从此寄存器的克隆中安装 Python 和
节点依赖和铬构建浏览器通道使用：
uv 同步 -- extra 打开路由器 -- extra dev
npm - 前缀工具 ci - ignore 语句
npm - 前缀工具/支付-ii ci - ignore-scripts
PLAYWRIGHT BROWSERS PATH="$PWD/.cache/ms-playwright"节点工具/支付-ui/node modules/playwright/cli.js 安装铬
连接道、准备状态检查和脚本运行只需
{\fn黑体\fs22\bord1\shad0\3aHBE\4aH00\fscx67\fscy66\2cHFFFFFF\3cH808080}请说节点和铬是用于MCP Apps和浏览器车道。
在Linux(Debian或Ubuntu)上，代之以用其系统库安装Chrimium. Linux没有被记录的释放证据覆盖：下面的命令符合CI工作流程，但在此没有被限定。
sudo apt-get 更新 QQ sudo apt-get 安装 -y nodejs npm # 或节点 20+ 从节点源或 nvm
卷曲 - LsSf https://astral.sh/uv/install.sh QQ sh
uv 同步 -- extra 打开路由器 -- extra dev
npm - 前缀工具 ci - ignore 语句
npm - 前缀工具/支付-ii ci - ignore-scripts
PLAYWRIGHT BROWSERS PATH="$PWD/.cache/ms-playwright" (中文(中国大陆) ).
节点工具/支付-i/node modules/playwright/cli.js 安装-带有-deps铬
3个功能依赖于 macOS 安全带( sandbox-exec) 并报告不支持
或在 Linux 上被屏蔽，而不是运行未关闭：-- 离线准备
模式，即网络拒绝验证员的运行后工作 -- -- 表面辅助工具，以及
本地模型内存简介。 连接道，脚本运行
准备状态检查和Docker后卫的 stdio目标并不取决于它。
与医生检查设置：
经营消费者-agent测试工具医生
它检查节点与地板安装的软件包申报
工具/付费-义包对照其固定版本，以及铬和
它的无头壳在.cache/ms-playwright下。 它报告任择
协议工具，并检查 Docker 只用于开发通道或配置
斯特迪奥MCP和CLI目标。 它不提出模型请求，即使存在
配置或 --model-profile选择一个模型。 每次检查失败或被屏蔽
打印 Stderr 上的一行及其理由，对于设置问题，命令
那样就解决了
明确运行提供机前飞行，发送两个简短的型号请求
并可能收取费用，加上 -- 型号飞行：
uv 运行消费者-agent-测试工具医生-配置 agent.yaml-模型-飞行前
脚本运行不需要模型账户，脚本运行命令全部三个
快速启动不需要配置文件。 Agent运行需要模型配置和
API密钥；模型提供者显示如何创建。 供应
通过环境变量获取证书，使其远离设想情景和
共享证据。 消费者- agent- testing-tool.yaml 默认设置文件
是本地和 git- ignored 的，没有发送配置文件。
这种快速启动保持回转，不需要账户。 完成设置，然后在第二个终端启动Cedar Stays型测试服务。 它在准备就绪时用它的URL打印一条JSON线；Ctrl-C停止：
uv 运行 python - m 消费者 agent 测试 tool. connector fixture - port 8765 - 变量良好 - 无
重放实例的脚本调用，没有模式：
uv运行消费者-agent-测试工具-连接器-运行实例/连接器/固定-预订。 雅姆尔
--driver 脚本 --scription example/connectionor/fixture-booking.calls.json 页面存档备份，存于互联网档案馆互联网档案馆的存檔，存档日期2013-12-22.
剧本搜索室，获得引文，书籍并给出最终答案。 那个
主机从假想的私人：值来填充嘉宾的电子邮件：
模式从不看，脚本的用户回答来自
场景权限：规则。 运行后，主机检查固定器
私人 oracle(一次确认预订，不重复收费)和预订
状态通过 get booking。 它还检查，每个发送的电话都与它的
批准卡(批准、发牌、批准、公布)和
最后的答复没有提出虚假的主张，并转发了总数和
确认(答复.no false 声称，答复.relay complete).
命令打印运行目录和连接器报告：.../report.html
并在每一次所需检查通过时退出 0。 运行被封存到
运行/连接-<id>/. 共享导出输出，而不是运行目录：
它的私有/ 文件夹持有主机- 私有值。
对于 agent 运行，请创建 agent. yaml，并使用 init 命令。
模型提供者，将OPENROUTER API KEY设置在您的外壳中，并运行：
uv运行消费者-agent-测试工具连接器-运行实例/连接器/固定-预订。 yaml-config agent.yaml
模型飞行前先运行，并打印自己的证据路径。 那么 agent 开始
将四个允许的工具视为本地功能，并使用主机- 私人电子邮件
从他们的计谋中移除，然后自己写下其他论点。 这个用
支付模型推论。 改变治疗方式而不编辑情景
-- project cli 通过一个执行命令为任一目标提供相同的工具
-- 暴露只搜索通过工具搜索来加载它们的参数
- 输出- schema 严格拒绝破坏工具输出的结果
官方 MCP SDK 这样做，和 -- 审判 N 进行独立审判和打印
过关了运行锁记录了投影、曝光、客户形象和
输出-schema 在覆盖下覆盖。
连接器运行的退出 0 每一次需要的检查通过；1 一次检查
未成功，答案是虚假的，或者 agent 限额已达；2 对 a
设置、控制或报告错误；和 3 当试验受阻或
不支持，例如用不确定的文字来调和，即不可观察
检查或客户描述限制。 要从自己的服务器开始，请运行
脚手架；在没有模型的情况下，重播一个可分享的 agent 试验
把它写成一个有脚手架的呼叫文件 -- 从运行。 那个
连接器车道参考涵盖情景、旗帜和
证据。 实例
列出每个已运出的情况及其固定命令，以及
通道扩展覆盖路线，自定义
连接器、浏览器子agent、钱包、时间表和其他
扩展。
连接器道与任何 MCP 服务器在可流的 HTTP 之上工作，而不仅仅是
捆绑测试服务。 这个快速启动点在"一切"
(@ modelcontextprotocol/ server- allthings) 互联网档案馆的存檔，存档日期2014-03-02.
MIT)，为客户端作者维护的参考服务器MCP项目。 内容
天气工具返回模拟数据，所以不需要账户或密钥。 开始吧
第二个终端；第一次运行下载 npm 的绑定发布：
PORT=3001 npx -y @modelcontextprotocol/server-Everything@1.0.0 可流 Http
看主持人从直播目录中推断什么。 脚手架在握手
和工具/列表，不调用工具，并用建议书写情景骨架
来自服务器说明的工具类：
uv运营消费者-agent-测试工具脚手架 http://127.0.0.1:301/mcp 把我的一切都说出来
uv运行消费者-agent-测试工具检查 http://127.0.0.1:301/mcp - 套装缪斯指南
第二命令静静地对照钉住的Muse连接器准则审查目录。 然后重放已运出的例子，它要求芝加哥的天气，首先使用本土工具，然后通过Muse风格的命令行投影在非正式的Muse模板客户端上：
uv运行消费者-agent-测试工具-连接器-运行实例/连接器/一切-天气。 雅姆尔
--driver 脚本 --scription example/ connectionor/ everything-weather.calls.json 页面存档备份，存于互联网档案馆。
uv运行消费者-agent-测试工具-连接器-运行实例/连接器/一切-天气。 雅姆尔
-- projection muse-cli - 客户简介 muse-template-unofficial - 2026-10\\x
--driver脚本 --script example/ connectionor/ everything-weather-cli.calls.json 页面存档备份，存于互联网档案馆
两个出口都是0号出口设想提供了服务器工具的模型三；a
经审查的许可名单将其余部分排除在清单之外，包括 " 获得-env "。
返回服务器进程的环境变量，以及
gzip- file- as- resources，获取它给定的任何 URL。 服务器没有
私人oracle，所以结果检查呼叫得到结构化的内容本身
agent停止后，回答检查查找温度
条件服务器返回。 使用模型配置文件，让 agent 也这样做
任务 3 次并打印 pass^k (付费推论)。 假设情况宣布不
重新设置 URL，这样连接器运行的警告，使试验共享服务器的状态；
此天气工具没有保存：
uv运行消费者-agent-测试工具-连接器-运行实例/连接器/一切-天气。 雅姆尔
--config agent.yaml -- 审判 3
检查还运行读取目录和文件的准备状态套件
运行 agent。 随着连接器 - 线路快速启动的固定仍在运行
审查其目录和对照钉住的Muse连接器显示的例子
准则：
uv运行消费者-agent-测试工具检查 http://127.0.0.1:8765/mcp - 一套缪斯指南？
- 明显的例子/连接/准备状态。
准则审查列出服务器的工具，并拨打无；-- drift- check
获取直播指南，只有在给定时才使用。 Poppy发现只读。
它的探测器只运行与 - 明显并采取行动在公司自己的主机：
开始会话，在已声明的测试账户中签名并确认一个已声明
操作。 设备套件读取一个本地文件。 合同套房检查a
目标在配置文件(-config)中命名。 mcp目标
未申报合同案例，其运营合同检查为
不支持和 t
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Consumer Agent Testing Tool (CATT) tests how an assistant uses your service. Run a scripted or model-driven agent against an MCP server, an MCP app, a website or an HTTP API. Check what the service did, inspect the evidence, and compare results across builds.
Muse and Instinct are supported surrogate emulation targets. Muse remains the default. Both use the shared connector testing contract, including routing, memory, payments, browser/HTTP tasks and scheduling. See the Instinct research for its sources and limits. The model that drives a trial is separate from the assistant being emulated.
Contributing | Security | Changelog | MIT license
A test host. CATT is a surrogate: it approximates an assistant's
host, not its private implementation. A passing test does not certify
compatibility with Muse or any hosted assistant. Each Muse-specific behavior
has a recorded basis: documented, unofficial, assumed or unknown.
This project is not affiliated with or endorsed by Meta, Instinct, OpenAI or Anthropic.
The connector lane records the basis for each Muse-specific behavior. Tool classes
and approval prompts follow documented guidelines. The muse-cli projection and
template client follow unofficial mirrors; the native projection is an assumed
generic baseline. The neutral cli projection supports both targets as a CATT
treatment. Instinct runs have a separate source record and label the
shared host algorithms as assumptions. Native/CLI MCP is CATT's test adapter;
no public Instinct MCP integration contract was found.
Run the local booking example under Instinct without a model or account:
uv run consumer-agent-testing-tool connector-run examples/connector/instinct-booking.yaml \
--driver scripted --script examples/connector/fixture-booking.calls.json \
--start "python -m consumer_agent_testing_tool.connector_fixture --port {port} --variant good --auth none"
Set emulation_target: instinct in your scenario or pass
--emulation-target instinct. Both targets use native or cli projection and the generic
MCP client. Instinct defaults to plain-text replies and does not run Muse guideline
checks. Its local wallet fixture covers payment behavior; live payment rails,
trusted-network protocol, Vault and Files remain outside the
shared contract.
Three quickstarts follow. The connector lane and the popular MCP server quickstarts run on loopback with no account; the Coverage Cat sandbox quickstart calls a hosted sandbox. Test your own service turns them into a workflow, and the documentation index lists every page.
You need Python 3.12 or newer (CI tests 3.12 on Linux and 3.14 on macOS),
uv, and Node.js 20 or newer with npm (the
installed packages declare >=20; tested with 25.9). The recorded release
evidence covers macOS. From a clone of this repository, install the Python and
Node dependencies and the Chromium build the browser lanes use:
uv sync --extra openrouter --extra dev
npm --prefix tooling ci --ignore-scripts
npm --prefix tooling/payment-ui ci --ignore-scripts
PLAYWRIGHT_BROWSERS_PATH="$PWD/.cache/ms-playwright" node tooling/payment-ui/node_modules/playwright/cli.js install chromium
The connector lane, the readiness checks and the scripted runs need only
uv sync. Node and Chromium are for the MCP Apps and browser lanes.
On Linux (Debian or Ubuntu), install Chromium with its system libraries instead. Linux is not covered by the recorded release evidence: the commands below match the CI workflow but have not been qualified here.
sudo apt-get update && sudo apt-get install -y nodejs npm # or Node 20+ from nodesource or nvm
curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync --extra openrouter --extra dev
npm --prefix tooling ci --ignore-scripts
npm --prefix tooling/payment-ui ci --ignore-scripts
PLAYWRIGHT_BROWSERS_PATH="$PWD/.cache/ms-playwright" \
node tooling/payment-ui/node_modules/playwright/cli.js install --with-deps chromium
Three features rely on macOS Seatbelt (sandbox-exec) and report unsupported
or blocked on Linux rather than running unconfined: the --offline readiness
mode, the network-denied validator workers behind run --surface api-tools, and
the local-model memory profile. The connector lane, the scripted runs, the
readiness checks and the Docker-backed stdio targets do not depend on it.
Check the setup with doctor:
uv run consumer-agent-testing-tool doctor
It checks Node against the floors the installed packages declare, the
tooling/payment-ui packages against their pinned versions, and Chromium and
its headless shell under .cache/ms-playwright. It reports the optional
protocol tooling, and checks Docker only for development lanes or configured
stdio MCP and CLI targets. It makes no model requests, even when an existing
configuration or --model-profile selects a model. Each failed or blocked check
prints one line on stderr with its reason and, for a setup problem, the command
that fixes it.
To explicitly run the provider preflight, which sends two short model requests
and may incur charges, add --model-preflight:
uv run consumer-agent-testing-tool doctor --config agent.yaml --model-preflight
Scripted runs need no model account, and the scripted commands in all three
quickstarts need no configuration file. Agent runs need a model profile and an
API key; Model providers shows how to create one. Supply
credentials through environment variables, and keep them out of scenarios and
shared evidence. consumer-agent-testing-tool.yaml, the default settings file,
is local and git-ignored, and no profile is shipped.
This quickstart stays on loopback and needs no account. Complete Setup, then start the Cedar Stays-like test service in a second terminal. It prints one JSON line with its URLs when it is ready; Ctrl-C stops it:
uv run python -m consumer_agent_testing_tool.connector_fixture --port 8765 --variant good --auth none
Replay the example's scripted calls, with no model:
uv run consumer-agent-testing-tool connector-run examples/connector/fixture-booking.yaml \
--driver scripted --script examples/connector/fixture-booking.calls.json
The script searches rooms, gets a quote, books and gives a final answer. The
host fills the guest's email from the scenario's private: values, which the
model never sees, and the scripted user answers each approval card from the
scenario's permissions: rules. After the run, the host checks the fixture's
private oracle (one confirmed booking, no duplicate charges) and the booking's
status through get_booking. It also checks that each sent call matched its
approval card (approval.dispatches_match, approval.disclosure_bound) and
that the final answer made no false claim and relayed the total and the
confirmation (answer.no_false_claims, answer.relay_complete).
The command prints the run directory and Connector report: .../report.html,
and exits 0 when every required check passed. Runs are sealed under
runs/connector-<id>/. Share the output of export, not the run directory:
its private/ folder holds host-private values.
For an agent run, create agent.yaml with the init command in
Model providers, set OPENROUTER_API_KEY in your shell, and run:
uv run consumer-agent-testing-tool connector-run examples/connector/fixture-booking.yaml --config agent.yaml
The model preflight runs first and prints its own evidence path. The agent then
sees the four allowed tools as native functions, with the host-private email
removed from their schemas, and writes the other arguments itself. This uses
paid model inference. To change the treatment without editing the scenario,
--projection cli offers the same tools through one exec command for either target,
--exposure search-only loads their parameters through tool_search,
--output-schema strict refuses results that break a tool's outputSchema as
the official MCP SDK does, and --trials N runs independent trials and prints
pass^k. The run lock records the projection, exposure, client-profile and
output-schema overrides under overrides.
connector-run exits 0 when every required check passed; 1 when a check
failed, the answer made a false claim or an agent limit was reached; 2 for a
setup, harness or reporting error; and 3 when the trial is blocked or
unsupported, for example by an uncertain write to reconcile, an unobservable
check or a client-profile limit. To start from your own server, run
scaffold; to replay a recorded, shareable agent trial without a model, turn
it into a calls file with scaffold --from-run. The
connector lane reference covers scenarios, flags and
evidence. The examples
list each shipped scenario with its fixture command, and
lane extensions covers routes, custom
connectors, the browser sub-agent, wallets, schedules and the other
extensions.
The connector lane works with any MCP server over Streamable HTTP, not only the
bundled test service. This quickstart points it at "Everything"
(@modelcontextprotocol/server-everything,
MIT), the reference server the MCP project maintains for client authors. Its
weather tool returns simulated data, so it needs no account or key. Start it in
a second terminal; the first run downloads the pinned release from npm:
PORT=3001 npx -y @modelcontextprotocol/server-everything@1.0.0 streamableHttp
See what the host infers from the live catalog. scaffold runs the handshake
and tools/list, calls no tool, and writes a scenario skeleton with suggested
tool classes from the server's annotations:
uv run consumer-agent-testing-tool scaffold http://127.0.0.1:3001/mcp --output my-everything.yaml
uv run consumer-agent-testing-tool check http://127.0.0.1:3001/mcp --suite muse-guidelines
The second command reviews the catalog statically against the pinned Muse connector guidelines. Then replay the shipped example, which asks for the weather in Chicago, first with native tools and then through the Muse-style command-line projection over the unofficial Muse template client:
uv run consumer-agent-testing-tool connector-run examples/connector/everything-weather.yaml \
--driver scripted --script examples/connector/everything-weather.calls.json
uv run consumer-agent-testing-tool connector-run examples/connector/everything-weather.yaml \
--projection muse-cli --client-profile muse-template-unofficial-2026-10 \
--driver scripted --script examples/connector/everything-weather-cli.calls.json
Both exit 0. The scenario offers the model three of the server's tools; a
reviewed allowlist keeps the rest off its list, including get-env, which
returns the server process's environment variables, and
gzip-file-as-resource, which fetches any URL it is given. The server has no
private oracle, so the outcome check calls get-structured-content itself
after the agent stops, and the answer checks look for the temperature and
conditions the server returned. With a model profile, let an agent do the same
task three times and print pass^k (paid inference). The scenario declares no
reset URL, so connector-run warns that the trials share the server's state;
this weather tool keeps none:
uv run consumer-agent-testing-tool connector-run examples/connector/everything-weather.yaml \
--config agent.yaml --trials 3
check also runs readiness suites that read catalogs and documents instead of
running an agent. With the connector-lane quickstart's fixture still running,
review its catalog and the example manifest against the pinned Muse connector
guidelines:
uv run consumer-agent-testing-tool check http://127.0.0.1:8765/mcp --suite muse-guidelines \
--manifest examples/connector/readiness.yaml
The guidelines review lists the server's tools and calls none; --drift-check
fetches the live guidelines, and only when given. Poppy discovery only reads.
Its probes run only with --manifest and act on the company's own hosts: they
start sessions, sign in the declared test account and confirm one declared
operation. The gadget suite reads one local file. The contract suite checks a
target named in the configuration file (--config). For a kind: mcp target
without declared contract_cases, its operation-contracts check is
unsupported and t
Auto-fetched and translated by AI Atlas. All rights belong to the original author.