CUGA 策略游戏场
两个活的CUGA agents，有和没有政策
企业文章发表
2026年11月3日 (英语).
有近200个agent吊带可供选择。 我们给了其中3个相同的开放量级模型和90个AppWorld任务，将牵引装置的完成率从30%提高到81%. 对于企业来说，CUGA 新增这个基准测试不测试的是什么：规则，在自己的边界内，绳子执行和部署。
社区列表如：真棒-agent-harness 和最佳-Agent-Harneses 追踪其中近200个，它们明确提出一个观点：牵引关系，而不仅仅是模型。 在最佳Agent-Harneses的一次分析中，同一GLM-5.2型号在SWE-bench Pro上从23%升至52%，当时只改变了吊带。
因此，正确的问题不是“哪一种方法是最好的？”而是“哪一种方法适合我的工作量，用我的模型？” 来自Nous Research的赫尔墨斯为常在的个人助手建造，来自LangChain的Deep Agents为研究等漫长，多步骤的任务建造。 两者都非常出色
企业工作是一种不同的工作。 Agents在CRM和支付等业务系统上采取行动，它们必须遵循持有任何模型决定的规则。 CUGA(英语：Configuble Generalist Agent)是IBM为这项工作而建造的开源吊带，在您自己的环境，包括主权云和空载部署中，具有开放重量或托管模型。 为了测量新增系系系系系系系系系系系系系系系系系系系系系系系系系系系系系系系系系系系系系系系系系系系系系系系系系系系系系系系系系系系系
我们在AppWorld上运行了CUGA,Hermes，以及Deep Agents，一个跨9个应用程序和457个API的多步任务基准测试，例如电子邮件，支付，和文件存储。 应用是消费者应用，因此AppWorld是一个代用：它测试企业自动化同样需要的子集能力，如登录，阅读API docs，通过结果呼唤，以及跨越多个应用的变化状态。 每个牵引装置都使用相同的开放量级模型GLM-5.3-flash，在30种情景下执行相同的90项任务，每个任务一次尝试。 所有跑步都使用"港湾评估框架"，我们的设置在"港湾","港湾叉"公开。
图1. AppWorld,30种情景中的90项任务，GLM-5.3-flash for every hape. TGC：任务通过。 SGC：三种变体都经过的情景。
如何诵此经。 这不是对赫尔墨斯或深Agents的判决。 也没有为AppWorld这样的多应用API工作而建，这就是重点：牵引必须适合工作。
最有用的数字是SGC. 一个情景只有在它的三个变体都通过时才会通过，所以SGC奖励的是一致性，而不是幸运的运行。 这里的差距更大：CUGA有20种情景，赫尔墨斯有8种，深Agents有2种。
适用三限。 这是一个基准测试上的90任务样本，因此结果可能无法概括到其他任务。 设置并不相同(见上文设置表)。 跑步没有使用政策，所以不会衡量政策的执行。
在90任务比较中，模式在每次运行中都是相同的，因此差异来自牵引。 每个CUGA agent运行相同的紧凑循环：准备，调用模型，执行，和重复。 同样的循环尺度从一个拥有自己工具的CugaAgent到一个CugaSupersor，协调多个agents，局部或远程的A2A. 这些CUGA功能在AppWorld设置中运行
规划工作
任务分解：将任务分成步骤，一个应用程序
重点工具：每个步骤都看到其应用工具以及几个共享工具，而不是全部469
工具短选器：寻找其它工具的搜索步骤
代码
代码法案：模型写作调用应用API的 Python
Sandbox：代码运行在受限环境中，有时间限制
动作前检查
执行前的校验：对运行前更改数据的每个代码块进行校验
动作后检查
自动反射：审查结果并让 agent 修正其计划
Pagination 审计：标记一个列表调用，当没有请求下一页时返回完整页
有了这些特点，这个模型做出的决定较少，在达到结果之前，这种利用抓住了更多的错误。
与上述清单一样，绳索控制的每一个步骤也是检查规则的地方。
大多数团队都写业务规则，如支付批准或将客户PII留在组织内部，进入系统迅速。 提示是指示，而不是执行。 该模型可以错误地读取，一个用户可以将agent从中解说出来，一个注入的信息可以在任务中间改变。
CUGA政策将这些规则从即时移出并进入了控制。 我们的ACM CAIS 2026 号文件《建设工程治理Agents》(arXiv)中描述了这一政策层。 这些政策在每个阶段有四个点：
指导引导模式，但审批和工具守护由运行时间执行。 将支付批准和PII处理等规则放入第二组，无论模型决定什么，他们都持有。
你可以看到自己在政策游乐场中的区别，它向两个活的agents，一个没有政策，一个与他们：
图2. 对于"支付我欠下的一切"，无保单的agent支付了90美元，包括一笔拒付的收款债权。 agent有保单持有付款请求批准，然后只支付了到期的10美元。
选择一个像模型一样小心的带子。 在我们的比较中，同一型号完成了30%到81%的AppWorld任务，这取决于吊带。 对于企业工作，CUGA给你：
——成绩突出，现在有开放量级模型。 CUGA凭借主机型号，在WebArena(2025年2月至9月)和AppWorld(2025年7月至2026年2月)举办了#1赛事。
- 规则运行时间执行。 批准和工具卫士停止调用无论是模型决定。
- 部署在你们的主权边界内 CUGA是Apache 2.0，与任何OpenAI兼容的端点合作，并在本地运行，在您的私人或主权云中运行Kubernetes，甚至在一个空中加载的主权集群中运行，OIDC的登录和单独的聊天和管理角色。 IBM主权核心以CUGA为基础，完全在客户边界内运行agents.
- 可计量成本。 预算限制开支，运行收据显示符号，工具呼叫，以及每次运行的时间。
- 配置，不是代码。 在UI或SDK中设置工具，预算和政策，测试一个草稿，并发布一个版本。
CUGA并不是每个工作的正确工具：
- 对于一个终端编码agent，使用opencode,Codex，或Claude Code.
- 对于一个个人的，总是在你的聊天应用程序中生活的助手，使用OpenClaw或Hermes.
- 对于空白-canvas图框，直接使用LangGraph. CUGA就是建立在它上。
CUGA是用于将跨API,apps，和Web UI的商业工作自动化，按照您可以执行的规则。
· 政策指导
- 重复比较
- 星CUGA在GitHub
- 试试CUGA：快速安装
CUGA团队运行了这一对比，CUGA使用了AppWorld配置，而其他机身并没有使用。 如果您可以配置赫尔墨斯或深 Agents 来做得更好，请发送配置到 hort-eval。 我们将重复执行同样的任务，公布结果，无论谁赢得胜利。
🛡
4
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
CUGA Policy Playground
Two live CUGA agents, with and without policies
Enterprise Article Published
November 3, 2026
There are nearly 200 agent harnesses to choose from. We gave three of them the same open-weight model and 90 AppWorld tasks, and changing the harness moved task completion from 30% to 81%. For enterprise use, CUGA adds what this benchmark does not test: rules that the harness enforces and deployment inside your own boundary.
Community lists such as awesome-agent-harness and best-of-Agent-Harnesses track nearly 200 of them, and they make one point clearly: the harness matters, not only the model. In one analysis that best-of-Agent-Harnesses cites, the same GLM-5.2 model went from 23% to 52% on SWE-bench Pro when only the harness changed.
So the right question is not "which harness is best?" but "which harness fits my workload, with my model?" Hermes from Nous Research is built for always-on personal assistants, and Deep Agents from LangChain for long, multi-step tasks such as research. Both are excellent at those jobs.
Enterprise work is a different job. Agents act on business systems, such as CRM and payments, and they must follow rules that hold whatever the model decides. CUGA (ConfigUrable Generalist Agent) is IBM's open-source harness built for that job, with open-weight or hosted models, inside your own environment, including sovereign clouds and air-gapped deployments. To measure what the harness adds, we kept the model the same and changed only the harness.
We ran CUGA, Hermes, and Deep Agents on AppWorld, a benchmark of multi-step tasks across 9 apps and 457 APIs, such as email, payments, and file storage. The apps are consumer apps, so AppWorld is a proxy: it tests a subset of the capabilities that enterprise automation also needs, such as logging in, reading API docs, paging through results, and changing state across several apps. Every harness used the same open-weight model, GLM-5.3-flash, on the same 90 tasks in 30 scenarios, with one attempt per task. All runs used the Harbor evaluation framework, and our setup is public in harbor-eval, our Harbor fork.
Figure 1. AppWorld, 90 tasks in 30 scenarios, GLM-5.3-flash for every harness. TGC: tasks passed. SGC: scenarios where all three variants passed.
How to read this. This is not a verdict on Hermes or Deep Agents. Neither was built for multi-app API work like AppWorld, and that is the point: the harness has to fit the job.
The most useful number is SGC. A scenario passes only when all three of its variants pass, so SGC rewards consistency, not a lucky run. Here the gap is larger: 20 scenarios for CUGA, 8 for Hermes, and 2 for Deep Agents.
Three limits apply. This is a 90-task sample on one benchmark, so the results may not generalize to other tasks. The setups were not identical (see the setup table above). And the runs did not use policies, so they do not measure policy enforcement.
In the 90-task comparison, the model was the same in every run, so the differences come from the harness. Every CUGA agent runs the same compact loop: prepare, call the model, execute, and repeat. The same loop scales from one CugaAgent with its own tools to a CugaSupervisor coordinating several agents, local or remote over A2A. These are the CUGA features that ran in the AppWorld setup, grouped by what they do:
🧭 Plan the work
Task decomposition: splits a task into steps, one app at a time
Focused tools: each step sees the tools of its app and a few shared ones, not all 469
Tool shortlister: find_tools searches for any other tool a step needs
🐍 Act in code
CodeAct: the model writes Python that calls the app APIs
Sandbox: the code runs in a restricted environment, with a time limit
🔍 Check before acting
Verification before execution: a check reviews each code block that changes data before it runs
🔁 Check after acting
Automatic reflection: reviews the results and lets the agent correct its plan
Pagination audit: flags a list call that returned a full page when the next page was never requested
With these features, the model makes fewer decisions, and the harness catches more mistakes before they reach the result.
Each step that the harness controls, like those in the list above, is also a place where it can check a rule.
Most teams write business rules, such as payment approval or keeping customer PII inside the organization, into the system prompt. A prompt is an instruction, not enforcement. The model can misread it, a user can talk the agent out of it, and an injected message can change it in the middle of a task.
CUGA policies move these rules out of the prompt and into the harness. We describe this policy layer in our ACM CAIS 2026 paper, Governance by Construction for Generalist Agents (arXiv). The policies act at four points in every run:
Guidance steers the model, but approvals and ToolGuards are enforced by the runtime. Put rules such as payment approval and PII handling in the second group, where they hold whatever the model decides.
You can see the difference yourself in the Policy Playground, which sends the same message to two live agents, one without policies and one with them:
Figure 2. For "Pay everything I owe.", the agent without policies paid $90, including a denied claim in collections. The agent with policies held the payment for approval, then paid only the $10 that was due.
Choose a harness as carefully as the model. In our comparison, the same model completed between 30% and 81% of AppWorld tasks, depending on the harness. For enterprise work, CUGA gives you:
- Strong results, now with open-weight models. With hosted models, CUGA held #1 on WebArena (February to September 2025) and on AppWorld (July 2025 to February 2026).
- Rules that the runtime enforces. Approvals and tool guards stop a call in the harness, whatever the model decides.
- Deployment inside your sovereign boundary. CUGA is Apache 2.0, works with any OpenAI-compatible endpoint, and runs locally, on Kubernetes in your private or sovereign cloud, or even in an air-gapped sovereign cluster, with OIDC sign-in and separate chat and management roles. IBM Sovereign Core builds on CUGA to run agents fully inside the customer's boundary.
- Measurable cost. Budgets limit spending, and run receipts show tokens, tool calls, and time for every run.
- Configuration, not code. Set tools, budgets, and policies in the UI or SDK, test a draft, and publish a version.
CUGA is not the right harness for every job:
- For a terminal coding agent, use opencode, Codex, or Claude Code.
- For a personal, always-on assistant that lives in your chat apps, use OpenClaw or Hermes.
- For a blank-canvas graph framework, use LangGraph directly. CUGA is built on it.
CUGA is for automating business work across APIs, apps, and web UIs, under rules you can enforce.
- 🛡️ Policy Playground · policies guide
- 🧪 Reproduce the comparison
- ⭐ Star CUGA on GitHub
- 🚀 Try CUGA: quick install
The CUGA team ran this comparison, and CUGA used an AppWorld configuration that the other harnesses did not. If you can configure Hermes or Deep Agents to do better, send a configuration to harbor-eval. We will rerun it on the same tasks and publish the results, whichever harness wins.
🛡
4
Auto-fetched and translated by AI Atlas. All rights belong to the original author.