新智元报道
现在的AI Agent已经越来越多地被组队部署到写代码、云运维、财务对账和企业审批等真实工作流中。
但现有的Computer-Use基准（如OSWorld、WebArena）几乎都默认了一个理想前提：环境信息永远一致，所有队友Agent都百分之百靠谱。
真实场景显然不是这样。在多Agent协作中，子Agent经常会因为读了过期缓存而汇报错误状态，或者在文件锁和权限上跟主Agent撞车。
比如：子Agent发消息说「线上服务一切正常，请立刻切流」，但实时监控面板的截图里其实已经在疯狂报错。这时候，主Agent如果盲信队友的文字汇报，就会直接酿成线上事故；它必须主动打开监控截图核实真实状态（Ground Truth），并动手回滚配置、释放共享锁。
来自Google、Emory、USC、UF、UChicago、UCSB和Columbia的研究团队联合发布了CLASHarness，首次对 15 个主流大模型在多智能体冲突下的真实表现进行了大考。
论文链接：https://arxiv.org/html/2610.19128
开源代码与数据集：https://github.com/google-research/CLASHarness
覆盖20个真实领域、600个多轮可执行冲突任务、7,970张4K视觉证据图片、3,284项必测检查点。
要构建这样一个包含代码库、终端日志、多Agent聊天记录和高清UI截图的冲突测试集，难点在于：纯靠人工编写根本没法大规模扩展；而如果直接让LLM一次性生成，又极易出现前后剧情矛盾、题面不小心泄题、甚至生成连标准答案都过不了的「死题」。
为此，CLASHarness设计了一套「三层八阶段」自动化生成与验证管线，从源头上保证100%的任务可解性：
图1：CLASHarness的三层八阶段生成管线。绿框代表零成本的代码自动修复，红框代表带报错反馈的局部单步重试。
1. 分步规划+局部自动纠错（图1）
管线先规划全局大纲和Agent角色权限，再按每3–4轮滚动生成具体任务，避免长文本上下文跑偏。每轮生成细分为8个小步骤：一旦出现路径或格式问题，直接用代码自动修复（0次模型调用）；出现逻辑冲突则只重试出错的那一小步，无需推倒重来，平均仅需95次API调用就能合成一个完整的10轮连续任务包。
图2：CLASHarness生成的4K视觉证据示例（涵盖监控面板、IAM权限对账表、算力调度监控与报销账单明细）。
2. 杜绝文本「走捷径」，关键证据只藏在4K图里（图2）
为了防止Agent不看图、光靠搜文本混过关，CLASHarness会先定好解决冲突的关键真值（比如真实账单金额、权限矩阵状态），再渲染出3840×2160（4K）分辨率的真实UI截图或实景照片，并用自动扫描器清空所有文本文件里的答案线索，逼着Agent必须看图取证。
3. 先在沙盒跑通标准答案，再生成评分器
所有任务的评分标准（34.9%精确代码检查+65.1%语义三票多数决检查）全部直接从真实沙盒里跑通的参考轨迹中提取生成。同时内置23个模块化冲突插件，开发者可以像搭积木一样按需合成新的冲突测试场景。
研究团队在600个冲突任务上评测了15个主流模型（包括Claude Opus5 / 4.8、Sonnet 5、GPT-6 Astra、GPT-5.6 系列、Gemini 3.x 系列，以及开源模型Kimi K3、DeepSeek V4 Pro / Flash、GLM-5.2），累计跑完了9,000+条真实沙盒轨迹。
图3：15个模型在CLASHarness（600个任务）上的总排行榜。左侧彩条为端到端任务通过率（Pass Rate），右侧为单项检查平均得分（Mean Score）。
发现1：单点得分不低（~60%），但端到端通过率无一突破30%
如图3所示，在标准20步限制下，Claude Opus 5（25.2%）和GPT-5.6 Sol（24.5%）排名最高；放宽步数后，GPT-6 Astra拿到最高的28.7%。开源阵营中，Kimi K3（21.8%，总榜第 5）和DeepSeek V4 Pro（19.8%）表现非常亮眼，追平甚至反超了多款闭源商用模型。
为什么顶尖模型能拿到55%–67%的单项步骤分（Mean Score），端到端通过率（Pass Rate）却连30%都不到？因为解决真实冲突是一条完整的链条：识破队友误导→打开图片核对真值→修改系统配置→完成交接收尾。只要中间漏掉一环，整个任务就会失败。
发现2：弱模型盲信队友文字，强模型主动看图查证
当子Agent的文字汇报和真实环境冲突时，较弱的模型很容易「耳根子软」，直接轻信错误的文字汇报；而强模型会主动调用工具去翻看监控截图或账单图片来核实真值（在视觉定位和证据引用上领先弱模型31–65个百分点）。
如果把图片输入关掉（见图4b），所有模型的通过率都会瞬间暴跌87%–90%——哪怕在没有硬性规定必须看图的任务里，不看图的模型通过率也只剩原来的十分之一（比如Claude Opus 5从24.8%跌到2.8%）。
发现3：「看出矛盾」容易，「动手修好并收尾」太难
为了搞清楚模型到底是「看不出问题」还是「修不好问题」，研究团队对比了同一批冲突场景下的两种任务要求：
只要求调查汇报：模型的平均通过率能达到28.1%；•要求动手修改系统状态：通过率直接掉到11.0%（净降18.2个百分点）；如果要求走完完整的多Agent协作流程，通过率更是只剩4.4%。
最容易栽跟头的「收尾环节」：即便是最强的GPT-6 Astra和Claude Opus 5，也有58%–72%的收尾动作失败。
模型经常在改完主配置文件后就直接结束任务，忘了释放共享锁、忘了关工单，或者忘了发消息通知下游子Agent接手。
发现4：思考Token越多反而越容易死循环，放开步数才能解锁深度探索
在数学和代码题里，让模型多思考（增加Thinking Tokens）通常能提分。但在多Agent交互环境里，结果却恰恰相反：
图4：三大消融实验。（a）放宽步数限制（20→30→40步）；（b）有视觉vs.无视觉输入；（c）换用不同生成模型（Gemini 3.6 Flash vs. Claude Sonnet 5）的排名一致性（ρ=0.96）。
1. 过度思考导致的「死循环陷阱」
在20步限制下，模型花在内部思考上的Token比例越高，成功率反而越低（相关性ρ=-0.70）。例如把Gemini 3.5 Flash的思考强度从Low调到High，它的通过率反而从13.5%暴跌到3.0%，单题解决成本飙升到$39.54（是GPT-5.6 Luna $0.72的55倍）。原因在于它在内部反复纠结、重复调工具，89.2%的任务都把20步额度耗光了还没做完正事。
2. 放开步数限制，解锁长线探索（图4a）
反过来，对于真正擅长多步探索的模型，20步上限反而限制了它的发挥。当把步数上限从20步放宽到40步和60步时，Gemini 3.8 Flash的通过率从12.8%一路飙升到29.2%和30.7%，反超所有20步基准模型，同时平均每道解出题目的花费还下降了21%（降至$3.50）。
除了作为评测基准，CLASHarness更是一套可扩展的多智能体训练数据合成引擎。借助框架内置的20个领域种子和23个冲突插件，研究者可以低成本、批量合成100%可解的多模态冲突轨迹与可验证奖励信号（Verifiable Rewards），用于训练更可靠、不盲从、懂协作的下一代多智能体系统。
目前，CLASHarness的生成管线代码、600个评测任务及交互式轨迹查看器已在GitHub开源。
参考资料：
https://arxiv.org/html/2610.19128
编辑：LRST
不喜欢
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
New Ji-won.
Now AI Agent is increasingly being deployed by teams in real workflow such as code writing, clouding, financial reconciliation and enterprise approval.
But the existing Computer-Use benchmarks (e.g. OSWorld, WebArena) almost all acquiesce to the ideal premise that environmental information is always the same and that all teammates Agent are 100 per cent reliable.
The real scene is clearly not. In the multiAgent collaboration, son Agent often reports an error by reading the expired cache or crashes with the main Agent on the document lock and permission.
For example: SubAgent has sent a message that "all services on line are normal, please cut off immediately" but the real-time surveillance panel screen is already making a crazy mistake. At this point, Lord Agent would have caused a direct online accident if his blind teammates had reported; it would have to proactively open a surveillance screenshot to verify the true state (Ground Truth) and start rolling back and release the shared locks.
The research team from Google, Emory, USC, UF, Uchicago, UCSB and Columbia, in conjunction with publish, carried out a study of the true performance of 15 mainstream LLM in the multi-AI agent conflict for the first time.
Paper link: https://arxiv.org/html/2610.19128
Open source code and data set: https://github.com/google-research/CLASHarness
Covered 20 real areas, 600 rounds of conflictable tasks, 7,970 images of 4K visual evidence and 3,284 mandatory checkpoints.
The difficulty in constructing such a series of conflict tests, including code libraries, terminal logs, multiAgent chat records and high-resolution UI intercepts, is that it is impossible to expand on a large scale by simply hand, and that, if LLM is created directly, it is highly susceptible to back-to-back and back-to-back dramas, issues inadvertently leaked, and even “dead” questions that cannot be met with even standard answers.
To this end, CLASHarness designed a three-storey eight-stage automated generation and validation pipeline to ensure 100 per cent mission solvency from the source:
Figure 1: CLASHarness three-story eight-story line generation. The green box represents zero-cost automatic repair of the code and the red box represents a local one-step retry with incorrect feedback.
1. Step plan + local automatic correction (figure 1)
Pipelines first plan the global outline and Agent role permission, then generate specific tasks per 3-4 wheel, avoiding context deviations from long text. Each round generation is broken down into eight small steps: when a path or format problem arises, it is automatically repaired by a direct code (0 calls from the model); and when a logical conflict occurs, only the small step of retry is wrong, without the need to push over, and an average of 95 API calls is required to synthesize a full 10-round successive mission package.
Figure 2: Examples of 4K visual evidence generated by CLASHarness (covering surveillance panels, IAMpermission reconciliations, algorithm control and detailed statement of reimbursement).
2. Putting an end to the text "facility", with key evidence hidden in 4K (figure 2)
In order to prevent Agent from viewing and simply searching for cleared, CLASHarness would have determined the key real values of conflict resolution (e.g., true billing amounts, permission matrix status), replaying 3840 x 2160 (4K) resolution real Ui screens or reality photographs, and emptied all text files with automatic scanners, forcing Agent to see the evidence.
3. Run the standard answers in the sandbox and generate the scorers
The rating criteria for all tasks (34.9% accurate code check + 65.1% semantic majority check) are all derived directly from the reference tracks running through the real sandbox. At the same time, 23 modular conflict plugins are built in which developers can synthesize new conflict test scenarios as needed, like building blocks.
The research team assessed 15 mainstream models (including Claude Opus5/ 4.8, Sonnet 5, GPT-6 Astra, GPT-5.6, Gemini 3.x, and open source models Kimi K3, DeepSeek V4 Pro/Flash, GLM-5.2) and managed to aggregate 9,000+ real sandbox tracks.
Figure 3: CLASHarness (600 missions) total ranking of 15 models. Left-to-end task pass rate (Pass Rate) and right-to-right average check score (Mean Score).
One single score was found (~60%), but no end-to-end pass rate exceeded 30%
Claude Opus 5 (25.2 per cent) and GPT-5.6 Sol (24.5 per cent) ranked highest under the standard 20-step limit, and GPT-6 Astra received the highest 28.7 per cent after the easing step. In the open source camp, Kimi K3 (21.8 per cent, Master Book 5) and DeepSeek V4 Pro (19.8 per cent) performed very brightly, pacing and even exceeding multiple closed source commercial models.
Why does the top model get 55%-67% single step fractions, while the end-to-end pass rate is less than 30%? Because the solution to the real conflict is a complete chain: the teammates are misleadingly opening up the image to verify the true value of the modified system configuration to finish the receipt. As long as there is a missing link, the entire mission will fail.
Discovered 2: Weak Model Blind Teammate Text, Strong Model Active Check
When Agent's textual reports and real environment conflicts, weaker models are easy to "soft ears" and directly trust the wrong text; and stronger models are proactive in calling tools to look through surveillance intercepts or billing pictures to verify true values (31-65 percentage points ahead of weak models in visual positioning and evidence references).
If the images are turned off (see figure 4b), all models will experience an instant drop of 87 per cent – 90 per cent – and even in tasks that do not have mandatory graphics, the non-graphic model pass rate is one tenth (e.g. Claude Opus 5 falls from 24.8 per cent to 2.8 per cent).
It's easy to find 3 "see the contradiction" and it's too hard to "do it and finish it."
In order to determine whether the model is a "no problem" or a "no problem" problem, the team compared the two mission requirements in the same conflict scenario:
Only survey reporting is required: the average pass rate of the model is 28.1 per cent; The pass rate fell directly to 11.0 per cent (a net decrease of 18.2 percentage points); if a complete multi-Agent collaborative process is required, the pass rate is only 4.4 per cent.
One of the most easy-to-penetrate "finishing" lines: even the strongest GPT-6 Astra and Claude Opus 5, 58 per cent-72 per cent of the closings failed.
The model often ends directly after the master profile has been changed, forgetting to release the shared lock, to turn off the worksheet, or to send a message to Agent downstream to take over.
Finds that the more Token is thought about, the easier it's going to be, the easier it's going to be to unlock the depths.
In mathematics and code, it's usually possible to make models think more (adding Thinking Tokens). But in the multi-Agent interactive environment, the result is the opposite:
Figure 4: Three digestive experiments. (a) Relaxation of step limits (20:30:40 paces); (b) visual vs. no visual input; (c) consistent ranking (old = 0.96), in lieu of a different generation model (Gemini 3.6 Flash vs. Claude Sonet 5).
1. "The Death Cycle Trap" due to overthinking
Under the 20-step limit, the higher the Token ratio of the model spent on internal thinking, the lower the success rate (old correlation = -0.70). For example, when Gemini 3.5 Flash's thinking strength was shifted from Law to High, its pass rate dropped from 13.5% to 3.0%, and the single-issue solution to the cost spike. Up to $39.54 (55 times the GPT-5.6 Luna $ 0.72). This is due to the fact that it is in-house and repetitive tools, and 89.2 per cent of the missions have exhausted 20 steps of quota.
Release step limits and unlock long line exploration (figure 4a)
In turn, the 20-step cap limits a model that is really good at multi-step exploration. When the maximum step was eased from 20 to 40 and 60 steps, Gemini 3.8 Flash went from 12.8 per cent to 29.2 per cent and 30.7 per cent, exceeding all 20-step benchmark models, while the average cost of deciphering the subject fell by 21 per cent (to $3.50).
In addition to being the benchmark for evaluation, CLASHarness is an extended set of multi-AI agent training data synthesis engines. Using 20 field seeds and 23 conflict plugins installed within the framework, researchers can use low-cost, mass synthesis of 100 per cent solventable multi-modular conflict trajectories and verifiable reward signals to train the next generation of multi-AI agent systems that are more reliable, unblind and collaborative.
Currently, the CLASHarness generate pipe code, 600 assessment missions and interactive track viewer are at GitHub open source.
References:
https://arxiv.org/html/2610.19128
Edit: LRST
I don't like it.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.