花式吊带使编码agents更昂贵但并不更好，研究发现

花式吊带使编码agents更昂贵但并不更好，研究发现

制作者将agent新套装如Claude Code或OpenCode的得分收益进行推送。 一项研究发现声称不能成立。 撕破的绳索比两次运行相同的设置更能产生效果，但成本可以增加三倍。

一个编码的agent将一个语言模型配对起来，并带有一个牵引，系统即时，工具和上下文管理，使模型在一个代码寄存器内发挥作用。 新的绳索每天都可以使用，领导板可以自由地混合它们，但山东大学的两位研究者测量了当您保持模型恒定时，绳索究竟有多重要。 报纸标题给出了答案："一个吊带能买什么？大部分是纪念品".

之前的比较是测量噪音的

据作者说，前几次研究报告显示，两条绳子之间有20多点差距，但这些数字几乎总是单行。 同一设置的纯重运行在SWE-bench Verifed上已经存在最多6分的差异。 例如，Zhipu AI用2.4分的领先优势在Claude Code上广告自己的吊带，这个比值远在那个噪声范围内。 哈内斯也常以不同方式进入wired，使得比较更加不可靠。

刘和汉通过运出版本的三套吊带运行了5种型号：Claude Code,OpenCode，以及迷你-SWE-agent. 迷你-SWE-agent只给予该型号一个Bash工具并跳过上下文管理。 Claude Code和OpenCode提供专门的文件和搜索工具，长的系统提示，以及自动历史压缩。

该型号为Qwen3.6,Qwen3.8,Deepseek-V4-Flash,GLM-5.3-Flash，以及HY4-Preview. Claude Opus 5仅在Claude Code中作为参考运行。 测试使用了来自SWE-bench的45个最难完成的任务验证后又增加了447个集合。 几个设定两次跑来测量噪音。

所有运行发生离线。 在使用网络访问的早期测试中，71%的Deepseek和Qwen模型尝试从GitHub直接拉出正确的固定而不是自行解决任务。 Claude型号几乎从未这样做。 研究者放弃了这些结果，不宣称任何在测试数据方面受过训练的制造者，但模式足够强大，以至于他们切断了所有剩余运行的网络访问。

扭动的动作结果和重跑一样多

在更大的447项任务中，Claude Code和最小的SWE-agent，最重和最轻的吊带，为两个测试过的模型分别坐不到两点。 在五分波段内，它们在统计学上相当。

研究人员预计，45项最艰巨的任务将产生最大的利用效应。 相反，每个模型的调值会因2到5个已解任务而不同，一个简单的重播相同的设置会将计数转移最多3个。 有时迷你-SWE-agent领先，有时Claude Code的确领先。

翻转率也一样在两次重播之间，中位结果在13%的任务上有所变化。 在两种不同的绳索之间，同样是13%. 拆卸模型将它推向22%。 给定的任务从运行到运行都赢得变化，匹配硬币翻转。 吊带空隙的行为就像"同一噪音的新画"，作者写道。

这些小样本也限制了任何比较所能发现的东西。 拥有45项任务，甚至一个13点的缺口显示只提升了一半的时间。 有了447项任务，大约5个点的空隙就能够可靠地探测出来。 兹Q 1Z声称的2.4点优势都低于两个阈值，许多其他广告公布的改进也低于这两个阈值。

除了OpenCode落在后面之外，所有吊带都一样

唯一可测量的牵引效应是OpenCode持续不足。 在大型任务池上，它追随其他两个最多达9分。 大约一半的缺口可以追溯到输出极限。 当一个模型在推理过程中中步超过这个限制时，Claude Code会促使它继续前进。 OpenCode 让步骤在那里结束，尝试通常失败。

在 OpenCode 中添加 Claude Code 风格的续集没有帮助，因为相同的任务很少在重试中触发相同的溢出。 对于其余的空白，作者们没有解释，只有一种观察：根据OpenCode，模型会更早停止工作。 在本地运行的模型上，OpenCode的时间限制也比Claude Code要紧得多，作者将Claude Code标为不受控制的变量。

迷你-SWE-agent运行在上下文窗口中，因为它缺乏上下文管理，大部分是在其他调用也失败的任务上。 OpenCode的搜索工具试图下载第一个使用的二进制文件，离线时中断。

系统快捷和工具描述设置帐单

使用相同的模型，相同的任务，以及相同的价格清单，单项任务根据牵引力的不同，成本可高达三倍。 工具输出几乎无法解释差距。 这三个工具每次调用都返回同样数量的文本。

真正的区别来自系统即时和工具描述。 他们每一步都感到怨恨，为Claude Code跑出约16,600个代币，为OpenCode跑出约7,000个代币，为迷你SWE-agent跑出不到1,000个代币。 Claude Code还将模型的推理输出保留在对话史上，基于研究人员的逆向工程。

第一次通话之后，背景以类似的速度增长，遍及所有机体。 系统迅速大小设定了基线成本，步骤数将倍增。 哪个因素占主导地位取决于模型：使用GLM-5.3-Flash,Claude Code成本几乎是迷你SWE-agent的两倍。 有了Deepseek-V4-Flash，迷你SWE-agent需要这么多的额外步骤，以至于两个最终均匀。 缓存信物的低价抑制了差距，但不要改变排名。

没有主场优势，在思考中公开提问

Deepseek自己的拉带的得分并不比其他拉带的得分高，但成本最高。 Claude 电话 Opus 5甚至在研究人员从Claude Code移除工具后也提供了相同的结果。 一个年老的版本以不到一半的成本解决了类似数量的任务，因为当时的"扩展思考"并不是默认的。

思考是唯一一个利用选择改变结果的情况。 没有思考，带有Qwen3.6的迷你-SWE-agent降11分，而Claude Code和OpenCode则停留在噪音内。 HY4-Preview在Claude Code中也损失了11分而不思考。 agent多少取决于推理，似乎与特定的模型-harness组合联系在一起。 提交人承认他们的测试网太小，无法解释原因。

研究主要依靠一个基准测试。 同一型号的Bench 2.1 显示号航站楼的小型交叉检查。 闭源顶部型号Opus 5只运行于Claude Code. 作者们最后认为，收购是一种“减少任务损失的方法，以及一项法案而不是通过率”。 他们建议在每次拉带比较的同时发布一个相同的设置的重播只有这样，才能辨别出多大的差别只是噪音。 数据和分析脚本可在GitHub上找到。

在其他研究中，也很少作出更多的努力。 在一个中，一个简单的Markdown文件击败了一个复杂的技能系统，用于AI编码agents. 早些时候的一项研究已经表明，框架的选择可以使同一模式的费用几乎增加三倍。 因此，哈妮斯设计得到了大量的关注。 Nvidia最近通过更好的agent方向盘将agents编码的象征性费用削减了近一半。

AI News Without the Hype – 由人类治愈

订阅THE DECODER免费阅读，每周AI通讯，我们独家的"AI雷达"前沿报告每年6次，全面存档访问，访问我们的评论部分。

现在订阅

读取全照。

订阅无韵报道。

- 全面查阅THE DECODER的每篇文章

- 没有广告

- 加入评论和社区讨论

- 每周通过邮件转载AI新闻

- 6x/年："AI雷达"——对AI最重要的话题进行深度潜水。

- 每日AI新闻，总是最新的

- 我们完整的十年档案

- 由一个在AI工作10年以上的团队负责

订阅到 The Decoder

本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。

Fancy harnesses make coding agents more expensive but not better, study finds

Makers tout score gains from new agent harnesses like Claude Code or OpenCode. A study finds that claim doesn't hold up. Swapping harnesses barely moves results more than running the same setup twice, but costs can triple.

A coding agent pairs a language model with a harness, the system prompt, tools, and context management that make the model work inside a code repository. New harnesses ship every day, and leaderboards mix them freely, but two researchers at Shandong University measured how much the harness actually matters when you hold the model constant. The paper's title gives away the answer: "What does a harness buy? Mostly tokens."

Earlier comparisons were measuring noise

Previous studies report gaps of more than 20 points between harnesses, but those numbers almost always come from a single run, according to the authors. Pure reruns of the same setup already vary by up to six points on SWE-bench Verified. Zhipu AI, for example, advertises its own harness with a 2.4-point lead over Claude Code, a margin that falls well within that noise range. Harnesses are also often wired into the benchmark differently, making comparisons even less reliable.

Liu and Han ran five models through three harnesses in their shipped versions: Claude Code, OpenCode, and mini-SWE-agent. mini-SWE-agent gives the model only a Bash tool and skips context management. Claude Code and OpenCode offer specialized file and search tools, long system prompts, and automatic history compression.

The models were Qwen3.6, Qwen3.8, Deepseek-V4-Flash, GLM-5.3-Flash, and HY4-Preview. Claude Opus 5 ran as a reference only in Claude Code. Tests used the 45 hardest tasks from SWE-bench Verified plus a pool of 447 more. Several setups ran twice to measure noise.

All runs happened offline. In early tests with web access, 71 percent of Deepseek and Qwen model attempts pulled the correct fix straight from GitHub instead of solving the task themselves. Claude models almost never did this. The researchers discarded those results and don't claim any maker trained on the test data, but the pattern was strong enough that they cut off web access for all remaining runs.

Swapping harnesses moves results about as much as a rerun

On the larger pool of 447 tasks, Claude Code and mini-SWE-agent, the heaviest and lightest harnesses, sit less than two points apart for both tested models. Within a five-point band, they're statistically equivalent.

The researchers expected the biggest harness effect on the 45 hardest tasks. Instead, harnesses differ by two to five solved tasks per model, and a simple rerun of the same setup shifts the count by up to three. Sometimes mini-SWE-agent leads, sometimes Claude Code does.

The flip rates tell the same story. Between two reruns, the median outcome changes on 13 percent of tasks. Between two different harnesses, it's also 13 percent. Swapping models pushes that to 22 percent. Which tasks a given harness wins changes from run to run, matching a coin flip. The harness gap behaves like a "fresh draw of the same noise," the authors write.

These small samples also limit what any comparison can detect. With 45 tasks, even a 13-point gap shows up only half the time. With 447 tasks, gaps of about five points become reliably detectable. Zhipu's claimed 2.4-point advantage falls below both thresholds, and so do many other advertised improvements.

All harnesses perform the same, except OpenCode falls behind

The only measurable harness effect is a steady deficit for OpenCode. On the large task pool, it trails the other two by up to nine points. About half of that gap traces back to an output limit. When a model exceeds this limit mid-step during reasoning, Claude Code prompts it to keep going. OpenCode lets the step end there, and the attempt usually fails.

Adding a Claude Code-style continuation prompt to OpenCode didn't help, because the same task rarely triggers the same overflow on a retry. For the rest of the gap, the authors have no explanation, only an observation: under OpenCode, models stop working much sooner. On locally run models, OpenCode also had a much tighter time limit than Claude Code, which the authors flag as an uncontrolled variable.

mini-SWE-agent runs into the context window because it lacks context management, mostly on tasks where the other harnesses also fail. OpenCode's search tools try to download a binary file on first use and break when offline.

System prompt and tool descriptions set the bill

With the same model, same tasks, and same price list, a single task can cost up to three times more depending on the harness. Tool outputs barely explain the gap. All three harnesses return similar amounts of text per call.

The real difference comes from the system prompt and tool descriptions. They get resent with every step and run about 16,600 tokens for Claude Code, roughly 7,000 for OpenCode, and under 1,000 for mini-SWE-agent. Claude Code also keeps the model's reasoning output in the conversation history, based on the researchers' reverse engineering.

After that first call, context grows at similar rates across all harnesses. The system prompt size sets the baseline cost, and the number of steps multiplies it. Which factor dominates depends on the model: With GLM-5.3-Flash, Claude Code costs nearly twice as much as mini-SWE-agent. With Deepseek-V4-Flash, mini-SWE-agent needs so many extra steps that the two end up even. Lower prices for cached tokens dampen the gaps but don't change the ranking.

No home-field advantage, open questions around thinking

Deepseek's own harness scores no better with its own model than the other harnesses do, but costs the most. Claude Opus 5 delivered identical results even after the researchers removed tools from Claude Code. A year-old version solves a similar number of tasks at less than half the cost, because Extended Thinking wasn't on by default back then.

Thinking was the only case where the harness choice actually changed the outcome. Without thinking, mini-SWE-agent with Qwen3.6 drops eleven points, while Claude Code and OpenCode stay within noise. HY4-Preview also loses eleven points in Claude Code without thinking. How much an agent depends on reasoning seems tied to the specific model-harness combination. The authors admit their test grid is too small to explain why.

The study relies mainly on one benchmark. A small cross-check on Terminal-Bench 2.1 shows the same pattern. The closed-source top model, Opus 5, ran only in Claude Code. A harness buys " fewer ways to lose a task, and a bill, not pass rate," the authors conclude. They recommend publishing a rerun of the same setup alongside every harness comparison. That's the only way to tell how much of a measured difference is just noise. Data and analysis scripts are available on GitHub.

More effort in the harness rarely pays off in other studies, either. In one, a simple Markdown file beat a complex skill system for AI coding agents. An earlier study already showed that the choice of framework can nearly triple costs for the same model. Harness design gets a lot of attention as a result. Nvidia recently cut nearly half its token costs for coding agents through better agent steering.

AI News Without the Hype – Curated by Humans

Subscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive "AI Radar" frontier report six times a year, full archive access, and access to our comment section.

Subscribe now

Read on for the full picture.

Subscribe for hype-free coverage.

- Full access to every article on THE DECODER

- No ads

- Join the comments and community discussions

- A weekly AI news recap via mail

- 6x/year: "AI Radar" — deep dives on the AI topics that matter most

- Daily AI news, always up to date

- Our full ten-year archive

- Covered by a team with 10+ years in AI

Subscribe to The Decoder

Auto-fetched and translated by AI Atlas. All rights belong to the original author.