前一天我让一个能自己操作浏览器的 Agent 去跑掘金的每日任务，本来只是想看看它对真实站点的适配能力，结果掉进了工程细节里：真正难的不是让模型「知道该点什么」，而是让整个循环在真实前端面前不出错。这篇文章不谈模型榜单，只谈我在这条链路上遇到的几个工程真问题，以及背后各家的取舍。
一、四条路线，同一个假设
目前主流的浏览器 Agent 大致分成四类：
- 视觉为主：Anthropic 的 computer use 属于这一类，给模型截图，输出鼠标坐标与键盘动作。优点是泛化——任何界面都能试；代价是一屏 1080p 截图动辄上千 token，坐标回归差几十像素就点空。
- 模型内置动作空间：OpenAI 的 Operator / CUA 类方案，把「浏览与点击」当成模型的原生动作，卖点是不需要站点配合。
- 多标签与人在环：Google 的 Project Mariner 取向，强调并行标签页与关键动作的人工确认，把不确定性包到交互设计里。
- 结构化 DOM / 可访问性树：国内 IDE 类工具（包括字节 Trae 的 browser 能力）与大量 MCP 服务的选择：把页面上可交互的元素规整成带 ref 的列表，模型只需要说「点 e42」。
四条路线共同的假设是：模型负责感知与决策，剩下的交给工程兜底。这个假设在读的场景里基本成立，在写的场景里会碎。
二、第一个真问题：上下文不是「塞进去」，是「编排」
一屏信息流页面原样吐进模型，几十万 token 不夸张。所以快照必须做可见性过滤、文本截断、重复节点合并。这一步本质上是在赌：模型不需要看到全部也能做对。
但真正影响成本与稳定性的是顺序能不能稳定。现在各家都在用 prompt caching / KV cache 降价，一个 Agent 循环里最理想的结构是：系统提示 + 工具定义 + 历史动作 保持字节级不变，只有末尾的快照在变。一旦快照里元素顺序、id 生成方式或者属性名拖了缓存的后腿，命中率一路下降，同一个任务的成本能差好几倍。
另一个坑是长任务的记忆。会话被自动压缩一次，如果交接摘要里没写清「哪几项已经做完、分别做了几次、有什么证据」，下一轮就会重复劳动。我后来固定四段式：已完成（带数字与证据）/ 未完成 / 明确禁令 / 下一步顺序。漏在摘要里的约束，等于从来没写过。
三、第二个真问题：写操作才是可靠性的分水岭
读页面的时候，Agent 几乎不会错；一碰到写操作，问题全出来。
举一个真实的例子：社区里的「点赞」按钮本质上是一枚 svg 图标，可访问性树上既没文本也没独立节点，工具只能拿到父容器再靠偏移坐标去猜。猜偏几像素，工具层返回 success，模型记一笔「已点赞」，而后端一个请求都没收到。
真正的验证只有两条：看网络请求里 interact_api/v1/digg/save 有没有出现；看 DOM 文案从「点赞」变成「已赞」。后者成本极低，因为 UI 的状态回写就是业务成功最好的断言。
所以我的写法是：工具契约里强制要求 postcondition。没有断言的写操作，不准它自己宣布完成。同理，重试逻辑要按幂等设计——一次点不中就停，宁可任务失败，也不要「点赞 + 取消」相互抵消后白刷一个风控标记。
四、第三个真问题：风控看的是行为特征
社区类反作弊现在很少只靠验证码下结论，而是看点击分布、停留时长、滚动曲线像不像人。两次零位移的连续点击，在风控眼里比一个慢慢滚动的脚本危险得多。
这一层对技术选型的影响比想象大：如果产品把 Agent 当功能，就必须同时定义允许的自动化频率上限；否则做的不是一个功能，而是一个自动刷号的入口。未来平台与 Agent 的约定，很可能不是靠协议，而是靠谁为行为负责。
五、一点思考：Agent 不会替代前端，但会重新定义好前端
当读页面这件事交给模型，语义化 HTML 就从无障碍合规变成了 AI 可用性接口。一个带 aria-label 的按钮，在 Agent 眼里的价值比三个装饰性 div 高。反过来说，那些靠 canvas 全量绘制、没有文本快照可用的界面，在 Agent 时代会自己变成信息孤岛。
至于验证应该放在模型还是 harness，我的判断很明确：模型负责意图，harness 负责可证实的状态迁移。这也解释了为什么同一个模型在不同 Agent 产品里表现差那么多——差异不在模型层，在工具层。
六、给准备上手的人
- 先区分读与写：读可以尽量自动化，写必须自带断言。
- 快照要可缓存：元素编号、顺序、属性集尽量稳定。
- 失败即停，按幂等设计重试，不要补刀。
- 长任务一定要写交接摘要，只留结论不留证据的摘要会直接导致重复劳动。
- 把需要信任的动作（支付、安全确认、公开发言）显式交回给人。
模型能力会一直卷下去，但能不能进真实工作流，取决于你能否兜住它的不确定。
（本文基于一次真实的浏览器 Agent 实验，数据为当日任务页面读数。想听听大家：你们公司的业务流程里，哪一个环节你敢交给 Agent，哪一个绝对不给？）
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
The day before, I had Agent, who was able to run his own browser for the daily task of digging for gold, simply wanted to see how well it fit into the real site, and fell into the details of the project: the real problem was not to let the model "know what it is" but to keep the whole cycle right in front of the real front. The article does not talk about model lists, but about a few real problems I've encountered on the road along the chain, and about the trade-offs behind it.
I. Four routes, same hypothesis
Current mainstream browserAgent is broadly divided into four categories:
- Visual: Anthropic's computer use belongs to this category, with a model screenshot, output of mouse coordinates and keyboard actions. The advantage is to be generalized - any interface can be tried; the cost is a thousand tokens of screen 1080p amplifiers, and the coordinates are left empty by dozens of pixels.
- Model built-in action space: Operator/CUA-type scheme for Open AI, using browsing and clicking as the original action of the model, does not require site cooperation.
- Multiple labels and humans: Project Mariner of Google, emphasizing parallel tabs and human confirmation of key actions, wrapping uncertainty into interactive design.
- Structured DOM / Accessible Tree: Domestic IDE-type tools (including byte Trae 's browser capacity) and choice of a large number of MCP services: group the interactive elements on the page into a list of ref, with the model simply saying "point e42".
A common assumption of the four routes is that the model is responsible for perception and decision-making and that the rest is left to the project's base. This hypothesis is basically in the context of a reading, and in the context of writing.
II. The first real question: context is not "pull in," it's "format."
A screen stream page in its original form, hundreds of thousands token without exaggerating. So snapshots must be visible filters, text cut-off, and duplicate nodes merge. It's basically a gamble: models don't need to see everything right.
But what really affects costs and stability is whether the sequence is stable. Now families are reducing prices with Prompt Caching / KV Cache, and the ideal structure of an Agent cycle is: System Tip + Tool Definition + History Action remains unchanged at byte level, with only final snapshots changing. Once the snapshot element sequence, id method of generation or attribute name drags cache's backside, the hit rate drops, and the cost of the same task is several times lower.
Another pit is the memory of the long mission. The session was automatically compressed once and the next round would be repeated if the handover of summary did not spell out "which have been done, how many have been done and what evidence has been obtained." I then fixed four segments: completed (with numbers and evidence) / not completed / clear ban / next step sequence. The constraint left in summary is equal to never written.
The second real problem: writing is the watershed of reliability.
When you read the page, Agent is almost correct; when you touch the writing operation, the problem comes out.
To give a true example: the "Zoo" button in the community is essentially a svg icon, with no text or separate nodes on the accessible tree, and the tool can only take the parent container and guess by shifting coordinates. Speculate a few pixels, the tool layer returns access, the model takes a "sweet" and the back end does not receive a single request.
There are only two genuine verifications: to see whether the web request appears; and to see that the DOM case has changed from "show" to "sweet". The latter is extremely low because the UI status writeback is the best assertion of business success.
So I'm saying that the tool contract is mandatory. It is not allowed to declare its own completion without a written assertion. By the same token, retry logic has to be designed in terms of Zen, etc. — to stop at once, rather than fail the mission, rather than "Zon + cancel" offset each other and use a white flag.
IV. THE THIRD REAL QUESTION: THE TYPE LOOKS AT ACTS
Community-based anti-deceptions now rarely rely on the conclusion of the certification code, but rather on the distribution of hits, the length of stay and the appearance of a rolling curve. Two consecutive Zero clicks are much more dangerous in the wind's eye than a slowly rolling script.
This layer has a greater impact on technology selection than would have been imagined: if the product were to use Agent as a function, it would have to simultaneously define an automated frequency limit; otherwise it would not have been a function but an entry point for an automatic brush. The future platform's engagement with Agent is likely to depend not on the agreement but on who is responsible for the act.
Five: Agent will not replace the front, but will redefine the front.
When the page reading was referred to the model, semantic HTML went from accessibility compliance to AI usability interface. A button with aria-label is more valuable in Agent 's eyes than three decorative div. In turn, those interfaces that canvas-based full-scale, without text snapshots become information isolation islands in the Age of Agent.
As to whether the validation should be in a model or a model, my judgment is clear: the model is responsible for the intent, and the model is responsible for a verifiable state migration. And that explains why the same model behaves so poorly in different Agent products -- the difference is not on the model level, on the tool level.
Six, for someone who's ready.
- Distinguishing between reading and writing: reading can be as automated as possible, and self-assertion must be written.
cache: Elements numbering, sequence, attribute set as stable as possible.
- If you fail, then stop. retry is designed, like, by Zulu.
- The long-term mission must be handed over to summary, and summary, which leaves no evidence, will lead directly to duplication of work.
- Returning to the public actions requiring trust (payment, security confirmation, public statements).
The power of the model will continue to roll, but the ability to enter the real workflow will depend on the uncertainty that you can hold it.
(This is based on a real browser Agent experiment with the data being read on the task page of the day. I want to hear from you: Which part of your business business is that you give to Agent and which one you don't give?
Auto-fetched and translated by AI Atlas. All rights belong to the original author.