11 月 3 日的 GitHub Trending 榜首换成了一个 Rust 项目：vercel-labs/agent-desktop。仓库 11 月 3 日中午还在推代码，累计 11874 星，npm 上的 agent-desktop 包上周下载量 20713。它的自我介绍只有一句话：让 AI agent 通过操作系统无障碍树去读写桌面应用。
截图加视觉模型早就能够点按钮，所以这句话本身不算新鲜。真正值得拆开看的是它把「看」和「点」拆成了两个独立且可验证的步骤，并且把每一步失败的原因写进了 JSON 契约。这决定了它能不能被无人值守的流水线直接调用，而不是只能在演示视频里跑一遍就结束。
桌面上没有 DOM，只有一棵无障碍树
网页 agent 的日子好过，因为有 DOM。一个按钮有 id、有 class、有可读文本，选择器写错会立刻报找不到元素，错误信息直接指到出错的那一行。桌面应用没有这套东西，窗口里的控件活在进程内存里，外部进程拿不到句柄之外的东西。
于是只剩两条路。一条是截图交给视觉模型判断坐标，另一条是读操作系统自己暴露的无障碍树。前者通用，代价是脆；后者覆盖面窄，但每个元素自带语义。agent-desktop 走的是后者，分别对接 macOS 的 Accessibility API、Windows 的 UI Automation 和 Linux 的 AT-SPI。
无障碍树的核心好处是定位依据不是像素。窗口换了主题、DPI 从 100% 调到 150%、系统切成深色模式，树的层级和节点名字不会跟着变，一个 ref 仍然指向同一个按钮。截图方案在这些变化下需要重新校准，校准一旦失效，整条流水线就停在等待坐标超时上，而且日志里只剩一句超时，看不出究竟是界面变了还是模型看错了。
这也是为什么它的文档反复强调 ref 属于某一次快照。快照是一次真实观测的结果，不是一份可以长期缓存的元素地图。UI 变了，旧的 ref 立即失效，工具会明确告诉你失效，而不是猜一个位置点下去。这个差别在长时间运行的批处理里非常关键，因为缓存坐标的做法会在界面更新后的某个清晨静默失败，而缓存 ref 的做法会当场报错。
一条 snapshot 命令换回一张 ref 清单
核心命令是 snapshot，最常用的形态是只列可交互元素。加上 -i 参数后，返回的 JSON 里每个可操作节点都带 role、name、ref_id 和 available_actions 四项。ref_id 形如 @s8f3k2p9:e8，前面的短串是这次快照的编号，后面是节点序号，两者绑定在一起，单写后半段会被拒绝。
信封结构固定为 version、ok、command 三个顶层字段，成功时数据落在 data 里，失败时落在 error 里。当前契约版本是 2.6。一次典型的返回里，data 会给出 app 名、窗口 id、窗口标题、snapshot_id、ref_count 和一个 complete 布尔值。
complete 这个字段在 2.2 版本才出现，意义不小。旧版遇到超大窗口会直接抛 TIMEOUT 且不给任何树，调用方只能靠加大预算硬试。新版改成返回 ok 为 true、complete 为 false、truncated 为 true，附上已经观测到的树和 nodes_observed 计数，被截断的父节点各自带 subtree_truncated 标记，读者可以沿着根一路走到每一处边界。调用方只要读 complete 就能判断手里这棵树是不是完整的，不用再从超时错误里反推。
窗口多的场景不会替你选。同一应用开了多个窗口时，快照返回 AMBIGUOUS_TARGET，候选窗口 id 放在 error.details 里，需要显式传 window-id 指定一个。对密集应用，官方建议先用 skeleton 加 compact 拿到一张骨架图，再用 root 参数从一个节点往下钻。root 下钻是替换已有快照里的 ref，不是新建快照，所以观测不完整时它仍然返回 TIMEOUT，不会给你半棵树。这条区别很容易被忽略，写循环的时候要按不同语义处理：第一次快照不完整可以加大预算重来，下钻不完整只能退回上一层重新选节点。另一种常见浪费是把骨架图当成最终结果直接拿去点，骨架里的节点往往没有 available_actions，点了会被判动作不可用，白白多一轮快照。
动作认 ref，不认坐标
拿到 ref 之后，动作命令的写法很直白：click 接一个 ref，type 接文本，set-value 直接改值。ref 必须带快照前缀，单独写 @e8 会被判 INVALID_ARGS 拒绝，这条规则防的是跨快照误用。
真正需要提前想清楚的是三种失败态。STALE_REF 表示这个 ref 在当前的 RefMap 里找不到，说明页面或窗口已经变了，恢复策略是重新快照再拿新 ref。AMBIGUOUS_TARGET 表示目标不唯一，多窗口场景常见。第三个不是错误而是结果状态：动作交付成功但没有验证，返回值里 delivery 为 delivered_unverified，同时 verification_scope 为 unavailable。文档说得很清楚，这种情况只说明动作发出去了，不证明应用层真的落库，所以不能拿它当成功依据往下跑。
默认是 headless 行为。ref 动作全部走无障碍 API，不移动鼠标指针，也不抢窗口焦点。要物理输入必须显式加 --headed。对后台跑批的任务，这一点比速度重要得多，因为抢焦点会打断用户正在做的事，而无人值守的第一原则就是不打扰。
表格最后一列那句话是这套方案的天花板。自绘画布、游戏画面、远程桌面里的内容，无障碍树里没有对应节点，ref 无从生成，这条路直接走不通。选型时先问一句：要操作的那个应用，它的控件有没有语义。
信封里的 disposition 决定该不该重试
这套工具最省事的设计是把「能不能重试」做成了机器可读字段，而不是留给人去猜。每个错误都带 disposition 对象，里面两个值：delivery 和 retry。delivery 有五种取值，unknown、not_delivered、delivery_uncertain、delivered_unverified、delivered_verified。retry 有三种，unknown、safe、unsafe。
规则是推导出来的，不是拍出来的。delivery 为 not_delivered 时 retry 一定是 safe，其余非 unknown 的取值一律 unsafe。unknown 意味着这次调用什么也没证明出来，比如一个可能已经落地的操作超时了，这种情况按不安全处理，因为重试可能造成重复提交。
所以正确的调用方代码里只该有一行判断：只有 error.disposition.retry 等于 safe 时才自动重试，其余分支读 error.code 和 message 走人工或换策略。recovery 对象只在 retry 为 safe 时出现，里面给出策略名和是否需要新快照，TIMEOUT 与 APP_UNRESPONSIVE 还会带 retry_after_ms，可以直接拿来做退避间隔。
2.4 版本又补了一层。set-value、type、clear、check、uncheck、toggle、expand、collapse 这些有状态的动作用一次交付后就校验后置条件，发现矛盾直接报错，details.kind 标为 post_action_verification。这一条针对的是最阴的一类事故：动作返回成功，界面看着变了，实际状态没改，于是后续每一步都基于错的前提往下跑，最后失败的位置离真正的根因差了十几个步骤。校验失败时工具不会替你重试，它把矛盾摆出来，因为此时再点一次很可能把错误状态放大。读这个错误时先看 details.kind，再决定是重新快照核对界面真实状态，还是回滚到上一步的业务动作。把这套约定接进流水线之后，重试逻辑就从「失败就重试三次」变成了「按错误码决定重试或停机」，后者的好处是可以事后统计哪类错误最多，而不是只留下一个笼统的失败率。
三个平台的权限账不一样
Windows 最省事，不需要任何授权，前提是目标应用和终端的完整性级别一致。管理员权限运行的应用可以被读取，但只有同样提权的 agent 才能往里输入。这个差异在混合权限环境里会表现为「看得到点不动」，不是权限没给对，而是级别不匹配，排查时要先看两个进程的完整性级别。
macOS 的门槛是辅助功能权限，截图另需屏幕录制，操作通知中心还要给 System Events 自动化权限。permissions 子命令可以只读地报告当前状态，加 request 才会发起请求，加 guide 会直接打开系统设置的对应页面。装完第一件事应该是跑一次 permissions，而不是等动作失败再逐个排查。
Linux 的前提是跑在用户图形会话里，并且装了 at-spi2-core。纯 TTY 或 SSH 会话看不到任何应用，因为没有那个会话的总线、显示和运行时环境。官方给的主目标很明确：Ubuntu 24.04 加 GNOME 46，glibc 地板 2.35。Ubuntu 22.04 的 ATK 桥接在 Action.GetActions 上会直接崩，属于已知不兼容，不是配置能修好的问题。
还有一个容易踩的坑是二进制签名。发布包没有签名，从浏览器下载会触发 SmartScreen 提示，杀毒软件可能直接隔离，全新装的 Windows 11 上 Smart App Control 会拦下来。走 npm 安装不触发这一套，所以 Windows 上优先 npm 而不是手动下载压缩包。如果确实要用压缩包，可以用 gh attestation verify 核对来源，因为发布页的校验和只能证明文件没损坏，证明不了它来自这个仓库。
接进自己的循环：从 CLI 到 FFI
安装和第一轮验证只需要几条命令。注意 npm 12 默认屏蔽安装脚本，被屏蔽时包会退回 Node 启动器，每次调用多出约 125 毫秒，用 install-scripts approve 或 --allow-scripts 放行后才是原生二进制的约 21 毫秒。这个差距在批量任务里会累积成分钟级差别。
npm install -g agent-desktop
agent-desktop version
agent-desktop permissions
agent-desktop snapshot --app Finder -i
agent-desktop click @s8f3k2p9:e8
agent-desktop snapshot --app Finder -i
PowerShell 用户要把每个 ref 用引号包住，因为裸写的 @ 开头标记会被当成变量展开运算符，命令会被解析成完全不同的东西。这是官方文档单独拎出来讲的一条，Windows 上第一次跑就撞的概率很高，而且报错信息完全不指向真正的原因。
二进制自带文档是这套工具另一个实用设计：skills get desktop 讲循环、ref、验证和重试规则，skills get platform 讲当前操作系统上的能力差异，skills get ffi 讲语言绑定的入口、内存归属和线程规则。文档跟着二进制版本走，不会出现照着网页上的旧示例写出跑不通的代码，这一点比很多只在 README 里写用法的项目要靠谱。
要在进程内调用而不是每条命令起一次子进程，可以走发布包里带的 libagent_desktop_ffi。以 Python 为例，用 ctypes 加载库，调用 ad_init 并核对返回值，再创建 adapter 句柄：
import ctypes
lib = ctypes.CDLL("./lib/libagent_desktop_ffi.dylib")
lib.ad_adapter_create.restype = ctypes.c_void_p
lib.ad_adapter_destroy.argtypes = [ctypes.c_void_p]
assert lib.ad_init(7) == 0
adapter = lib.ad_adapter_create()
lib.ad_adapter_destroy(adapter)
那段 assert 校验的是 ABI 主版本号，头文件里的 AD_ABI_VERSION_MAJOR 必须和库匹配，否则后面所有调用都是未定义行为，可能不报错但结果全错。命令总数是 60 个，其中 56 个可直接用，剩下四个 key-down、key-up、mouse-down、mouse-up 在守护进程能释放被按住的键或按钮之前拒绝运行。这个限制是有意为之，防止一次崩溃把系统留在按键卡死的状态。
它替代不了什么
第一件事是隐私。快照内容、剪贴板读取、截图、命令输出和调试产物都可能带着私人桌面内容，官方文档专门提醒不要把它们放进公开日志或上传。可选的 JEV helper 会把屏幕描述发给你配置的模型服务商，那是独立组件，不是原生 CLI 的一部分，接不接要单独判断，不能因为装了 CLI 就默认同意了这件事。
第二件事是覆盖面。它只处理有语义的控件，自绘界面和游戏画面不在射程内。浏览器里做自动化，Playwright 那种按 DOM 选择器操作的方式仍然更精确，因为 DOM 比无障碍树信息量大得多，能读属性、能拦截请求、能等到具体网络事件。两套东西不是替代关系，是按场景分工。
第三件事是节奏。这套工具把一次操作拆成观察、动作、再观察三步，每次观察都是一次真实快照。对单人手工调试的脚本，这比一次性点完所有按钮要慢；对需要留下可审计轨迹的自动化任务，这三步就是它的价值所在。慢下来的部分正是可以被验证的部分，也是事后能复盘的部分。
放在 2026 年 11 月的这个时间点看，agent-desktop 的意义不在它多快，而在于它把桌面操作从「看得见就点得到」改成了「有语义才点得到，点完能证明」。前面那半句限制了适用范围，后面那半句决定了它能不能进生产流水线。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
On November 3rd, GitHub Trending took over as a Rust project: Vercel-labs/agent-desktop. The warehouse was still pushing code at noon on November 3rd, cumulatively 11874 stars, angent-desktop packages downloaded last week 20713. Its self-presentation is only one sentence: allow AI angent to read and write desktop applications through the system-accessible tree.
The screenshot and visual models have been able to click buttons long ago, so the sentence itself is not new. What really deserves to be seen is that it has broken "look" and "dot" into two separate and verifiable steps and has written the reasons for each failure into the JSON contract. This determines whether it can be called directly by unguarded water lines, rather than simply running through the demonstration video.
There's no Dom on the desktop. There's only one barrier-free tree.
The web site angent had a better day because there was DOM. A button has id, class, readable text. An error in a selector immediately indicates that the element cannot be found, and the error information directly points to the wrong line. The desktop application does not have this set, the controls in the window live in the process memory and the external process does not have anything beyond the handle.
So there were only two ways. One is an amplification to the visual model to determine the coordinates, and the other is an accessible tree exposed by the reading operating system itself. The former are universal and the cost is narrow; the latter have a narrow coverage, but each element has its own semantics. Agent-desktop goes to the latter, which connects MacOS to MacOS' Access API, Windows' UI Automation and Linux' AT-SPI.
The central benefit of barrier-free trees is that positioning is not based on pixels. Window changes the theme, DPI from 100% to 150%, system cuts to dark mode, tree level and node names do not change, one ref still points to the same button. The cut-off scenario requires recalibration under these changes, and once calibration has failed, the whole flow line is on the coordinates timeout and only one sentence of timeout is left in the log, and it is not clear whether the interface has changed or the model is wrong.
And that's why its document repeatedly emphasizes that the ref belongs to a snapshot. The snapshot is the result of a real observation and is not an elemental map that can be cache in the long term. UI changed, the old ref went off immediately, and the tool will tell you clearly that it's not working, rather than guessing a position down. This difference is critical in batch processing that is running over a long period of time, because the cache coordinates will fail silently at some point in the morning after the interface is updated, and cache ref will report errors on the spot.
A snapshot command for a ref list
The core command is snapshot, the most common form being a column of interactive elements. Adds the -i parameter, each of the operational nodes in the JSON returned has four role, name, ref id and aviable actions. In ref id form @s8f3k2p9:e8, the previous short string is the number of this snapshot, followed by the nodal number, which is tied together and the latter part of the single file is rejected.
The envelope structure is fixed to version, ok, top layer field, with data falling in the data when successful and in the error when failed. The current contract version is 2.6. In a typical return, Data gives app name, window id, window title, snapshot id, ref count and one complete boolean value.
Complete, this field appears only in version 2.2, which means a lot. The old version will throw TIMEOUT without giving any trees, and the caller will have to do it with a larger budget. The new version is to return ok to true, complete is false, truncaed is true, with the observed tree and nodes observed count, and the cut-off parent node is marked with subtree truncaed, and the reader can travel along the root to each border. The caller can determine whether the tree in hand is complete without being pushed back from the timeout error.
There's a lot of scenes in the windows that don't pick for you. When the same application opens multiple windows, the snapshot returns AMBIGUUUS TARGET, and the candidate window id is placed in error.details, which requires a visible upload window-id designation. For intensive applications, the official suggestion is to get a skeleton map with a skeleton plus a compact, then drill from a node with root parameters. The root drill replaces the ref in the already existing snapshot, not the new one, so when the observations are incomplete, it returns to TEMEOUT and will not give you half a tree. This distinction can easily be ignored and can be written in different semantics: The incompleteness of the first snapshot would increase the budgetary re-entry, and the incompleteness of the bottom drill would only return to the upper re-election nodes. Another common waste is taking the skeleton map directly as the end result, and the nodes in the skeleton are often uncapable actions, and they're condemned to be inoperable.
Ref, no coordinates.
When you get the ref, the action command is written plain: click next to the ref, type to text, set-value is changed. Ref must wear a snapshot prefix and write @e8 alone will be sentenced to INVALID ARGS refusal, a rule that prevents the misuse of trans-facing.
There are three failures that really need to be thought in advance. STALE REF indicates that this ref is not available in the current RefMap, indicating that the page or window has changed, and that the recovery strategy is to re-grace the ref. AMBIGUUS TARGET indicates that the target is not unique and multi-window scenes are common. The third is not an error but a result state: the action was delivered successfully, but did not verify, and returned to the value of the action was devied unverified, while verification scope is unaviable. The document makes it clear that this only indicates that the action was sent out and that the application layer did not prove to have actually landed, so it cannot be used as a basis for success.
Default is headless behaviour. Ref moves all accessible API, does not move the mouse pointer, does not grab the focus of the window. To physically enter, you have to add --headed. This is much more important than speed for backstage assignments, because a focus would interrupt what the user is doing, and the first principle that no one has to adhere to is not to disturb.
The last line of the table is the ceiling of the programme. The contents of self-drawn canvass, video games, remote desktops, with no corresponding nodes in accessible trees and no ref generation, are not accessible directly. Select the type first: the application that you want to operate is not semantic.
Disposition in the envelope determines whether or not retry
The most cost-effective design of the tool is to make "can retry" a machine-readable field, rather than leaving one to guess. Each error carries a dispossession object with two values: Delivery and retry. There are five extracts: unknown, not divered, Delivery uncertain, Delived unverified, Delived verified. Retry has three types: unknown, safe, unsafe.
Rules are derived, not taken. Retry must be safe when everything is not deleved, and the rest of the unknown is unsafe. Unknown means that this call has proved nothing, such as an operation that may have landed at timeout, which is treated as unsafe because retry may have duplicated the submission.
So there should be only one line of judgement in the correct caller code: retry is automatic only if the correct caller code equals safe, and the rest of the branches read error.code and message walk or change strategy. Recover objects only appear when retry is safe, which gives a strategic name and whether a new snapshot is needed. TEMEOUT and APP UNRESPONSIVE will also take the retry after ms, which can be used directly to avoid.
2.4 An additional layer was added to the version. Set-value, type, clear, check, uncheck, toggle, extend, collapse, which are active in a state of action, verify the post-fix condition and find a direct error, marked post action verification. This one deals with one of the worst types of accidents: a successful return to action, a change in the interface, and no change in the actual state of affairs, and then every subsequent step runs down on the basis of the false premise, and the final failure is more than a dozen steps away from the real root cause. When the verification failed, the tool was not going to help you with retry, and it put out the contradiction, because it was likely that the error would be magnified once more. Read this error by looking at details.kind, and then deciding whether to recheck the interface for real or roll back to the business action. With this package of agreements in place, the logic of retry went from "failure to retry three times" to "miscalating retry or shutting down", which has the advantage of counting what type of error is most likely after the event, rather than leaving a general failure rate.
permission accounts on three platforms are different.
Windows is the most economical, and no delegation of authority is required, provided that the target application and terminal integrity level are consistent. The application that the administrator permission is running is readable, but only the same delegate can enter. This difference is shown in the mixed permission environment as a "show-and-see" rather than an incorrect permission, but rather as a mismatch between the levels, with a first look at the integrity of the two processes.
The threshold for MacOS is the auxiliary function permission, the screen recording of which is required for the screens, and the operation notification centre for System Events automated permission. The permissions sub-command will report the current status only read-only, and then the request will be made with the request, and the side will open the corresponding page set up by the system. The first thing to do is run the missions once, instead of checking them one by one when the actions fail.
Linux presupposes running in a user graphic session and loading at-spi2-core. Pure TTY or SSH sessions do not see any applications because there is no bus, display and running environment for that session. The official target is clear: Ubuntu 24.04 plus GNOME 46, glibc floor 2.35. The ATK bridge at Ubuntu 22.04 will collapse directly on Action.GetActions, which is known to be incompatible and is not a problem for configuration to repair.
There is also a pit that is easy to step on is a binary signature. The publish package is not signed and downloading from the browser triggers the SmartScreen tip that poison software may be directly isolated and that a brand-new Windows 11 on Smart App Control will be stopped. Walk npm installation does not trigger this set, so Windows gives priority to npm instead of manually downloading compression packages. If a compressed package is indeed to be used, the source can be verified by gh attestation verifiy, as the reconciliation of the publish page only proves that the document was not damaged and that it did not come from this warehouse.
Connect to your cycle: from CLI to FFI
Only a few orders are required for installation and first round certification. Note that the npm 12 default shield installed scripts are returned to the Node launcher when shielded, with approximately 125 milliseconds of extra calls, using install-scripts approve or --allow-scripts before being released. This gap accumulates into minutes-scale differences in bulk assignments.
npm install -g agent-desktop
agent-desktop version
agent-desktop permissions
agent-desktop snapshot --app Finder -i
agent-desktop click @s8f3k2p9:e8
agent-desktop snapshot --app Finder -i
PowerShell users want to wrap each ref with a quotation sign, because a nudist @ starts mark will be used as a variable to expand the operator, and the command will be deciphered to something completely different. This is a separate item in the official document, with Windows having a high probability of a first run, and miscalculating information that does not point to the real cause.
The binary home-tape document is another practical design of the tool: skills get desktop on loops, ref, authentication and retry rules, skills get plainform on the current operating system, skills get ffi on language bound access, memory attribution and thread rules. The document follows a binary version and does not appear to be a code that can't run from an old example on the web page, which is more reliable than many items written only in README.
To call a sub-process within the process instead of once in each command, you can go to the libagent desktop ffi in the publish bag. Take Python, for example, load the library with ctypes, call ad init and check the return value, and create an adapter handle:
import ctypes
lib = ctypes.CDLL("./lib/libagent_desktop_ffi.dylib")
lib.ad_adapter_create.restype = ctypes.c_void_p
lib.ad_adapter_destroy.argtypes = [ctypes.c_void_p]
assert lib.ad_init(7) == 0
adapter = lib.ad_adapter_create()
lib.ad_adapter_destroy(adapter)
The ABI main version number, the AD ABI VERSION MAJOR in the front file must match the library, otherwise all subsequent calls are undefined and may not be misreported but the result is all wrong. The total number of commands is 60, of which 56 are directly usable, leaving four key-down, key-up, mouse-down, mouse-up before the daemon can release the pressed key or button. This limitation is intentional and prevents a collapse from leaving the system in the condition of a keycard.
It can't replace anything.
The first thing is privacy. Sniper content, clipboard reading, screenshots, command output and debugging products may carry private desktop content, and official documents specifically caution against posting them in open logs or uploading. An optional JEV helper will send the screen description to your configured model service provider, which is a stand-alone component, not part of the original CLI, and who does not have to be judged individually, and cannot acquiesce to this because the CLI is installed.
The second thing is coverage. It only handles semantic controls, and self-drawing interfaces and game images are not within range. Automation in browsers continues to be more accurate in the way Playwright uses the DOM selection, which is much larger than accessible tree information, reads attributes, intercepts requests and waits for specific web events. The two sets of things are not a substitute relationship. They are divided by scenery.
The third thing is rhythm. The tool breaks down one operation into three steps of observation, action and observation, each of which is a real snapshot. The manual manual debugging script for a single person is slower than the completion of all one-time buttons; the three steps are the value of automating the need to leave an auditable track. The slow down part is what can be verified and what can be repeated afterwards.
At this point in November 2026, the value of angent-desktop is not how fast it is, but rather how it changes the desktop operation from "to see" to "to have a semantic point of view, to have proof." The preceding sentence limits the scope of application, and the following sentence determines whether it can enter the production stream.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.