Vercel实验室拥有发布 agent-desktop，一个用于在macOS,Windows和Linux上使用AI agent计算机的开源 CLI. agent-desktop通过操作系统的无障碍树阅读每个应用程序，而不是通过截图。 agent得到一个JSON的快照，上面有它能够作用的每个元素的参考。 1.0.2版本于2026年10月30日发运，在Apache-2.0下。 对于开发者来说，任何可以 shell cap coding agent 现在都可以在没有视觉模型的情况下驱动本地应用。
TL；数据
- 图书馆：Rust CLI和一个C图书馆。 60个指挥名称，56个行动
-运行于：macOS 13+,Windows 10 1809+或Server 2019+,Ubuntu 24.04+，在x64和ARM64上。 Plain CPU，无GPU.
- 性能：决定性、安全性。 目前还没有独立的计算机用途基准测试。
- 最佳：密集的Electron应用软件通过骨架转盘(自报)的信号减少78%至96%。
agent台顶是什么？
agent-desktop是一个来自Vercel Labs的Rust CLI和C库，用于阅读和操作桌面应用。 它是agents调用的工具，而不是agent. 它不嵌入LLM，也没有GUI. 每个命令都打印一个JSON信封和出口。 观察行动核实循环留在你的agent
agent台台如何运作？
每个命令都通过固定的链条流动。 CLI或C库称agent-desktop-core，它只能通过一个平台Adapter特性到达OS. 一个平台板条箱然后与本地无障碍API交谈：
- macOS: AX API
- Windows:UI自动化
- Linux:AT-SPI号卫星
Core从不进口一个平台箱每个二进制被编译为驱动完全一个OS.
截图和校对
一个快照会指派合格的裁判，如 @s8f3k2p9:e8. 裁判内部的快照ID使其自我描述。 以 INVALID ARGS 拒绝一个赤色的 @e8。 当 UI 更改时，一个动作返回 STALE REF 或 AMBIGUUS TARGET。 agent再拍一张新鲜的快照。
每个错误都包含着一个处理。 Agents只应在称安全时重试。 校正解析度严格，未能关闭而非猜测。
滑石和钻井
敏捷的app可以曝光数千个无障碍节点。 A-skeleton快照只读深3级。 剪切容器报告儿童数量，并获得一个钻探员。 shapt - root 然后将一个区域读入同一shapt。
建筑商报告，Slack,VS代码和Notion的全树垃圾场有78%到96%的标志切。 在Apple地图中，他报告了3,811个令牌，每张表情下降到273个。 一个模拟的52步任务从198k降为31k令牌。 这些是自报的数字，不是独立的基准测试。
默认无头
校正动作会通过可访问性 API 不移动指针或聚焦。 这就是让agents在继续使用机器的同时工作的原因。 全球头旗允许在必要时进行物理输入。 每个校对动作还自动等待最高达5000 ms的可见，启用，稳定的目标。
它与像素计算机的使用相比如何？
模型侧计算机使用工具从截图和像素坐标工作。 兹Q 1Z的文档说每个截图大约花费1000到1800个输入符。 它的工具集定义新增每请求约4500个输入符。 兹Q 2Z 工具同样返回点击、 键入和滚动动作，您的代码执行。
agent-desktop是模型不可知和返回结构，而不是图像。 兹Q 1Z的UFO2也采用UI自动化，但舰只作为完整的Windows agent.
如何安装和使用 agent 桌面？
安装 npm 安装 - g agent 桌面。 Bun, Yarn和pnpm的工作也一样。 包将本地二进制捆绑为x64和ARM64.
- macOS：以agent-台式计算机权限授予无障碍权限 -- 指南。
- Windows：在您的终端完整性水平上不需要应用程序的赠款。
- Linux：需要一个图形化会话，使用-spi2-core。
Agents通过捆绑的docs通过agent-desktop技能学习工具获得桌面。 技能也列在技能.sh上。 C库让Python,Swift,Go,Ruby和Node主机在进程中调用。 对于Electron网页内容，发布-cdp打开一个DevTools端口用于agent-浏览器。
关键外卖
- agent-台式机通过OS访问树给予任何agent计算机的使用。
- 它运行于Apache-2.0下的macOS,Windows和Linux,x64和ARM64上。
- 无头反弹的动作从不移动指针或偷重点
- Skeleton快照将信物切78%至96%(自报)。
- 没有无障碍树的Apps仍然无法接近。
请检查date=中的日期值 (帮助) GitHub repo, Docs and npm 包。 并加入我们的通讯。 等等你在电报里吗现在你也可以加入我们的电报。
需要和我们合作推广您的 GitHub Repo OR Hugging Face Page OR Product Release OR Webinar 等吗？联系我们
米哈尔·萨特(Michal Sutter)是一名数据科学专业人员，拥有帕多瓦大学的数据科学硕士学位。 在统计分析、机器学习和数据工程方面有一个坚实的基础，Michal在将复杂的数据集转化为可操作的见解方面表现突出。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Vercel Labs has released agent-desktop, an open-source CLI for AI agent computer use on macOS, Windows and Linux. agent-desktop reads each app through the operating system’s accessibility tree, not through screenshots. An agent gets a JSON snapshot with a ref for every element it can act on. Version 1.0.2 shipped on October 30, 2026, under Apache-2.0. For developers, any shell-capable coding agent can now drive native apps without a vision model.
TL;DR
- Library: Rust CLI and a C library. 60 command names, 56 operational.
- Runs on: macOS 13+, Windows 10 1809+ or Server 2019+, Ubuntu 24.04+, on x64 and ARM64. Plain CPU, no GPU.
- Performance: Deterministic, retry-safe ref actions. No independent computer-use benchmark yet.
- Best: 78% to 96% fewer tokens on dense Electron apps via skeleton traversal (self-reported).
What is agent-desktop?
agent-desktop is a Rust CLI and C library from Vercel Labs for reading and operating desktop apps. It is a tool that agents call, not an agent. It embeds no LLM and has no GUI. Every command prints one JSON envelope and exits. The observe, act and verify loop stays in your agent.
How does agent-desktop work?
Each command flows through a fixed chain. The CLI or C library calls agent-desktop-core, which reaches the OS only through a PlatformAdapter trait. One platform crate then talks to the native accessibility API:
- macOS: AX API
- Windows: UI Automation
- Linux: AT-SPI
Core never imports a platform crate. Each binary is compiled to drive exactly one OS.
Snapshots and refs
A snapshot assigns qualified refs such as @s8f3k2p9:e8. The snapshot ID inside the ref makes it self-describing. A bare @e8 is rejected with INVALID_ARGS. When the UI changes, an action returns STALE_REF or AMBIGUOUS_TARGET. The agent then takes a fresh snapshot.
Every error carries a disposition. Agents should retry only when it says safe. Ref resolution is strict and fails closed rather than guessing.
Skeleton and drill-down
Dense apps can expose thousands of accessibility nodes. A --skeleton snapshot reads only 3 levels deep. Cut containers report a children_count and get a drill ref. snapshot --root then reads one region into the same snapshot.
The builder reported a 78% to 96% token cut versus full-tree dumps in Slack, VS Code and Notion. In Apple Maps, he reported 3,811 tokens per look falling to 273. A simulated 52-step task dropped from 198k to 31k tokens. These are self-reported figures, not an independent benchmark.
Headless by default
Ref actions go through the accessibility API without moving the pointer or taking focus. That is what lets agents work while you keep using the machine. The global --headed flag allows physical input when needed. Every ref action also auto-waits up to 5,000 ms for a visible, enabled, stable target.
How does it compare with pixel-based computer use?
Model-side computer use tools work from screenshots and pixel coordinates. Anthropic’s docs say each screenshot costs roughly 1,000 to 1,800 input tokens. Its toolset definition adds about 4,500 input tokens per request. OpenAI’s tool similarly returns click, type and scroll actions your code executes.
agent-desktop is model-agnostic and returns structure, not images. Microsoft’s UFO² also uses UI Automation, but ships as a full Windows agent.
How do you install and use agent-desktop?
Install with npm install -g agent-desktop. Bun, Yarn and pnpm work too. The package bundles native binaries for x64 and ARM64.
- macOS: grant Accessibility permission with agent-desktop permissions --guide.
- Windows: no grant needed for apps at your terminal’s integrity level.
- Linux: needs a graphical session with at-spi2-core.
Agents learn the tool from bundled docs via agent-desktop skills get desktop. A skill is also listed on skills.sh. The C library lets Python, Swift, Go, Ruby and Node hosts call it in process. For Electron web contents, launch --cdp opens a DevTools port for agent-browser.
Key Takeaways
- agent-desktop gives any agent computer use through OS accessibility trees.
- It runs on macOS, Windows and Linux, x64 and ARM64, under Apache-2.0.
- Headless ref actions never move your pointer or steal focus.
- Skeleton snapshots cut tokens 78% to 96% (self-reported).
- Apps without an accessibility tree remain out of reach.
Check out the GitHub repo, docs and npm package. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
Michal Sutter is a data science professional with a Master of Science in Data Science from the University of Padova. With a solid foundation in statistical analysis, machine learning, and data engineering, Michal excels at transforming complex datasets into actionable insights.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.