2026 年 10 月 17 日，The Verge 刊出一篇调查报道：OpenAI、Meta 和 Google 都在向用户承诺，AI Agent 可以替你订餐厅、清收件箱、买鞋，把网上的事全包了。可同一周里，Meta 的 Muse、xAI 的 Grok、OpenAI 的 Dots 的用户陆续发现，自己的 Agent 在一批主流网站上吃了闭门羹。这不是零星故障，而是一场有组织的关门潮，而且关门的手不只有一只。对做 Agent 产品的人来说，搞清楚谁在关门、门装在哪一层、怎么合法地把门敲开，已经成了必修课。这篇文章把过去两个月的公开报道、声明和条款整理成一张全景图，再给你一段可以直接跑起来的自检脚本，帮你在客户投诉之前先发现自己被拦在了哪一层。
关门名单正在变长
把过去两个月的公开报道拼起来，这份名单相当可观，而且还在每周增加。
9 月 21 日，彭博社报道亚马逊开始封锁 Meta 的 Muse，Agent 无法再浏览亚马逊的商品目录，更无法替用户下单。TechCrunch 在 10 月 6 日的报道里补充了更多名字：Yelp 明确表示只放行参加付费数据授权计划的 Agent；达美航空确认目前没有任何允许第三方 Agent 替客户查票订票的集成，其全球传播总经理 Heena Chavda 的表态是，未来若开放也必须以安全和客户体验为前提；联合航空则直接把记者指向使用条款里那句「不得使用任何 robot、spider 或其他自动化设备监控或复制本站内容」。
金融行业的动作更硬。The Verge 报道，摩根大通和嘉信理财都确认在阻止 AI Agent 访问客户账户，理由一致是安全考量。Cars.com 和美铁 Amtrak 也出现在用户的抱怨清单里，属于悄悄关门、不做公告的那一类，用户只能从「以前能用、现在不能用了」里察觉变化。
最戏剧化的还是亚马逊和 Perplexity 的旧案。2025 年 11 月，路透社报道亚马逊起诉 Perplexity，起因是后者的代理购物功能替用户在亚马逊下单。亚马逊声明里的措辞值得记住：第三方公司替客户购物可以，但「应当公开行事，并尊重服务方是否参与的决定」。Perplexity 的回应是一篇标题为 Bullying is not innovation 的博客。一边想进门，一边装防盗门，这就是 2026 年 Agent 生态的真实写照。
拦截是怎么发生的
开发者需要搞清楚的第一件事，是你的 Agent 在哪一层被拦。目前主流站点拦 Agent 的手段大致分四层。
第一层是规则与条款。robots.txt、使用条款、API 协议写明了什么能做什么不能做。联合航空引用的就是这一层，它未必当场拦你的请求，但为之后拦你甚至起诉你备好了依据。亚马逊起诉 Perplexity 走的也是这条路，技术手段之外叠加法律手段。很多团队把 robots.txt 当成建议，这是误解：在判例里，它是「你已被告知不得进入」的书面证据。
第二层是 WAF 和流量特征识别。Cloudflare、Akamai、PerimeterX 这类厂商会检查 TLS 握手指纹、HTTP/2 帧序列、请求头的顺序与缺失情况。真实 Chrome 浏览器的指纹和 Python 脚本、无头浏览器的指纹差异非常大，这一层不读页面内容就能把机器流量分拣出来。
第三层是行为判断。鼠标轨迹、滚动节奏、页面停留时长、点击位置分布，全部参与打分。Agent 的操作往往又快又均匀，一分钟内完成的操作序列在人类身上几乎不可能出现，分数自然垫底。有些厂商还会做主动探测：在页面里埋不可见的链接或表单字段，真人用户永远不会点到，照单全收的自动化程序却会一踩一个准。
第四层最显眼，就是交互式人机验证：点按「确认你是人类」的按钮、按住不放、选图验证码。TechCrunch 报道的沃尔玛案例最有代表性。沃尔玛和 Muse 是官方合作伙伴，9 月的 Meta Connect 大会上官宣过，理论上大门敞开。可用户实际下单时，页面弹出的人类验证按钮要求一次真实的人类点击，Agent 的流程在这里被打断，验证失败，会话被踢出。不是沃尔玛想拦 Agent，而是它沿用的反垃圾系统压根没为「被用户授权的合法 Agent」留通道。误伤和故意，从外面看一模一样，这正是当前最让人挫败的地方。
对开发者来说，这一层还有一个容易被忽视的细节：验证组件通常由第三方厂商提供，网站自己未必清楚它的具体判定逻辑。也就是说，你被拦的时候，对方客服很可能也说不清为什么。把排查精力放在「如何证明自己不是恶意流量」上，往往比「找客服申诉」更有效率。
一张表看清各家态度
这张表的分化很明显：零售商在「开门做生意」和「保护数据资产」之间摇摆，金融机构一致选择安全优先，内容平台则干脆把 Agent 访问做成了一门生意。Yelp 的思路值得单独看一眼，它不争论对错，直接标价：想进来，买授权。这种明码标价反而可能是对开发者最友好的关门方式，至少规则清晰、预期稳定。
自检：你的 Agent 被拦在哪一层
如果你的产品需要以 Agent 身份访问外部站点，先别急着研究绕过手段，先搞清楚自己面对的是哪一层的门。下面这个脚本只做两件合规的事：读目标站的 robots.txt 确认抓取权限，再以声明了身份的真实 UA 发一次请求，按响应状态码和页面特征把结果归类。
import urllib.request
import urllib.error
import urllib.parse
import urllib.robotparser
CHALLENGE_HINTS = [
"cf-chl", "cf-turnstile", "verify you are human",
"press and hold", "g-recaptcha", "akamai", "perimeterx",
]
def robots_allows(url, ua):
parsed = urllib.parse.urlparse(url)
robots_url = parsed.scheme + "://" + parsed.netloc + "/robots.txt"
parser = urllib.robotparser.RobotFileParser(robots_url)
try:
parser.read()
return parser.can_fetch(ua, url)
except Exception:
return None
def probe(url, ua):
req = urllib.request.Request(url, headers={"User-Agent": ua})
try:
with urllib.request.urlopen(req, timeout=10) as resp:
body = resp.read(60000).decode("utf-8", "ignore").lower()
hits = [h for h in CHALLENGE_HINTS if h in body]
if hits:
return "challenge_page", hits
return "ok", []
except urllib.error.HTTPError as e:
if e.code in (401, 403):
return "hard_block", [e.code]
return "http_error", [e.code]
except Exception as e:
return "network_error", [str(e)[:80]]
UA = "MyShoppingAgent/1.0 (+https://example.com/bot)"
print(robots_allows("https://example.com/", UA))
print(probe("https://example.com/", UA))
判读逻辑很直接。robots_allows 返回 False 就别去了，法律与条款风险远超技术收益。probe 返回 challenge_page，说明你被第三层或第四层拦了，页面能打开但内容是验证墙。返回 hard_block 则是第二层直接拒答。这个脚本的价值不在多聪明，而在把「为什么不行」从玄学变成一行可读的结论，省得团队在错误方向上烧时间。踩过的坑里最常见的一个，是把 challenge_page 误判成 ok：响应码是 200，标题也正常，但正文全是验证组件的脚本引用，真正的内容一个字符都没有。
脚本只依赖标准库，保存后直接 python 运行即可。实际使用时建议把目标清单、UA 字符串和结果输出改造成配置文件加日志文件的形式，每天跑一次，你就能在客户投诉之前先发现自己被拦了。对做监控的团队，把 challenge_page 和 hard_block 接进告警通道，比事后的用户工单要便宜得多。
合规活下去的四条路
看清门在哪之后，可选的路径其实不多，一共四条。
第一条，走官方通道。Meta 在 Muse 应用里维护着一份 connectors 合作方清单，沃尔玛在列，而且清单还在持续扩充。对开发者的含义很直接：想接入谁，先看它有没有官方 API、合作计划或者面向 Agent 的服务端点，有就签合同走正门。正门的手续麻烦，要审资质、要谈配额，但永远比爬窗户稳定，更不会在半夜被一次规则更新打回原形。
第二条，买数据。Yelp 模式会越来越普遍，内容型站点把 Agent 访问做成授权生意。对调用方是新增成本，对站点是新增收入，双方都有动力把这条路修平。如果你的业务依赖某家站点的数据，尽早询价，比等被拦之后再谈价格要主动得多。
第三条，盯紧正在成形的开放标准。TechCrunch 报道，Meta、沃尔玛、Stripe、Sierra、Genesys、Rocket、NiCE、Decagon 已经在联合制定一套开放协议，专门规范电商场景下 Agent 与商家之间的通信，目标是把「替用户办事的好 Agent」和恶意爬虫从协议层面区分开。这套标准一旦落地，Agent 的身份认证、权限声明、行为承诺都会有统一格式，被误拦的概率会大幅下降。现在跟进它的草案，比它成为事实标准之后再追赶要便宜得多。
第四条，如果只能自己抓取，至少把礼节做足：UA 里写明身份和联系地址，就像上面脚本里那样；严格尊重 robots.txt；限速加指数退避；只在缓存失效时重新抓取；遇到验证墙就停下来记录，而不是换 IP 继续撞。这些做法不能保证不被拦，但能保证你在任何争议里站在可辩护的一边，这一点在监管和诉讼越来越密集的今天，价值会越来越大。
这场博弈的下一步
Meta 在宣布开放标准时说过一句话：拒绝一个个人 Agent，等于拒绝它背后的那位客户。这句话点破了网站们真正的困境。Agent 背后站着的是带着明确购买意图的真实用户，一刀切地关门，关掉的可能是生意本身。达美航空那句「继续评估新技术如何改善客户互动」的官话背后，其实也是这层顾虑。
短期内指望形势反转并不现实。金融机构的安全理由站得住脚，零售平台的数据利益也真实存在。更可能的剧本是分化继续加深：有授权的 Agent 走标准协议进出，没授权的在越来越聪明的反爬系统面前逐渐失去操作空间。亚马逊诉 Perplexity 的官司会是一个关键风向标，它将告诉全行业，未经授权的代理购物在法律上到底站在哪一边。
对开发者来说，现在就该做的事很清楚：把 Agent 的身份标识、抓取礼节、降级策略当成基础设施来建设，而不是等被拦了再补。具体到今天就可以动手的有三件：给 Agent 起一个诚实且带联系方式的 UA，把 robots.txt 检查写进抓取流水线的第一步，再把上面那段自检脚本挂进日常监控。门正在一扇扇关上，但门上也正在装门铃，先学会按门铃的人，才进得去。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
On October 17, 2026, The Verge published a survey report: Open AI, Meta and Google are all promising users that AI Agent can book you a restaurant, pick up a box, buy shoes, and wrap up everything on the Internet. But in the same week, the users of Meta's Muse, XAI's Grok, Open AI's Dots discovered that their own Agent ate the shut-down on a number of mainstream websites. This is not a sporadic failure, but an organized closure, with more than one hand. It has become compulsory for those who do the Agent product to know who's closing the door, on which level it is dressed and how it is legally opened. This article brings together the public reports, statements and articles of the last two months into a panorama, and gives you a self-censorship script that can run directly and help you find out which floor you were stopped before the client filed a complaint.
The closing list is getting longer.
Putting together the public reports of the past two months, the list is considerable and is increasing every week.
On September 21st, Bloomberg reported that the Muse of Meta had begun to block Amazon, and Agent was unable to browse through the Amazon catalogue and to list users. TechCrunch added more names to the October 6 report: Yelp made it clear that only paid data authorization was allowed, Agent; Dharma Airlines confirmed that there was currently no integration allowing third parties, Agent, to check bookings for clients, and Heena Chavda, its global communications manager, stated that opening up in the future must also be subject to security and customer experience; and UTA directed journalists directly to the phrase in the usage clause that "no robot, Spider or other automated devices should be used to monitor or copy the content of the station".
The financial sector is moving harder. The Verge reported that Morgan Chase and C.I.C. confirmed that they were preventing AI Agent from accessing client accounts for reasons consistent with security considerations. Cars.com and Iron Amtrak also appear in the list of complaints from users, who close silently and do not make announcements, and who can only detect changes from what they "had been able to use before, not now."
The most dramatic is the old case of Amazon and Perplexity. In November 2025, Reuters reported on the Amazon prosecution of Perplexity, as the latter's surrogate shopping function placed a bill for customers in Amazonia. The language in the Amazon statement is worth remembering: third-party companies can shop for customers, but "should act openly and respect the decision of the service provider to participate". The response of Perplexity is a blog entitled Bullying is not inclusion. And while trying to get into the door, while trying to protect the door, that's the true picture of the 2026 Agent ecology.
How did the intercept happen?
The first thing the developer needs to know is where you're stopped. At present, the mainstream site's means of stopping Agent are roughly four layers.
The first is the rules and provisions. robots.txt, use terms, API protocol says what can and cannot be done. That's the level that UTA quotes. It doesn't have to stop your request on the spot, but it has a basis for stopping you later and even prosecuting you. Amazon vs. Perplexity is on the same path, where technical means are superimposed on legal means. Many of the teams use robots.txt as a suggestion, which is a misunderstanding: in jurisprudence, it is written evidence that "you have been told not to enter".
The second level is WAF and flow characterization. Claudflare, Akamai, PerimeterX, such factories, examine TLS handshake fingerprints, HTTP/2 frame sequences, order of request headers, and missing. The real Chrome browser's fingerprints are very different from the Python scripts, the headless browser's prints, so that without reading the pages on this level, the flow of machines can be separated.
The third level is behavioural judgment. Mouse trajectories, scroll rhythms, page stop times, click location distribution, all involved in scoring. Agent's operations are often fast and even, and the sequences of operations completed within a minute are almost impossible to show in humans, and the fractions are natural. Some manufacturers will also proactively detect: if an invisible link or form field is placed on the page, the real user will never be called, and the automated program that is collected on a single basis will step on it.
The fourth most visible level is interactive human validation: It's not like you're a human being. TechCrunch reports the most representative case of Wal-Mart. Wal-Mart and Muse are official partners, and the September Meta Connect congress officer declared that the door was in theory open. But when the user actually lowers the list, the human authentication button that appears on the page requires a real human click, where the process of Agent is interrupted, the validation failed and the session kicked out. Instead of Wal-Mart trying to stop Agent, the anti-spam system that it uses does not leave any way out for "legitimate Agent authorized by the user." Misdemeanour and intent, as seen from the outside, are the most frustrating places of the day.
For developers, there is also an easily neglected detail on this level: Validation components are usually provided by third-party manufacturers, and the website itself is not necessarily aware of its specific decision logic. In other words, when you're stopped, the other party's customer service probably doesn't know why. It is often more efficient to focus on "how to prove that you're not in bad traffic" than to "see for a client."
One watch shows the attitude of each family.
This table is clearly divided: retailers swing between "open doors for business" and "protecting data assets", financial institutions choose security priorities, and content platforms simply make Agent's access a business. Yelp's idea is worth looking at alone, and it doesn't dispute right or wrong, directly priced: want to come in and buy authorization. Rather, such a coded price may be the most friendly way to close doors for developers, at least with clear rules and expected stability.
Self-censor: Which floor did your Agent be stopped?
If your product needs access to an outside site as Agent, don't rush to figure out what level of door you're dealing with. This script is only for two compliance items: robots.txt, reading the target station, confirms the capture of permission, then sends a request to state the identity of the true UA, classifying the results by response status code and page characteristics.
import urllib.request
import urllib.error
import urllib.parse
import urllib.robotparser
CHALLENGE_HINTS = [
"cf-chl", "cf-turnstile", "verify you are human",
"press and hold", "g-recaptcha", "akamai", "perimeterx",
]
def robots_allows(url, ua):
parsed = urllib.parse.urlparse(url)
robots_url = parsed.scheme + "://" + parsed.netloc + "/robots.txt"
parser = urllib.robotparser.RobotFileParser(robots_url)
try:
parser.read()
return parser.can_fetch(ua, url)
except Exception:
return None
def probe(url, ua):
req = urllib.request.Request(url, headers={"User-Agent": ua})
try:
with urllib.request.urlopen(req, timeout=10) as resp:
body = resp.read(60000).decode("utf-8", "ignore").lower()
hits = [h for h in CHALLENGE_HINTS if h in body]
if hits:
return "challenge_page", hits
return "ok", []
except urllib.error.HTTPError as e:
if e.code in (401, 403):
return "hard_block", [e.code]
return "http_error", [e.code]
except Exception as e:
return "network_error", [str(e)[:80]]
UA = "MyShoppingAgent/1.0 (+https://example.com/bot)"
print(robots_allows("https://example.com/", UA))
print(probe("https://example.com/", UA))
The logic of interpretation is straightforward. Robots allows returns to False, and the law and the clause are far more risky than the technology. Probe returns challenge page, indicating that you are stopped on the third or fourth floors, that the page can be opened, but the content is a check wall. Returns hard block is the second level of direct refusal. The value of this script is not so smart, but the shift from "why not" to a line-readable conclusion saves the team time in the wrong direction. One of the most common types of pits that are stepped on is the miscalculation of kallenge page as ok: the response code is 200 and the title is normal, but the text is full of script references for the authentication component, and there is no real character.
Scripts only rely on the standard library and can be saved and run directly after python. When actually used, it is recommended that the target list, the UA string and the output of the results be changed to a configuration file with a log file and run once a day, so you can find yourself stopped before a client complains. For the monitoring team, it's much cheaper to connect challenge page and hard block into the alarm channel than a subsequent user list.
Four ways to live.
After seeing where the door is, there's really not a lot of options. There's four.
First, take the official route. Meta maintains a list of collaborators in the Muse application, Wal-Mart is in place and the list continues to expand. The implications for developers are straightforward: to whom access is sought, it has no official API, cooperative schemes, or service endpoints for Agent, and to sign the contract. The formalities at the front door are cumbersome, with qualifications reviewed and quotas discussed, but they are never stable compared to climbing windows, much less being hit back in the middle of the night by an update of the rules.
Second, buy data. The Yelp model will become more and more common, with content sites making Agent access an authorized business. The caller is an additional cost and the site is an additional revenue, and both parties have the incentive to level the road. If your business relies on data from a particular site, it would be much more proactive to ask for an early offer than to wait until it is blocked.
Third, focus on emerging open standards. TechCrunch reports that Meta, Wal-Mart, Stripe, Sierra, Genesys, Rocket, Nice and Decagon have jointly developed an open-door agreement to regulate communication between Agent and businesses in the context of the electricity business scene, with the aim of separating "good Agent" for users from "bad reptiles" at the level of the agreement. Once this set of standards has landed, Agent's identification, permission statements, and behavioural commitments will be presented in a uniform format and the probability of being delayed will be significantly reduced. It would be much cheaper to follow up on its draft now than to pursue it once it had become a de facto standard.
Fourth, if only by itself, the protocol is at least sufficient: the UA indicates the identity and contact address, as in the script above; strictly respects robots.txt; speed-added index ejection; recapts only when cache lapses; stops the record when encountered with the verification wall, instead of replacing IP. These practices do not guarantee that they will not be stopped, but they do ensure that you stand on the defence side of any dispute, which is of increasing value in today ' s increasingly intense regulatory and litigation processes.
next step of this game.
Meta, in declaring open standards, said that rejecting an individual, Agent, was tantamount to rejecting the client behind it. That's the point of breaking the real dilemma of the website. Agent stood behind a real user with a clear intent to buy it, closing the door, probably the business itself. It's really the same concern behind the official speech of Dame Airlines to "continue to assess how new technologies can improve client interaction."
It is not realistic to expect a reversal of the situation in the short term. The security reasons for financial institutions are valid and the data benefits of retail platforms are real. The more likely script is that it continues to deepen: the authorized Agent goes through standard protocols and the unauthorized gradually loses operational space in the face of increasingly intelligent anti-crawling systems. The Amazon v. Permexity case would be a key windmark, which would inform the industry as a whole of the legal side of unauthorized agency shopping.
For the developers, it is clear what should be done now: to treat Agent’s identification, capture etiquette, downgrade tactics as infrastructure, rather than waiting to be replaced. There are three things that can be done today: give Agent an honest and contactable U.A., write robots.txt in the first step of capturing the flow line, and put the self-censored script on the screen. The door is closing in a single door, but the door is also setting a bell and learning to ring the bell before entering.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.