安德森角
所有 " Ok " 网络响应的20% AI Gets Are not okay
在 Google 上的首选来源中添加 Unite.AI
我最近通过人工智能协助的网络搜索实验，当然，这种缩减的动因是AI网络搜索的惊人增长，而AI对新数据的毫不妥协的需求最终会让域域脱机。
这是一个很好的借口，让域来做他们中许多人长期想要做的事情：只有当用户登录(虽然Googlebot通常会获得免费通行证)时才能访问其内容。 Facebook于2023年完成了这项政策的逐步通过，Twitter/X也于同年完成。 Reddit本月晚些时候将采用。
这样，遵守任何针对具体国家的年龄限制规定，对域而言就容易一些，因为用户已经登记。 更重要的是，用户的数据可以保存起来，用于本地分析，如果当地立法允许，则出售给数据经纪人和AI公司。
Cloudflare表示，在其他地方，旨在抵御bots(而不是注册用户)的措施逐年上升了58%以上；所有不断升级的bot活动自然鼓励网站仔细检查VPN和代理地址，可能拒绝合法用户访问，以及任何“恶意”流量。
我甚至可能放弃一个开始加载的站点；由于已发现CAPTCHA - 墙会阻遏用户和潜在的客户，人们可以想象
所有这些新的和不断加剧的摩擦部分是后勤问题(因为并非所有领域都能承受攻击)，也常常是一种谈判策略
由于AI爬行者必须大规模工作，他们倾向于在Playwright框架下使用Chrome的"无头"例子。 这些天，对于越来越多的域来说，这将给你带来一个403( " 禁止 " ) 错误( 即主机知道你是自动流量，现在滚开):
想要的是`200'(OK)客户的反应，即使人们必须把一堆钱交给一个居民代理农场
没有比200个响应更好的了；也就是说，没有一个客户端响应代码具有更高水平的正信号。 这也不是一个模棱两可的回答；它从来没有“几乎可以”、“待会可以”或“更可以”；它“很好！” 如果情况不妙
哈啰！走开
但是，根据一份新文件，这种缺乏颗粒性的情况可能严重扭曲AI数据集、统计估计和报告 " 良好 " 状况。
该研究* 估计，200类答复中约有五分之一根本没有可用的页面内容，而是提供了空网站壳、CAPTCHA挑战、访问限制或替代网页。
该文件指出：
`20.9%的静态客户端获得的具有200级状态的页面并不包含该页面的主要内容：应用程序贝壳、机器人挑战、块页、错误和占位符页以及部分内容。
因此，不耐烦的自动化系统被它们本身的稀缺性所绊倒，因为 " 200 " 响应被允许等同于成功获取，即使获取只是产生CAPTCHA墙、付费墙或其他障碍。
本文概述了七类内容
对于研究来说，在三天的时间里，10,307个网页同时使用一个基本的网络牵引程序和一个能够加载JavaScript的浏览器进行了测试，其响应按照检索的内容分类，而不是他们的HTTP状态代码。
" 无法使用 " 的202个答复中最大的类别是空网站壳和错误或替代页，总共占失败的三分之二以上。
关于空网站壳，一个网站可能显示其菜单和导航，同时将所请求的文章或产品保留在空白位置；在后一种情况下，可能会出现一个不相关的网页或主页，而不是所请求的内容，但两者都可以得到同样的HTTP 200状态，作为提供有用和有用内容的成功检索。
作为补救，本文建议网易工具报告实际检索的内容，而不是依赖HTTP状态代码，因为简单的检查异常短的页面，可疑的重定向和类似不存在页面的内容可以识别大多数失败。
然后，只有当一个页面似乎无法正常加载时，才能使用资源密集型的浏览器工具。 此外，AI评价可以检查所引用的资料来源是否实际包含预期信息，而不是假设HTTP 200回复意味着内容被成功检索。
新作品的作者指出认为“廉价heuristics”可以略微识别大约86%的无法使用的答复
为什么这么多错误的200个答复？
多年来，在开发者社区中，有一些观点认为，202( " 接受 " )错误( " 要求接受但尚未完成 " )可以消除这些 " 失败 " 200个地位要求；但实际上，202不能满足新文件中概述的所有情况，而且没有现有的代码可以。
虽然该文件没有深入讨论这一问题，但域名所有人允许200个错误的原因在未来的相关作品中可能值得进一步研究。
其中一个原因是，任何不是200码的东西本质上都是某种错误页或障碍，包括最常见的404( " 未找到 " )错误，因为目标内容完全缺失。 一些用户认为，过量的这种错误会导致在SEO(大概是GEO)中的惩罚，尽管Google否认这一点。
事实上，错误的200个错误被Google视为`软'的404个错误，显然可能比普通的404个错误更被视为一个问题，因为这些错误基本上是 " 欺骗 " 的结果，有意或不故意的。
引用
如果所有这一切看起来都是踏足，那么就必须考虑为了AI培训的目的，例行收集(或试图收集)的数据的线粒体规模。 如果五分之一的数据不可靠或完全是欺骗性的，下游的后果就很难像某种 " 附带的不准确性 " 那样缩水；相反，这种错误的反应可能会对数据集的分布以及将这类数据纳入AI模型培训的结果造成重大和不利的影响。
例如，围绕错误的200个答复的一个显著问题涉及`引用漂移 '，即最初作为证据引用的网页不再支持AI生成的答案，即网页可能已经改变，相关文本可能已经删除，或者链接现在可能通向其他地方。
新研究发现，当所有HTTP 200答复被接受为成功检索时，无论它们包含什么内容，明显引用率从5.8%上升到21.9%。
这意味着，一个来源可能被错误地判断为更改、消失或停止支持AI生成的主张，仅仅是因为CAPTCHA、空页或其他障碍已被检索到取代了原始材料。
说吧
蓄意不存在网址的测试也表明了这个问题。 当从网站随机请求发明的路径时，13.1%的响应者返回了200(OK)状态，即使请求的页面不存在。
`13.1%回答随机路径探测器的场地返回了200个(S1.10.5%,S2 13.3%,S3 17.3%,S4 13.7%,S5 13.1%)。 每个这样的反应都是通过建造而软化的404，但不是每个称这样。
“在172个软-404回复的探索性分类中，标签者称98个错误或替代页，32个空壳，28个内容：一个主页或列表服务于随机路径，读起来像正常页。
结论
如果这个问题在更高层次上得到承认，那么IANA可以设想建立一个新的和更合适的http状态代码，例如[X]. (`请求成功，但请求的内容没有交付 ' ) 虽然这实际上可以描述正在发生的事情，但这自然要求东道方对交易有不太可能的深入了解。
类似的代码已经存在，尽管没有一条真正适合：451( " 出于法律原因无法获取 " )，模棱两可，而且相当令人震惊(!);206( " 部分内容 " )，技术上是合适的，但是为了不完全的数据(fetches)；以及203( " 非权威信息 " )，这是1997年创建的，它只是当前使用案例的边缘。
也许最近的候选者其实是非直觉地命名的103(`Early Hints ' )，这表明尚未装满URL的所有资源(尽管这显然是对代码意图的严重误解)。
* 标题为“200 OK不是答案：在代理网络内容获取中悄悄获取失败”。 请注意，该文件的撰写显然主要是机器可读的，也许需要大赦国际的质询，其本身并非易读。
我对机械学的引言表示歉意，但如我上文所指出，整篇论文的写法是 " 机器可读 "，而不是人文可译，尽管它没有以JSON格式提出。
2026年11月4日，星期三首次发表
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Anderson's Angle
20 Percent of All ‘Ok’ Web Responses AI Gets Are Not Okay
Add Unite.AI to your preferred sources on Google
My recent experiments in AI-assisted web-scraping have brought home to me the growing extent to which the web is becoming ‘locked-down’. The motivating event for this retrenchment, of course, is the shocking growth in AI web-scraping, whose uncompromising demand for new data can end up taking domains offline.
This is a good excuse for domains to do what many of them have wanted to do for a long time: make their content accessible only when the user is logged in (though naturally, Googlebot typically gets a free pass). Facebook completed its gradual adoption of this policy in 2023, and Twitter/X did likewise the same year. Reddit will adopt it later this month.
In this way, compliance with any country-specific age-limitation enforcement becomes a little easier for the domain, since the user is already registered. More importantly, the user’s data can be saved, exploited for local analytics, and – local legislation permitting – sold to data brokers and AI companies.
Elsewhere, measures squarely aimed at defending against bots (rather than enrolling users) have risen more than 58% year-on-year, according to Cloudflare; and all that escalating bot activity naturally encourages websites to scrutinize VPN and proxy addresses, potentially denying legitimate users access, together with any ‘malicious’ traffic.
The anecdatal aspect of this is that I seem to come up against ‘blocking’ responses or challenges more and more, to the point where I may even abandon a site that begins loading them; and since CAPTCHA -walls have been found to deter users and potential customers, one can imagine the pressure that causes domain owners to adopt them in spite of that downside.
All this new and increased friction is partly logistical (because not all domains can withstand the onslaught) but also often a kind of negotiating tactic to see if AI companies will offer a single or collective form of remuneration for all that data they so desperately crave, and which has been free for so long, until now.
Since AI crawlers have to work at scale, they tend to use ‘headless’ instances of Chrome under the Playwright framework. These days, for an increasing number of domains, that will get you a 403 (‘forbidden’) error (i.e., host knows you’re automated traffic, now scram):
What’s wanted is the ‘200’ (‘OK’) client response, even if one has to hand over a pile of money to a residential proxy farm, in concert with other diverse non-human sneaky tricks, in order to obtain it.
There’s nothing better than the 200 response; which is to say, there is no client response code with a higher level of positive signal. Neither is it an ambiguous response; it’s never ‘nearly okay’, ‘going to be okay later’, or ‘getting more okay’; it’s ‘okay!’. If it’s not okay, it’s not a 200 response.
Hello! Go Away
Except that this lack of granularity, according to a new paper, could be significantly skewing AI datasets, statistical estimations, and reporting ‘good’ status when that is not the case.
The study* estimates that around a fifth of 200-category responses contain no usable page content at all, instead delivering empty website shells, CAPTCHA challenges, access restrictions, or substitute pages.
The paper states:
‘20.9% of the pages that a static client receives with status 200 do not contain the page’s main content: app shells, bot challenges, block pages, error and placeholder pages, and partial content.’
In this regard, impatient automated systems are tripped up by their own sparseness, since the ‘200’ response is allowed to be equated with successful fetches, even when the fetch just produces a CAPTCHA-wall, a paywall, or some other obstacle.
The paper outlines the seven types of content found to frequently complicate a 200 response:
For the study, over three days, 10,307 web addresses were tested using both a basic web-fetching program and a browser capable of loading JavaScript, with responses classified according to the content retrieved, rather than their HTTP status codes.
The largest categories of ‘unusable’ 202 responses were empty website shells and error or substitute pages, together accounting for more than two-thirds of the failures.
Regarding the empty site shells, a website might display its menus and navigation while leaving the requested article or product listing blank; in the latter case, an unrelated page or homepage might appear instead of the requested content – but both could receive the same HTTP 200 status as a successful retrieval offering usable and useful content.
By way of a remedy, the paper recommends that web-fetching tools report what was actually retrieved, rather than relying on HTTP status codes, since simple checks for unusually short pages, suspicious redirects and content resembling nonexistent pages could identify most failures.
More resource-intensive browser tools could then be used only when a page appears to have failed to load properly. Additionally, AI evaluations could check whether cited sources actually contain the expected information, rather than assuming that an HTTP 200 response means the content was successfully retrieved.
The author of the new work also notes that ‘cheap heuristics’ can trivially identify around 86% of unusable responses, using basic checks on page length, redirects and similarities to nonexistent pages:
Why So Many Errant 200 Responses?
There has been some argument over the years, in developer communities, that the 202 (‘Accepted’) error (‘Requested accepted but not yet completed’) could disambiguate these ‘failed’ 200-status requests; but really, 202 cannot accommodate all the cases outlined in the new paper, and there is no existing code that can.
Though the paper does not deal with the issue in depth, the reason why domain owners permit 200 errors might be worth further investigation in future related works.
One reason could be that anything that is not a 200 is essentially some kind of error page or obstruction, including the most common, 404 (‘not found’) errors – where the target content is simply missing. Some users believe that an excess of such errors will lead to penalties in SEO (and, presumably, GEO), though Google denies this.
In fact, erroneous 200 errors are considered as ‘soft’ 404s by Google, and clearly likely to be treated as more of a problem than plain 404 errors, because these are essentially ‘deceptive’ outcomes, intentional or not.
Citation Drift
If all this seems like pedantry, one has to consider the petabyte-level scale of the data that is routinely gathered (or attempted to be gathered) for the purposes of AI training. If a full fifth of that data is unreliable or outright deceptive, the downstream consequences are difficult to shrug off as some kind of ‘collateral inaccuracy’; rather, such false responses could significantly and adversely affect the distribution in datasets and the outcomes of training such data into AI models.
For instance, one notable issue around erroneous 200 responses concerns ‘citation drift’, where an AI-generated answer is no longer supported by the web page that was originally cited as evidence, i.e., the page may have changed, the relevant text may have been removed, or the link may now lead somewhere else.
The new study found that apparent citation drift rose from 5.8% to 21.9% when all HTTP 200 responses were accepted as successful retrievals, regardless of what they contained.
This means that a source could be wrongly judged to have changed, disappeared or ceased to support an AI-generated claim, simply because a CAPTCHA, empty page or other obstruction had been retrieved in place of the original material.
Just Say It’s So!
The problem was also demonstrated by tests involving deliberately nonexistent web addresses. When random, invented paths were requested from websites, 13.1% of those that responded returned a 200 (‘OK’) status, even though the requested pages did not exist†.
‘13.1% of the sites that answered the random-path probe returned 200 (S1 10.5%, S2 13.3%, S3 17.3%, S4 13.7%, S5 13.1%). Every such response is a soft 404 by construction, but not every one says so.
‘In an exploratory breakdown of the 172 soft-404 responses, the labeller called 98 an error or substitute page, 32 a shell, and 28 content: a homepage or listing served at a random path reads like a normal page.’
Conclusion
If the problem was recognized at a higher level, the IANA could conceivably create a new and more apposite http status code, such as [X] (‘Request succeeded, but the requested content was not delivered’). While this would actually describe what is happening, it would naturally require an unlikely amount of insight into the transaction on the host’s side.
Similar codes already exist, though none really fit: 451 (‘Unavailable For Legal Reasons’), is ambiguous and rather alarming (!); 206 (‘Partial Content’), is technically apt, but intended for incomplete data (fetches) ; and the surprising-it-exists 203 (‘Non-Authoritative Information’), which was created in 1997, and is only peripheral to the use case at hand.
Perhaps the nearest candidate is actually the unintuitively-named 103 (‘Early Hints’), which indicates that not all the URL’s resources are loaded yet (though this would clearly be a gross misinterpretation of the code’s intent).
* Titled ‘When 200 OK Is Not an Answer: Silent Fetch Failures in Agentic Web Content Acquisition.‘ Please be aware that the paper is written, apparently, to be primarily machine-readable, perhaps to be queried by AI, and is not an easy read in itself.
† I apologize for the mechanistic quote, but the entire paper is written this way, as I note above, to be ‘machine readable’ rather than human-interpretable, though it stops short of being presented in JSON format.
First published Wednesday, November 4, 2026
Auto-fetched and translated by AI Atlas. All rights belong to the original author.