AI 模型和平台
Google计划支持共享机器人.txt AI培训控制
在 Google 上的首选来源中添加 Unite.AI
2026年10月30日的Google 宣布计划支持在机器人.txt中共享AI训练控制，给网站所有者一个单一的方式，在多个AI提供者中表达培训偏好。 承诺与Microsoft和OpenAI的同一天通知一并送达。
Google 将如何应用控制
Google搜索中心博客文章中，控制基于互联网工程任务组(IETF)AI优惠工作组的工作，并基于自2023年以来其提供的Google-Extend,AI培训控制Google 称。
网站所有者不会为每个AI开发者管理单独的指令，而是能够使用机器人内部共享的政策符号.txt来设定参与的AI提供者的广泛偏好。 当一个网站使用符号选择退出培训，且未明确指定Google-Extend,Google计划对此进行解释，仿佛出版商否决了Google-Extend. 其他AI提供商可以类似地处理信号，因此所有者可以立刻在所有参与的提供商中选择退出，或者选择。
根据该帖子，现有的公司特定控制将继续与联合信号一起运行，因此网站所有者可以通过行业信号表达广泛的选择，同时仍然通过Google-Extend等控制来针对单个开发者定制指令。 特别允许Google-Extend的网站有资格在响应中出现，并通过Gemini Apps的地面接收点击。
Google 称提案反映了来自出版商、民间社会团体、监管者和其他利益攸关方的反馈，这些利益攸关方呼吁对AI培训进行全行业控制，而AI主要供应商的支持将产生一个在整个行业发挥作用的持久标准。 该公司还提出称的建议，符合欧盟AI法案下的通用AI模型提供者的欧盟业务守则，该守则承认机器人.txt是网络出版商表达机器可读偏好的主要方法。 Google旨在支持新的控制，与其他AI提供商一起，在2027年和称，它将提供其最新进展。
OpenAI在同一天作出了平行承诺。 兹Q 1Z的公告称公司将尊重常见的机器人.txt信号，旨在在未来几个月内实施，并将保持其现有的爬行控制并得到尊重。 OpenAI 称它正在加入AI优惠工作组其他参与者的公告，并鼓励AI其他供应商尊重信号。
信号后面的 IETF 草稿
由Anthropic的Kevin Kelley编写、日期为2026年10月30日的题为“可读培训优先信号”的互联网草案中规定了这一机制。 该文件用于标准轨迹，于2027年5月3日到期；根据IETF数据跟踪仪，它是单个的呈文，没有得到IETF的认可，在标准进程中没有正式的地位。
在草案下，一个服务运营商通过写普通机器人将使用偏好与路径绑定。 txt 组合按键为每个目的的政策符，usage ai-training，位于用户-Agent线。 除了/press-kit/下的路径之外，推出机制没有新的指令名称，也没有新的语法：路径语言，最长的匹配优先级，以及机器人的组结构.txt标准，RFC 9309，没有变化，所以已经部署在互联网上的解析器会忽略他们不承认的政策调子组。 Wildcard groups，那些有User-Agent:*，不能被解释为匹配政策符，以及机器人。 txt文件没有 usage ai-training group 表示完全没有偏好。
该标志的含义来自AIPREF词汇对AI培训的定义：“ 利用资产修改基因AI模型的学习参数。 ” 该定义排除了专门用于非遗传性任务的模型的培训，即使模型有能力进行遗传性任务。 草案还强调，该标志表明倾向于使用内容，而不是对爬行行为的指令，尽管消费者可能利用偏好为爬行提供信息；具体表达方式应优先于一般表达方式，私人协议可能优先于表达的偏好。
Kelley 宣布在2026年10月30日的IETF的AI控制邮件列表上的草稿，写道Anthropic已经开始对其实施支持。 他说，该工作是对Anthropic现有爬行器控制的一种补充，而不是替代。
Microsoft的收养和过渡日期
Microsoft 宣布于同日由Microsoft Bing的首席产品经理Krishna Madhavan在Bing博客上通过。 Microsoft 称是第一家为AI培训和定位提供发布器控制的公司，指2023年NoArchive和NoCache标记它推出。
Microsoft正在采用网站和路径共享的AI Training偏好，并通过meta标记和HTTP响应头扩展至页面级别，同时增加了单独的页面级别AI Louting偏好。 除了允许和Disallow之外，基于新增的首选是一个新的DisallowExplated选项，它限制对Microsoft操作体验的地面使用，如AI Mode,AI Overview,Copilot，和Microsoft 365，同时排除了对第三方的地面服务Microsoft提供，如Web IQ，并通过在Foundry的API访问进行地面使用。 两种偏好都默认在不发布偏好时允许。 Microsoft警告说，选择Disallow或DisallowExtend用于打基础会降低AI体验中的品牌知名度；偏好不会改变传统Bing搜索结果的资格，也不会建立许可或支付安排。
新的首选将取代NoArchive和NoCache进行训练及搁浅，并随着Microsoft的爬行而于2027年2月1日生效。 2026年12月15日在Bing Webmaster Tools中为使用自当日起的旧标签的网站开放临时定位过渡偏好。 这些网站必须在2027年3月1日之前公布其新的首选，并记录临时的首选，之后NoArchive和NoCache将不再被解读为培训或定位的首选。 临时首选项一直有效，直到Microsoft重新绘制一个页面并发现新的定位首选项，或者直到2027年9月30日，以先到者为准。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
AI Models & Platforms
Google Plans Support for Shared Robots.txt AI Training Control
Add Unite.AI to your preferred sources on Google
Google announced on October 30, 2026 that it plans to support a shared AI training control in robots.txt, giving website owners a single way to express training preferences across multiple AI providers. The commitment arrived alongside same-day announcements from Microsoft and OpenAI.
How Google Will Apply the Control
The plan was laid out in a Google Search Central Blog post by John Mueller, a Google Search Advocate. The control is based on work from the AI Preferences working group at the Internet Engineering Task Force (IETF) and builds on Google-Extended, the AI training control Google says it has offered since 2023.
Instead of managing separate instructions for every AI developer, website owners will be able to use a shared policy token within robots.txt to set broad preferences across participating AI providers. When a website uses the token to opt out of training and has not explicitly specified Google-Extended, Google plans to interpret that as if the publisher had disallowed Google-Extended. Other AI providers can handle the signal similarly, so owners can opt out across all participating providers at once or pick and choose.
According to the post, existing company-specific controls will continue to function alongside the joint signal, so website owners can express a broad choice through the industry signal while still tailoring instructions to individual developers through controls such as Google-Extended. Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks through grounding in Gemini Apps.
Google said the proposal reflects feedback from publishers, civil society groups, regulators, and other stakeholders that have called for industry-wide controls on AI training, and that support across major AI providers would produce a durable standard that works across the industry. The company also said the proposal is consistent with the EU Code of Practice for general-purpose AI model providers under the EU AI Act, which recognizes robots.txt as a primary method for web publishers to express machine-readable preferences. Google aims to support the new control, along with other AI providers, in 2027 and said it will provide updates on its progress.
OpenAI made a parallel commitment the same day. OpenAI’s announcement said the company will honor the common robots.txt signal, aims to implement it in the coming months, and will keep its existing crawler controls in place and respected. OpenAI said it was joining other AI Preferences working group participants in the announcement and encouraged other AI providers to honor the signal.
The IETF Draft Behind the Signal
The mechanism is specified in an Internet-Draft titled “Machine-Readable Training Preference Signal,” authored by Kevin Kelley of Anthropic and dated October 30, 2026. The document is intended for the standards track and expires May 3, 2027; per the IETF datatracker, it is an individual submission that is not endorsed by the IETF and carries no formal standing in the standards process.
Under the draft, a service operator binds usage preferences to paths by writing ordinary robots.txt groups keyed by a per-purpose policy token, __usage_ai-training, in the User-Agent line. The draft’s example group expresses a preference against AI training for an entire site except for paths under /press-kit/. The mechanism introduces no new directive names and no new grammar: the path language, longest-match precedence, and group structure of the robots.txt standard, RFC 9309, are unchanged, so parsers already deployed across the internet will ignore policy-token groups they do not recognize. Wildcard groups, those with User-Agent: *, must not be interpreted as matching the policy token, and a robots.txt file with no __usage_ai-training group expresses no preference at all.
The token takes its meaning from the AIPREF vocabulary’s definition of AI training: “Using an asset to modify the learned parameters of a generative AI model.” The definition excludes the training of models used exclusively for non-generative tasks, even when the model is capable of generative tasks. The draft also stresses that the token states a preference about content use rather than a directive for crawling behavior, though consumers may use preferences to inform crawling; specific expressions should take precedence over general ones, and private agreements may override expressed preferences.
Kelley announced the draft on the IETF’s ai-control mailing list on October 30, 2026, writing that Anthropic has already started implementing support for it. He described the work as a complement to Anthropic’s existing crawler controls rather than a replacement.
Microsoft’s Adoption and Transition Dates
Microsoft announced its adoption the same day in a Bing blog post by Krishna Madhavan, Principal Product Manager for Microsoft Bing. Microsoft said it was the first company to provide publisher controls for AI training and grounding, pointing to the NoArchive and NoCache tags it introduced in 2023.
Microsoft is adopting the shared AI Training preference for sites and paths and extending it to the page level through meta tags and HTTP response headers, while adding a separate page-level AI Grounding preference. Alongside Allow and Disallow, the grounding preference adds a new DisallowExtended option that limits grounding use to Microsoft-operated experiences such as AI Mode, AI Overview, Copilot, and Microsoft 365 while excluding grounding services Microsoft provides to third parties, such as Web IQ and grounding through API access in Foundry. Both preferences default to Allow when no preference is published. Microsoft cautioned that selecting Disallow or DisallowExtended for grounding reduces brand visibility in AI experiences; the preferences do not change eligibility for traditional Bing search results and do not establish licensing or payment arrangements.
The new preferences will replace NoArchive and NoCache for training and grounding and take effect starting February 1, 2027, as Microsoft crawls. Temporary grounding transition preferences open in Bing Webmaster Tools on December 15, 2026 for sites using the older tags as of that date. Those sites must publish their new preferences and record a temporary grounding preference before March 1, 2027, after which NoArchive and NoCache will no longer be read as preferences for training or grounding. Temporary preferences remain in effect until Microsoft recrawls a page and discovers the new grounding preference or until September 30, 2027, whichever comes first.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.