Google, Anthropic等支持新的AI爬行选择退出标准
Google LLC,Microsoft公司，Anthropic PBC和OpenAI Group PBC 今天，宣布计划支持即将到来的网站爬行标准。
该技术将使网站运营商能够阻止人工智能开发商使用其内容来培训其模型。
许多网站包含名为“robots.txt”的文本文件，出版商可以使用该文件来阻挡横行的刮刮，或者只使特定的网页无法访问。 也有可能包括某些其他指示。 例如，公司可以要求爬行者不要太频繁地访问其网站。
网站运营商在历史上使用机器人.txt文件来管理搜索引擎如何与其属性互动。 目前，该技术还有助于监管人工智能提供者收集培训数据的刮刮机的活动。
想要拒绝与训练有关的刮刮作业的公司，必须对其网站的机器人.txt文件进行多次编辑。 特别是，它需要为每一个AI提供者添加一个单独的条目。 如果AI供应商使用多个刮刮机收集数据，每个刮刮机都需要自己的机器人.txt输入。
爬行标准Google,Microsoft,Anthropic和OpenAI都是支持旨在简化过程。 它将允许网站操作员屏蔽所有收集使用单一机器人.txt条目的基因AI模型培训数据的刮刮器。 这种设置不适用于“完全执行非遗传性任务”的模式。
机器人.txt文件的结构遵循一个标准格式，称为机器人排除协议。 Google,Microsoft,Anthropic和OpenAI计划将其爬行标准纳入格式。 他们已经向维持《机器人排除议定书》的非营利组织因特网工程工作队提交了提案草案。
Microsoft在博客文章中表示，以回答用户问题，Microsoft的设置将使网站运营商能够阻止其服务发送此类请求。 该功能还将延伸至第三方AI工具，这些工具使用Microsoft技术处理提示。
OpenAI计划提供类似的控制。 ChatGPT开发商和Microsoft都打算在接下来的几个月里加入对爬行标准的支持，而Google将在明年的某个时候效仿。 Meta平台公司表示支持技术背后的想法，但尚未承诺实施。
图像：取消斜拉
SiliconANGLE联合创始人John Furrier的留言：
支持我们的使命，通过与CUBE社区接触，保持内容的开放和自由。 加入CUBE的校友信托网络，其中技术领导者相互连接，分享情报，创造机会。
- 15M+观看CUBE视频，为AI、云层、网络安全等各种对话提供动力
- 11.4k+CUBE校友——通过独特的信任网络与11400多名技术和企业领袖建立联系，塑造未来
你是AWS客户吗？在财务上支持SiliconANGLE，从我们的市场门户页面和链接购买您的AWS服务：https://siliconangle.com/aws-marketplace/.
关于 SiliconANGLE 媒体
SiliconANGLE Media由技术视觉家约翰·弗瑞尔(John Furrier)和戴夫·韦兰特(Dave Vellante)创立，构建了产业领先数字媒体品牌的动态生态系统，达到1500+百万精英技术专业人士。 利用CUBEAi.com神经网络，帮助科技公司以数据为导向做出决策
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Google, Anthropic and others to support new AI crawling opt-out standard
Google LLC, Microsoft Corp., Anthropic PBC and OpenAI Group PBC today announced plans to support an upcoming website crawling standard.
The technology will enable website operators to block artificial intelligence developers from using their content to train their models.
Many websites contain a text file titled “robots.txt” that informs crawlers what content they may access. A publisher can use the file to block scraping across the board or only make specific webpages inaccessible. It’s also possible to include certain other instructions. For example, a company can ask crawlers not to visit its website too often.
Website operators historically used robots.txt files to manage how search engines interact with their properties. Nowadays, the technology also helps regulate the activity of the scrapers with which AI providers collect training data.
A company that wishes to disallow training-related scraping must make multiple edits to its website’s robots.txt file. In particular, it needs to add a separate entry for every single AI provider. If an AI provider uses multiple scrapers to collect data, each scraper requires its own robots.txt entry.
The crawling standard Google, Microsoft, Anthropic and OpenAI are backing aims to simplify the process. It would allow website operators to block all scrapers that collect training data for generative AI models with a single robots.txt entry. The setting wouldn’t apply to models that “perform exclusively non-generative tasks.”
The structure of robots.txt files follows a standardized format called the Robots Exclusion Protocol. Google, Microsoft, Anthropic and OpenAI plan to incorporate their crawling standard into the format. They have already submitted a draft version of their proposal to the Internet Engineering Task Force, the nonprofit that maintains the Robots Exclusion Protocol.
Microsoft stated in a blog post that it will also provide a crawling setting called AI Grounding. It’s designed for situations where an AI model seeks to access a webpage in order to answer a user question rather than to collect training data. Microsoft’s setting will enable website operators to prevent its services from sending such requests. The feature will also extend to third-party AI tools that use Microsoft technologies to process prompts.
OpenAI plans to offer similar controls. Both the ChatGPT developer and Microsoft intend to add support for the crawling standard in the coming months, while Google will follow suit sometime next year. Meta Platforms Inc. expressed support for the idea behind the technology but hasn’t yet committed to implementing it.
Image: Unsplash
A message from John Furrier, co-founder of SiliconANGLE:
Support our mission to keep content open and free by engaging with theCUBE community. Join theCUBE’s Alumni Trust Network, where technology leaders connect, share intelligence and create opportunities.
- 15M+ viewers of theCUBE videos, powering conversations across AI, cloud, cybersecurity and more
- 11.4k+ theCUBE alumni — Connect with more than 11,400 tech and business leaders shaping the future through a unique trusted-based network
Are you an AWS customer? Support SiliconANGLE financially by buying your AWS services from our Marketplace portal page and links: https://siliconangle.com/aws-marketplace/
About SiliconANGLE Media
Founded by tech visionaries John Furrier and Dave Vellante, SiliconANGLE Media has built a dynamic ecosystem of industry-leading digital media brands that reach 15+ million elite tech professionals. Our new proprietary theCUBE AI Video Cloud is breaking ground in audience interaction, leveraging theCUBEai.com neural network to help technology companies make data-driven decisions and stay at the forefront of industry conversations.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.