AI firms back shared publisher opt-out signal for web scraping
OpenAI, Google, Microsoft and Anthropic say they will support a shared signal allowing publishers to opt out of AI companies scraping their websites. The proposal is intended to let publishers set their preferences once, rather than configuring them separately for each company, as AI crawlers continue to collect online content for model development.
The signal, proposed by Anthropic standards lead Kevin Kelley through the Internet Engineering Task Force, would be added to a site’s robots.txt file. Anthropic has already adopted it; Google and Microsoft plan to support it next year, while OpenAI says it will do so in the coming months. Microsoft will also let publishers distinguish between AI training and AI grounding, while keeping sites in Bing search results.
In short
- Four major AI firms back a shared scraping opt-out signal.
- Publishers would add the preference to robots.txt.
- Rollout timing varies, with support planned from 2025.
Steelman: both sides, in good faith
The strongest fair case each way — we don't pick a winner.
The case for
A shared opt-out signal could give publishers a simple, consistent way to control whether AI firms use their material, without requiring separate settings for each company. This may strengthen publishers’ ability to protect their work and negotiate how it is used, while common standards make those preferences easier for firms to respect.
The case against
An opt-out signal still places the burden on publishers to identify and configure their sites, and may not resolve questions about material already collected or how preferences are enforced. Restricting scraping could also limit the material available to develop AI systems and, depending on how preferences are applied, reduce uses such as grounding while publishers may want their sites to remain discoverable through search.
Explained simply
AI companies use automated programmes, known as crawlers, to collect material from websites. That material can be used to train AI models or to help them answer questions with information from the web.
Website publishers can use a robots.txt file to tell crawlers which parts of a site they may access, but preferences may need to be set separately for different companies. The proposed shared signal is a common instruction that participating AI firms can recognise, giving publishers a simpler way to express their choices.
AI Cricket Government Politics Sport Technology
Read the full article at the source →
Originally published by The Verge as “The biggest names in AI are backing a new way to opt out of scraping”.