Google, OpenAI, Anthropic and Microsoft back shared AI scraping signal
Google, OpenAI, Anthropic and Microsoft have committed to recognising a proposed shared robots.txt signal that lets publishers state whether their content may be collected for generative AI training. The change could make it easier for website owners to set one preference across several AI companies, rather than managing separate crawler rules for each.
Publishers can add a dedicated user-agent entry to robots.txt to block AI-training crawlers, while allowing access to selected pages or sections. The directive does not stop ordinary crawling or prevent content appearing in Google Search, and it cannot reverse scraping that has already happened. Anthropic supports the signal now; OpenAI says it will implement it in the coming months, while Google and Microsoft plan to do so in early 2027 and on 1 February 2027 respectively.
In short
- Four major AI firms back a shared web-scraping signal.
- Publishers can block future AI-training crawls in robots.txt.
- The change cannot undo past scraping or block Google Search.
Steelman: both sides, in good faith
The strongest fair case each way — we don't pick a winner.
The case for
A shared robots.txt signal could give publishers a practical, consistent way to express whether their work may be used to train generative AI, reducing the burden of setting different rules for each company. Recognising the signal across major providers could make those preferences easier to respect while preserving publishers’ ability to allow access to selected pages or sections.
The case against
The signal is voluntary and will take time to be adopted, so publishers may still face uneven coverage and uncertainty about whether their preferences are being followed. It also cannot undo past scraping or stop content from appearing in ordinary search, which may limit its value for publishers seeking broader control over how their material is collected and used.
Explained simply
Robots.txt is a standard file that websites use to tell automated programmes which parts of a site they may visit. Search engines and other services use these programmes, known as crawlers, to find and collect web pages.
Generative AI companies collect material from websites to help train systems that produce text and other content. Publishers can use crawler rules to express preferences, but each company may recognise different rules, so managing them can be complicated.
A shared signal would give publishers a more consistent way to state whether AI training crawlers may collect their material, including choices for particular pages or sections. It concerns future collection: it does not undo material already collected, and it does not by itself stop pages appearing in ordinary search results.
Read the full article at the source →
Originally published by Engadget as “Google joins OpenAI and Anthropic in promising new scraping controls”.