Google, OpenAI, Anthropic and Microsoft back shared AI scraping signal

Google, OpenAI, Anthropic and Microsoft have committed to recognising a proposed shared robots.txt signal that lets publishers state whether their content may be collected for generative AI training. The change could make it easier for website owners to set one preference across several AI companies, rather than managing separate crawler rules for each.

Publishers can add a dedicated user-agent entry to robots.txt to block AI-training crawlers, while allowing access to selected pages or sections. The directive does not stop ordinary crawling or prevent content appearing in Google Search, and it cannot reverse scraping that has already happened. Anthropic supports the signal now; OpenAI says it will implement it in the coming months, while Google and Microsoft plan to do so in early 2027 and on 1 February 2027 respectively.