Google joins OpenAI and Anthropic in promising new scraping controls

The new robots.txt directive will restrict companies from scraping content for generative AI training.

tadamichi/Shutterstock

Google, Anthropic, OpenAI, and Microsoft have agreed to honor a new robots.txt directive that lets publishers specify how their content may be scraped for generative AI training. The companies are adopting a proposal under review at the IETF, the internet standards body, authored by Kevin Kelley, Anthropic’s technical lead for standards.

OpenAI and Google note that site owners currently must create separate robots.txt rules for each crawler. Kelley’s proposal would enable multiple AI providers to recognize a single, shared signal. Publishers would add a dedicated user‑agent entry in robots.txt for AI training, formatted as shown:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

This directive blocks crawlers that are intended to collect data for AI training. The Allow clause lets operators permit scraping for specific sections, such as a press kit. However, the user‑agent section does not stop all crawling or remove content from general search results. For example, Google will treat the rule as a stop sign for the Google‑Extended crawler, which gathers material for training Gemini models.

It’s important to note that this directive cannot retroactively undo any scraping already performed; any content previously crawled remains in the companies’ datasets. It merely prevents future scraping of the specified pages. Anthropic has already implemented support, Microsoft plans to roll out support on February 1, 2027, Google will follow in early 2027, and OpenAI has pledged to add the signal in the coming months.

Source link