Cloudflare Lets Sites Block AI Training Without Losing Search Visibility

Summary

Cloudflare has introduced a Disallow AI Training setting that lets website owners refuse AI training while allowing qualifying mixed-use crawlers to continue indexing pages for traditional search. The change addresses crawlers such as Applebot, Googlebot, and Bingbot that can serve both search and training purposes, making a simple block potentially harmful to search visibility. Cloudflare classifies crawler activity as Search, Training, or Agent and applies separate domain-level controls. Under the new setting, Cloudflare publishes a no-training preference through robots.txt, keeps Accountable mixed-use crawlers available for search, and blocks other training crawlers, including training-only crawlers associated with Amazon, Anthropic, Meta, and OpenAI. Full blocking still stops mixed-use crawlers entirely, including their search activity, while blocking pages with ads applies to training crawlers on detected ad-serving pages. Cloudflare says the new controls are available to customers on all plans and will replace the less granular Block AI Bots and Managed Robots.txt systems, with existing settings migrated automatically. New domains will receive recommended presets based on whether they monetize through advertising. Apple, Google, and Microsoft qualify as Accountable under Cloudflare’s framework, which requires training opt-outs, AI-summary controls, URL-level visibility or reporting, and assurances that training opt-outs do not affect traditional search. Their implementations differ: Apple uses Applebot-Extended and is developing URL inspection; Google offers Google-Extended controls and reporting; Microsoft currently relies on NOARCHIVE and other tools while targeting robots.txt support for early 2027. Cloudflare also plans to let customers control how much content appears in AI summaries, rather than offering only a site-wide yes-or-no choice. The company cites data showing that AI summaries may reduce visits but that referrals from AI Search can convert at higher rates, so the appropriate choice depends on each site’s business model. The controls are available in Cloudflare domain Security Settings.