Cloudflare and OpenAI Pilot Uses Network Signals to Improve AI Web Discovery

Summary

Cloudflare and OpenAI launched a research pilot in July 2026 to test whether signals from Cloudflare’s network could help OpenAI’s search crawler discover useful web pages more efficiently. Cloudflare says the pilot used a participating cohort representing about 1% of its total network and helped OpenAI discover and index hundreds of millions of previously unknown pages for potential use in ChatGPT Search. Cloudflare supplies discovery signals, while OpenAI independently evaluates and ranks pages for relevance and usefulness. The companies say this can help ChatGPT surface timely information and link users back to original websites and publishers. The pilot also examines whether a URL leads to an accessible, distinct page rather than a nonexistent page, a redirect, or duplicated content, allowing crawlers to avoid wasted fetches. They are testing signals for potentially harmful individual pages, such as a compromised page containing a fake login form, so that a specific page can be excluded without penalizing an entire domain. Change signals are intended to tell crawlers when a page deserves another visit, limiting repeated downloads of unchanged content and reducing unnecessary load for participating Cloudflare customers. Cloudflare notes that discovering new pages can initially increase crawling while those pages are assessed, so the goal is to remain current while making crawling more efficient. The next phase will refine page selection, test faster and more reliable change signals, and measure outcomes after discovery. The companies also emphasize that a discovery signal does not grant permission to access a page: the work uses web metadata for search and discovery, not page content for model training. Website owners can use Cloudflare’s AI Crawl Control to understand and manage how AI systems access their sites.