robots-check Reveals Which AI Crawlers Your Live robots.txt Allows

Summary

The GitHub project robots-check checks the robots.txt file that a website actually serves over HTTPS, addressing cases where Cloudflare changes the response without matching the origin file. It detects whether the file contains Cloudflare-managed content and points users to the Cloudflare AI Crawl Control setting. The command-line tool reports whether major search engines such as Googlebot and Bingbot can crawl, which AI training or crawling bots, including GPTBot, ClaudeBot, Google-Extended, CCBot, and Bytespider, are blocked or allowed, and whether a sitemap is declared. It requires Node 18 or newer and uses only Node's built-in https module, with no API key, signup, or dependency installation. The project cannot inspect the origin server separately; it reports the HTTPS response delivered to crawlers, which the author presents as the operative version when a CDN or WAF is in front of the site.