How to Classify AI Crawlers and User Fetches in Server Logs

Summary

Canonry examines how website operators can distinguish several kinds of AI-related traffic in server logs: automatic crawlers, user-triggered fetches, citations, and browser referrals. A question asked in ChatGPT or Claude always reaches the provider's backend, but only some questions trigger a separate request to an external site. When that fetch occurs on the provider's network, the origin may see a provider IP address and a user-agent such as ChatGPT-User or Claude-User rather than the person's browser. Because the provider retrieves HTML without executing the site's JavaScript, GA4 tags and marketing pixels may record no visit. These requests differ from automatic crawlers such as GPTBot, OAI-SearchBot, and ClaudeBot, while a citation requires inspecting the answer and its source links rather than merely observing a fetch. Canonry stores hourly rollups from Cloudflare, Cloud Run, Vercel, and WordPress, separating identity, normalized path, response status, and verification state. In a seven-day window on canonry.ai, it counted 947 requests using AI user-fetch identities and 1,667 using four crawler identities. Of 927 ChatGPT-User requests, 64 matched Canonry's stored May 17 OpenAI IP snapshot; checking OpenAI's current file later produced 76 matches. Most unmatched requests did not look like ordinary user fetches: 789 arrived in five hourly bursts and nearly all returned redirects or 404s while probing paths including /.env and /.ssh/id_rsa. The article says that pattern resembles scanning, but an IP mismatch alone cannot identify the caller, so user-agent claims remain unverified when they do not match published ranges. Its central conclusion is that crawls, user fetches, citations, and referrals must remain separate to avoid turning different events into one misleading AI traffic number.