Using HTTP Status Codes to Push Back Against AI Crawlers

Summary

This opinion article examines how a website can use HTTP responses to push back against AI crawlers that request content. It notes that the site receives thousands of crawler requests despite having no published articles, while Cloudflare still exposes aggregate request statistics. The proposed 402 Payment Required response would ask AI companies to pay for access, but the author says it is unclear whether protocols such as Coinbase’s x402, Lightning Labs’ L402, or Stripe’s Machine Payments Protocol will gain adoption. The article also recalls Web Monetization, an earlier micropayment concept that failed to gain browser support. Other suggested responses include 403 Forbidden, 410 Gone, 418 I’m a teapot, 423 Locked with XML, and 426 Upgrade Required with a fictional Human protocol. A 429 response could impose a long retry delay, while 451 Unavailable For Legal Reasons could present a legal or political objection. The author treats most of these mechanisms as jokes or probably ineffective because crawlers may imitate browsers or ignore unusual responses. Beneath the humor is a serious concern that AI companies use online content without a reliable mechanism to compensate copyright owners, leaving creators uncertain about how to survive as AI systems continue collecting web material.