PatronView

AI Crawlers Devour Web Traffic as Scraping Ratios Hit 38,000 Pages per Human Visitor

Websites are facing an unprecedented surge in automated scraping from AI training and inference bots, with some receiving over 35,000 page requests per human visitor delivered. Developers behind PatronView documented 3.6 million daily requests from hundreds of thousands of IPs, mostly from China, forcing them to block entire countries at the Cloudflare edge. Anthropic's Claude-SearchBot alone requested 420,680 pages in one week while sending only 12 human visitors, and similar patterns appear with OpenAI and Amazon crawlers. The Numbers site, a 30-year-old film database, went offline for a week after scraping attacks escalated to targeted reconnaissance for prediction market advantages. Cloudflare data shows training bots now treat the open web as a one-way data extraction pipeline rather than a reciprocal traffic source. Site owners report that blocking regions and aggressive rate limiting have become standard defensive measures against models like Qwen and Claude.