AI Crawlers Devour Web Traffic as Scraping Ratios Hit 38,000 Pages per Human Visitor

Websites are facing an unprecedented surge in automated scraping from AI training and inference bots, with some receiving over 35,000 page requests per human visitor delivered. Developers behind PatronView documented 3.6 million daily requests from hundreds of thousands of IPs, mostly from China, forcing them to block entire countries at the Cloudflare edge. Anthropic's Claude-SearchBot alone requested 420,680 pages in one week while sending only 12 human visitors, and similar patterns appear with OpenAI and Amazon crawlers. The Numbers site, a 30-year-old film database, went offline for a week after scraping attacks escalated to targeted reconnaissance for prediction market advantages. Cloudflare data shows training bots now treat the open web as a one-way data extraction pipeline rather than a reciprocal traffic source. Site owners report that blocking regions and aggressive rate limiting have become standard defensive measures against models like Qwen and Claude.

HabrTranslated from Russian

Read at source

AI-powered crawlers have transformed web traffic patterns, with robots now accounting for the vast majority of requests on many sites. One developer tracking charitable donors on PatronView measured 1.28 million page deliveries where fewer than 6,000 came from actual humans, producing a ratio of roughly 214 automated loads per human view.

Four main categories of bots are active. Honest SEO crawlers such as SemrushBot and AhrefsBot identify themselves and index content normally. Training bots like GPTBot and ClaudeBot perform bulk downloads to feed large language models. Search-oriented AI crawlers (Claude-SearchBot, Amzn-SearchBot) and agentic bots executing live user prompts generate the heaviest load.

In April, PatronView recorded 3.6 million requests in a single day from 361,844 unique IPs concentrated in China. Cloudflare blocked 1.18 million attempts over ten hours, yet a troubling share still bypassed CAPTCHA challenges. The site owner ultimately blocked entire countries at the edge, later adding Vietnam and Singapore, because 95.9 percent of its real audience is in the United States.

Traffic exchange ratios have collapsed. Cloudflare previously reported Anthropic crawlers at 3,000 pages per visitor; measurements on PatronView reached 35,000 to one. Claude-SearchBot fetched 420,680 pages in a week while Claude-User delivered only 12 humans. Amazon’s Rufus-related crawler requested 117,000 pages daily and returned zero visitors.

Founder Bruce Nash of The Numbers observed even steeper imbalances: Google at roughly five pages per visitor, OpenAI above 1,000, and Anthropic exceeding 38,000. The 1997-era film database suffered a week-long outage in March 2026 after scraping campaigns escalated into targeted reconnaissance for Polymarket prediction markets. The team ultimately rebuilt a reduced version on new infrastructure rather than risk repeated compromise of legacy code.

Meridian digestThe week's most consequential security stories, distilled into one briefing.See the digest

More in AI Security

Logical Argument Claims Full Alignment of Superintelligent AGI Is Fundamentally Impossible
Claude Account Bans Linked to IP Reputation, Device Environment and Payment Channels, But No Confirmed Formula
Local LLMs in AppSec: What Open Tools and Models Can Handle in Application Security