Asger Risom
AI companies are not being good little spiders, as growing concerns are raising fresh questions about whether the AI boom is quietly hollowing out the open web.
AI companies are scraping the web at an unprecedented scale, but new data suggests they are sending far fewer users back to the sites that created the content. That growing imbalance is raising fresh concerns about whether the AI boom is quietly hollowing out the open web.
As artificial intelligence models grow more powerful, they rely on one resource above all else: vast amounts of human-created data.
But new figures suggest some of the biggest AI companies are extracting that data at scale without returning much value to the websites that host it.
Updated data from Cloudflare shows that AI firms including Anthropic and OpenAI are dramatically increasing how often their bots crawl websites, while referral traffic back to those sites continues to shrink.
The result is a widening gap between how much these companies take from the web and how much they give back.
For years, the internet operated on an informal bargain. Websites allowed search engines to crawl their content for free in exchange for visibility and traffic, which could then be monetised through ads, subscriptions, or services. Generative AI is now straining that arrangement.
Instead of sending users to original sources, AI chatbots increasingly provide direct answers, reducing the need to click through to the sites that produced the information in the first place.
What the Cloudflare data shows
Cloudflare, which helps run roughly 20% of the world’s websites, began tracking AI crawling behaviour in 2025. It measures how often company bots request data from websites compared with how many referrals those companies send back.
The result is a crawl-to-refer ratio — a simple indicator of how aggressively firms are extracting data relative to the traffic they return.
For the first week of January 2026, the gap was striking. Anthropic’s bots made roughly 65,000 crawl requests for every single referral sent back to websites. OpenAI’s ratio stood at around 1,400 to one. By comparison, Google’s ratio was far lower, at roughly five to one.
The data suggests that Anthropic and OpenAI are leaning heavily on the web as a raw material, while contributing relatively little traffic in return.
Rising costs for website owners
The imbalance is not just theoretical. Business Insider has previously reported that heavy AI crawling has driven up cloud-computing costs for some website owners.
In one case, a web developer said a client’s hosting bill doubled within months due to repeated requests from AI bots. That means some sites are effectively paying more just to be scraped — without seeing corresponding benefits in audience growth.
Cloudflare’s latest figures also show that crawling activity from Anthropic and OpenAI has increased compared with late 2025, suggesting the trend is accelerating rather than stabilising.
Industry silence and growing scrutiny
Anthropic did not respond to questions about the latest data. Previously, the company said it could not verify Cloudflare’s methodology and pointed to the launch of web search features inside its Claude chatbot as a way to increase referrals over time.
OpenAI also did not respond to a request for comment.
Cloudflare notes that the data focuses on web traffic and excludes activity inside native apps, which could slightly reduce the ratios if included. Still, the same methodology applies to every company measured, making the comparison meaningful.
Google’s lower ratio is likely tied to its traditional search engine, which still directs users to external websites. However, even Google has begun integrating more AI-generated answers into search results — raising questions about how long that balance will hold.
As AI companies continue to scale, the question looming over the industry is whether the web can remain healthy if its largest new users consume far more data than they return.
Sources: Business Insider, Cloudflare
