Kenneth Glad
Digital publishing faces new pressures as automated systems change how information reaches audiences. Platform rules are evolving too, leaving website owners to reconsider the costs of access and participation.
A relatively small change in clicking behavior could have large consequences for websites that depend on Google.
The effect shows up clearly in data from Pew Research Center. When Google displayed an AI-generated summary, users clicked a traditional search result on just 8% of visits. Without one, that figure rose to 15%. Very few people followed the links embedded in the summaries themselves, which drew clicks on only 1% of visits.
AI summaries are only one part of a much bigger shift in how people reach news and information online. Social platforms send less traffic than they once did, search habits have changed, and publishers have spent years adapting to a weaker digital advertising market.
Even so, the drop in Google traffic has been sharp for some outlets. Similarweb data reported by The Wall Street Journal showed that Business Insider’s organic search traffic fell 55% between April 2022 and April 2025. HuffPost lost more than half of its search traffic over the same period, while The Washington Post saw a similarly steep decline.
That matters because publishers make money from the people who actually arrive on their sites. Fewer visitors can mean fewer ad impressions, fewer subscriptions and fewer chances to turn occasional readers into regular ones.
Publishers now have another choice
Google changed the options available to publishers on August 31, 2026, when it completed the worldwide rollout of its Search generative AI control in Search Console.
Website owners can now exclude their material from generative features including AI Overviews and AI Mode without losing eligibility to appear or rank elsewhere in Google Search. Excluded pages, however, also lose the potential links, impressions and traffic that could come from those generative products.
Before the new control was introduced, publishers lacked this direct Search Console option for separating conventional Search from Google’s generative Search features.
Google treats AI training as a separate issue. Through Google-Extended, site owners can limit whether their content is used to train future Gemini models or support some Gemini and Vertex AI features. Google says using the setting does not change how a site appears or ranks in Search.
Remaining available to generative Search may produce exposure through AI products. Opting out preserves ordinary Search eligibility but removes the site from another increasingly prominent part of Google’s search experience.
Website owners must increasingly decide not merely whether they want search visibility, but which forms of it they are prepared to accept.
Crawlers are adding direct costs
Search referrals are only part of the changing economics.
Websites also have to serve automated systems collecting material from them. Cloudflare reported in July that automated bots generated roughly 57% of web requests across its network. That total includes many legitimate and non-AI forms of automation, so it should not be interpreted as evidence that generative systems account for most internet activity.
Individual AI crawlers can still produce substantial workloads.
Read the Docs, an infrastructure service widely used to host software documentation, reported that one crawler downloaded 73 terabytes of compressed HTML files during May 2024. Nearly 10 terabytes were downloaded in a single day. The activity resulted in more than $5,000 in bandwidth charges before the crawler was blocked.
The service said a crawler bug repeatedly downloaded the same files and lacked bandwidth limiting and standard HTTP caching mechanisms that could have reduced unnecessary transfers.
A separate incident documented by 404 Media involved iFixit, the repair website and documentation platform. Anthropic’s Claude crawler made nearly one million requests to the site within 24 hours in 2024. The company’s chief executive said the activity consumed DevOps resources, and iFixit subsequently modified its robots.txt file to block Anthropic’s crawlers.
For smaller websites, automated collection can become a direct operating expense, appearing in bandwidth charges or the workload of technical staff.
Human discussion has acquired new value
Material that people originally posted simply to communicate with one another has also become commercially useful to AI developers.
Reddit contains years of product advice, technical solutions, personal experiences, specialist discussions and informal conversation. Such user-generated material can be useful when training systems intended to understand and produce natural language.
Reuters reported in February 2024 that Google’s agreement with Reddit was worth about $60 million annually and made Reddit content available for training Google’s AI models.
Commercial interest in those conversations has also given marketers another reason to influence what appears on the platform.
In June 2026, 404 Media reported that moderators of Reddit’s Biohackers community said peptide and hormone-replacement businesses had been placing promotional material on the platform in an effort to influence information later surfaced by chatbots and AI search products. Moderators responded by banning new posts on those subjects.
That creates an obvious incentive for marketers. If AI systems are learning from Reddit discussions, companies have a reason to plant comments that could later influence the answers those systems produce.
The money is moving elsewhere
In a YouTube video, comedian and television host Adam Conover, best known for Adam Ruins Everything, argues that AI is reshaping the internet by changing how people create, share and find information. Much of his criticism comes back to a simple concern: The people doing the work are not always the ones benefiting from it.
Newsrooms still have to pay reporters and editors. Documentation sites still need developers and servers. Online communities depend on people spending time answering questions, posting advice and keeping discussions active.
AI companies and search platforms can make use of all that material in ways that do not always send people back to the websites where it came from. A search engine might answer a question directly. A model can be trained on material gathered from the web. A platform can license years of user posts to another company.
Publishers and website operators have not all responded the same way. Google’s August 31 update now allows publishers to keep their pages out of its generative Search features without giving up ordinary Search visibility. Some sites have restricted crawlers after seeing heavy automated traffic. Reddit, meanwhile, has chosen to sell access to its content through licensing deals.
Search and AI products will still send visitors to websites, but fewer searches now need to end with a click.
Many publishers depend heavily on people arriving through search. But when Google can answer a question directly on the results page, there is less reason for the user to open the original article at all.
Newsrooms pay reporters, documentation sites pay developers, and online communities need moderators and infrastructure. When fewer people visit those sources directly, less of the money generated around that information makes its way back to the people who created or maintained it.
Sources: Adam Conover video on YouTube; Pew Research Center; The Wall Street Journal; Google Search Central; Read the Docs; 404 Media; Reuters; Cloudflare
