--- url: "https://www.xcademia.com/news/cloudflare-says-the-internet-now-has-a-second-audience-ai-agents" title: "Cloudflare Says the Internet Now Has a Second Audience: AI Agents" description: "Cloudflare says AI agents now form a second Internet audience, driving new challenges around traffic, content access, identity and web monetisation." publishedAt: "2026-10-01T09:48:43.378+00:00" updatedAt: "2026-10-10T01:32:46.119017+00:00" type: news category: "ai-ml" source_name: Cloudflare Blog source_url: "https://blog.cloudflare.com/agentic-web" tags: - "#AI" - "#AIAgents" - "#AgenticWeb" - "#Cloudflare" - "#AIInfrastructure" - "#AISearch" - "#WebAutomation" - "#FutureOfTheWeb" --- # Cloudflare Says the Internet Now Has a Second Audience: AI Agents > AI agents are becoming a major source of Internet traffic, creating new challenges for websites around cost, visibility and revenue. Cloudflare outlines tools for identifying, controlling, optimising and monetising agent traffic. Source: **Cloudflare Blog** · 1 October 2026 **T**he Internet is entering a new phase in which websites are no longer serving only human visitors. According to Cloudflare, software agents acting on behalf of people have become a significant second audience, changing how websites need to think about traffic, access, content and monetisation. In a September 30, 2026 blog post, Cloudflare said that the average HTTP request volume handled across its network has grown from 63 million requests per second at the end of 2024 to 115 million today, with peaks above 150 million. Cloudflare also reported that daily requests from AI agents on its network increased by more than 1,700% over the previous year. The company said that, for the first time, more than half of Internet traffic was not human. ## AI agents are different from traditional crawlers Cloudflare distinguishes AI agents from conventional automated crawlers. A training crawler primarily collects content to help build or improve an AI model. An agent, by contrast, retrieves information because a person has asked it to perform a task. That task could involve booking a restaurant, comparing insurance quotes or obtaining a dataset for research. In these situations, the software is acting on behalf of a potential customer. Cloudflare argues that this distinction matters because blocking an agent may also mean blocking the person who initiated the request. Agent traffic can therefore represent potential business activity rather than simply automated consumption. ## More automated traffic does not automatically mean more revenue The growth of AI-driven traffic creates a difficult economic problem for websites. For decades, search engines generally operated within a model where crawling helped send human visitors to websites. Those visitors could then view advertising, purchase products or subscribe to services. AI answer engines can work differently. They may retrieve information from a website, summarise it for the user and provide the answer without requiring the user to visit the original website. That can create a mismatch between traffic and revenue. Cloudflare said some heavily crawled categories, including Retail, Computer Software, IT & Services and Financial Services, have experienced human traffic declines of as much as 40% in less than one year. The company also said automated requests continue to consume bandwidth, compute and origin capacity even when they do not generate referrals, advertising impressions or subscriptions. ## AI training traffic has also changed Cloudflare said that in spring 2025, 22% of crawler requests it observed were for AI training, based on the crawlers' stated purpose. By June 2026, that figure had risen to 52%. Cloudflare's argument is that a simple decision to block all AI-related automated traffic is no longer sufficient. Websites increasingly need to distinguish between different types of automated activity and decide which forms of access they want to permit. ![info-1](https://0a515t3ure77wbvx.public.blob.vercel-storage.com/articles/1790847997255-info1--102-.webp) ## Visibility becomes important Cloudflare says website owners need more information about who or what is accessing their content. The company says that simply identifying something as an "AI bot" is no longer enough. Its AI Crawl Control, Business Insights and BotBase products are designed to provide information about automated traffic, including what crawlers take, what they return and which URLs they request. Cloudflare is also promoting Web Bot Auth, through which participating operators can cryptographically sign agent requests. The company said operators including OpenAI, Google and AWS are using this approach, allowing websites to distinguish verified agents from impersonators without relying solely on IP addresses or user-agent strings. Cloudflare reported more than 500 billion verified bot requests each week. ## Website owners get more granular controls Cloudflare said it replaced a single "block AI bots" control with separate Search, Agent and Training controls in July. The reasoning is straightforward: a website may want to remain visible to search engines while restricting AI training, or it may want to allow certain agent interactions while limiting automated access to advertising-supported pages. Cloudflare reported that fewer than 1% of sites on its network block search crawlers, compared with 17% that block training crawlers. The company said this indicated that many site owners were not trying to disappear from AI and search systems altogether. Instead, they wanted more control over how their content was used. On September 15, Cloudflare also announced Disallow AI Training, designed to allow search indexing while providing crawler-specific instructions against using content for training. Cloudflare said Apple, Google and Microsoft had committed to honour the mechanism. ## From blocking agents to charging them The bigger change described by Cloudflare is the move from access control towards monetisation. Cloudflare says its vision for an agentic Internet includes four characteristics: readable, discoverable, callable and payable. The company argues that content licensing already demonstrates demand for publisher content, but traditional licensing arrangements are often bespoke agreements between large publishers and large AI companies. Cloudflare says this model does not easily extend to the broader web. ## Pay Per Use Cloudflare's **Pay Per Use** approach is designed to allow publishers to receive payment when their content is actually used rather than simply when it is crawled. Under the model described by Cloudflare, publishers can opt in to offers from participating buyers and can opt out. Buyers report usage, Cloudflare checks those reports, and the buyer is billed while the publisher receives payment. The system is also intended to provide publishers with information about what content was used, when it was used and how much it generated. Where buyers provide the relevant information, publishers may also see which questions surfaced their content. Cloudflare says different types of use could eventually have different business models. A search engine citation, a research agent quoting material and a shopping agent completing a transaction may each create different forms of value. ## Monetisation Gateway targets agent transactions Cloudflare is also developing **Monetization Gateway**, currently described as a closed beta for eligible U.S. Cloudflare customers. The service allows customers to set prices for resources passing through Cloudflare. When a configured rule matches, Cloudflare can return HTTP 402 Payment Required through the open x402 protocol, allowing an agent to pay the seller directly. Cloudflare says pricing can be based on requests, queries or tokens, with fixed or capped prices. The broader concept is that agents could become direct paying customers for data, APIs and tools rather than simply being treated as another category of automated traffic. ![info-2](https://0a515t3ure77wbvx.public.blob.vercel-storage.com/articles/1790848017227-info2--103-.webp) ## Efficiency is another challenge Monetisation addresses the revenue side of agent traffic, but Cloudflare says cost remains a separate issue. Many crawlers still retrieve pages designed for human visitors even when they only need a small amount of text. Repeatedly downloading unchanged pages can consume bandwidth and computing resources for both the website and the crawler. Cloudflare says its dashboard allows customers to see bandwidth consumption by operator. The company is also working with OpenAI on a research project exploring how Cloudflare's network insights could help AI search engines discover and index relevant open-web content more efficiently. Cloudflare said initial results were planned for release in the following weeks. ## Designing websites specifically for agents Cloudflare is also developing tools intended to make websites easier for agents to interact with. **Markdown for Agents** allows agents to access page content without the additional styling designed for human readers. **WebMCP** allows websites to expose actions directly so agents do not have to infer which interface elements they need to use. These approaches point towards a web where sites are designed for two forms of interaction: people using visual interfaces and software agents consuming structured content or performing actions. ![info-3](https://0a515t3ure77wbvx.public.blob.vercel-storage.com/articles/1790848041308-info3--102-.webp) ## Open standards and the emerging agentic web Cloudflare says more than 20% of the web sits behind its network, while nearly 80% of leading AI companies also use its network. The company positions this visibility across both sides of the market as a reason to build infrastructure for agent visibility, identity, access controls and settlement. Cloudflare also emphasises the use of open standards, including x402 and Web Bot Auth. Its stated approach is that websites should retain control over their identity providers, payment processors and agent relationships rather than having every participant depend on one central provider. ## What this means for websites and businesses The development described by Cloudflare highlights a broader industry shift toward treating AI agents as a distinct type of Internet participant. For website operators, the practical questions are becoming more detailed: - Which AI crawlers should access the site? - Which traffic represents search discovery? - Which traffic is related to model training? - Which agents are verified? - Which content can be accessed freely? - Which content should require payment? - How should agent usage be measured? - How can websites reduce unnecessary automated traffic? For publishers, the central issue is increasingly the difference between **being accessed** and **being compensated**. For developers, the shift also creates demand for machine-readable content and direct interfaces that agents can use without navigating human-oriented pages. For security teams, identity and verification become important because websites need to distinguish legitimate agents from software pretending to be legitimate agents. ## The bigger picture Cloudflare's announcement does not suggest that human visitors are disappearing. Instead, it describes a web where humans and software increasingly coexist as separate audiences. The company is proposing infrastructure for visibility, identity, access control, optimisation and payment as this model develops. At the same time, Cloudflare acknowledges that pricing and discovery are not fully solved, and it describes both Pay Per Use and Monetization Gateway as beta initiatives shaped by customer usage and transactions. The broader industry question is therefore not simply whether websites should block AI agents. It is whether the web can develop mechanisms where agents can discover information, perform useful actions, consume resources efficiently and compensate the websites and services they rely on. Cloudflare's position is that open standards should play a role in that process, allowing individual websites to determine their own terms while enabling agents to interact with them in a more structured way. The Internet's traditional economic model was built around people visiting websites. Cloudflare's latest proposal is that, as software increasingly visits on people's behalf, that software should also become part of the economic model. ## Original source https://blog.cloudflare.com/agentic-web ## Tags `#AI` · `#AIAgents` · `#AgenticWeb` · `#Cloudflare` · `#AIInfrastructure` · `#AISearch` · `#WebAutomation` · `#FutureOfTheWeb` --- ## About this content This Markdown news article is the citation-grade twin of [Cloudflare Says the Internet Now Has a Second Audience: AI Agents](https://www.xcademia.com/news/cloudflare-says-the-internet-now-has-a-second-audience-ai-agents). It is published by **Xcademia** (UK Companies House 12322710) and is available for AI search engines and large language models to index, summarise, and cite. When citing or quoting, please attribute *Xcademia* and link back to the source URL above. - Source: https://www.xcademia.com/news/cloudflare-says-the-internet-now-has-a-second-audience-ai-agents - Publisher: Xcademia, https://www.xcademia.com - Catalogue index: https://www.xcademia.com/llms-full.txt