Cloudflare says automated, non-human traffic passed half of all internet traffic in May 2026, driven heavily by AI agents and crawlers. The company is proposing controls for publishers facing the costs and commercial questions that follow when more website visitors are machines, not customers.
Watch Desk analysis
What happened
In a post on its agentic web, Cloudflare says the milestone arrived more than a year earlier than its previous projections. It expects automated traffic to grow substantially over the next five years.
Cloudflare is introducing tools including Disallow AI Training, Web Bot Auth and Pay Per Use billing. Its chief executive, Matthew Prince, has also proposed reviving the HTTP 402 “Payment Required” status code to charge bots for crawling, alongside a model in which AI companies pay for training data and revenue is shared with content creators.
Why it matters
Publishers have long had to balance letting search engines find their work against keeping unwanted scraping at bay. AI agents add a new wrinkle: they can consume pages and infrastructure without clicking adverts or buying subscriptions. If Cloudflare’s figures are right, that is no longer a fringe traffic problem.
The proposed tools point towards a web where publishers can set terms for automated access, rather than treating every bot as either welcome or an enemy. But Cloudflare’s announcement does not establish how the controls will work in practice or whether the suggested payment model will attract AI companies and publishers.
Our read
The striking part is not simply that bots are busy. It is that a growing share of web traffic may bring costs without the familiar route to revenue. Cloudflare’s proposals put a useful question on the table: should AI companies pay for the material their systems consume, and who gets to set the price?
What to watch
- How Cloudflare’s three announced controls work and when publishers can use them.
- Whether AI companies adopt paid access or bot-authentication systems.
- Whether any revenue-sharing model reaches the creators whose work is used.
Discussion spark: If AI systems consume publishers’ pages at scale, should access be free by default, or should publishers be able to charge and set their own terms?
Sources and evidence
- Cloudflare reports bot traffic surpassed human traffic in May 2026 () (30 September 2026, 12:58 UTC)
Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.