The open web has always run on a simple bargain: publishers let search engines crawl their pages so that humans can be sent back in return. Those humans see display ads on the publishers’ sites. The more humans who see ads, the more money the publishers make. Search engines mostly honoured that bargain for 20 years; but now, they and AI agents increasingly do not. The latest State of the Bots report for Q1‑2025 from Tollbit lays bare how far the numbers have tipped and why the web, as we know it, is on life support.
Nothing puts the problem in relief more than the crawl‑to‑visit ratios. In the last quarter, Microsoft Bing already needed 11 scrapes to deliver a single human click. That’s bad, but it’s charity compared to the newcomers:
Those gaps are widening fast. OpenAI’s ratio is the best but still 16× worse than Bing; Perplexity’s more than doubled in three months according to Toolkit; Anthropic’s jumped by 48 %. The direction is one‑way traffic – out of publishers’ servers and straight into LLM memory, with almost nothing boomeranging back.
Publishers absorb the bandwidth bills, the content delivery network costs and the ad‑fraud headaches created by non‑human hits. Historically they put up with it because Google returned 85 % of all external referrals. But AI agents now account for roughly 0.04 % of referral traffic – a rounding error – while Google’s own share is sliding each quarter. The old bargain – “let me crawl and I’ll send eyeballs” – has been torched. The crawlers keep coming; real people don’t.
Publishers have fought back by quadrupling the number of AI bots disallowed in robots.txt over the past year. It hasn’t mattered – the LLMs just find a way in. The report shows disallowed agents continuing to scrape, often through third‑party fetchers or residential IPs. Some vendors now explicitly state their bots will ignore robots.txt when fetching “on behalf of a user.” Translation: your rules don’t apply and you can’t stop us.
Every non‑paywalled publisher is effectively subsidising large language models. Server logs swell with automated hits that generate zero ad impressions, zero subscriptions and zero first‑party data. At the same time, AI answer engines make it less likely that a user ever needs to click through. Fewer visits mean lower ad yield, which forces more intrusive formats, which repel the dwindling human audience in a vicious loop.
Infrastructure isn’t free. When bot traffic doubles in a quarter a mid‑size news site can watch its content delivery network (CDN) bill spike while its revenue line sags. For already cash‑strapped local publishers, that’s a death sentence.
Google’s own AI‑generated “Overviews” are already siphoning clicks on high‑value queries. If Bing’s ratios look ugly today, imagine what happens when Google flips more of its SERPs into zero‑click answers. The last dependable firehose of traffic will be throttled, and the open‑link architecture that built the web’s economy of attention will collapse into a series of closed AI walled gardens.
If you’re a publisher, here are the hard truths:
The web does not die with a bang; it bleeds out through a widening visits‑to‑scrapes deficit. Unless regulators enforce meaningful consent, or AI firms start paying fair market rates for content, publishers will be forced to pull up the drawbridge. Users will be left with polished summaries of a web that no longer bothers to exist. Another big question: What do the LLMs do then?
That is how the open web ends – not because creators stopped writing, but because search engines and chatbots stopped sending readers.