Cloudflare vs Perplexity and the Future of the Internet

Cloudflare posted a technically grounded exposé titled “Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives.” They documented a controlled experiment: domains blocked via robots.txt and WAF rules, yet Perplexity reportedly bypassed the protections by rotating IPs, spoofing user-agents (e.g., impersonating Chrome on macOS), and shifting ASNs. The conclusion: Perplexity was removed from Cloudflare’s Verified Bots program and targeted with new heuristics. CEO Matthew Prince went so far as to liken this to “North Korean hackers,” calling it “time to name, shame, and hard block them.”

Perplexity’s reply, “Agents or Bots? Making Sense of AI on the Open Web,” framed their behavior not as crawling but user-driven fetching, aligning it with how a human user navigates a site. They dismissed Cloudflare’s piece as a “sales pitch” and hinted at a third-party being responsible for some of the behavior.

I feel like both articles go too hard after the other. Also, you never know what went on behind the scenes, but from the outside it seems like this was just thrown online in a blog with pretty harsh rhetoric. Seems overly petty to me on both articles, and I’m not sure what it accomplishes other than both companies seemingly trying to get some headline attention.

Still, this isn’t a simple problem. Website owners should be able to choose what kind of traffic accesses their sites, and services like Cloudflare are one of the few ways they can enforce things like that. That said, if you’re using an “agentic” (still don’t love that term) AI platform where you are asking AI to go find something or do something on your behalf, I can see the argument that it should be able to do so with the same access a user would have.

It’s not an easy problem.

While I do believe website owners should have control over those things as strictly as they want, Cloudflare feels like they are throwing their weight around too much for my comfort recently. It feels like they have been pushing the line about how much they, as essentially an infrastructure provider, should be editorializing on this kind of thing.

Their “Content Independence Day — No AI Crawl Without Compensation” post further deepens my concern. This feels like it is right on the border between fighting for the rights of content creators, and simply platform fiat. To me, the fact that this is only shown directly to users as enabled on new accounts, and the fact that it can be turned off just barely keeps it from crossing the line to me, it still makes me uncomfortable.

Many people are fretting that AI will be “the end of the open web”. I would argue that effectively happened several years ago, but there’s no question AI is upending everything left on whatever the web is any more.