Despite the pushback, Cloudflare has since delisted Perplexity as a verified bot and introduced new tools to help sites block these kinds of covert crawlers.
Cloudflare Tightens the Screws on AI Crawlers
Cloudflare CEO Matthew Prince has been increasingly vocal about AI companies exploiting content without permission. Last month, Cloudflare rolled out new features that allow websites to charge AI firms for access to their data and began blocking AI bots by default.
“We view this behavior as part of the broader existential threat AI poses to publishers,” Prince said in a recent statement.
As more AI companies race to scrape the web for data, the Perplexity controversy may be a sign of an escalating arms race between content owners and data-hungry AI platforms.
Related Links
