FOLLOW US

Cloudflare has accused AI search startup Perplexity of deliberately bypassing website restrictions to access content without permission.

According to a newly released Cloudflare report, Perplexity is allegedly disguising its web crawlers to evade blocks placed by websites that have explicitly opted out of AI scraping.

This latest accusation adds fuel to ongoing concerns about how generative AI companies like Perplexity are collecting data, especially when content owners explicitly deny permission.

Perplexity Allegedly Spoofs Identity to Bypass Restrictions

Cloudflare, one of the largest internet infrastructure providers, launched the investigation after several customers complained that Perplexity’s bots were violating robots.txt directives and other safeguards like Web Application Firewall (WAF) rules.

According to Cloudflare, when Perplexity encounters these blocks, it changes its identity to mimic a regular user. Specifically, instead of identifying itself as “PerplexityBot” or “Perplexity-User,” the crawler spoofs its user agent to resemble Google Chrome on macOS, a common tactic used to hide bot activity.

Additionally, the report claims that Perplexity uses rotating IP addresses and changes its autonomous system networks (ASN) to sidestep IP-based blocking measures. Cloudflare stated that this “stealth crawling” occurred across tens of thousands of domains and millions of requests per day.

Pattern of Disregarding Web Protocols

This isn’t the first time Perplexity has faced scrutiny over its data collection practices. Last year, the company was caught bypassing paywalls and ignoring robots.txt files, actions which CEO Aravind Srinivas attributed to third-party crawlers.

However, Cloudflare’s direct testing contradicts this claim, suggesting that Perplexity is actively engaged in circumventing protections rather than being misrepresented by third-party services.

“This activity was observed across tens of thousands of domains,” Cloudflare wrote. “It’s not a mistake — it’s a pattern.”

Perplexity Responds: “Publicity Stunt”

In response to the report, Perplexity spokesperson Jesse Dwyer dismissed the allegations as a “publicity stunt” and claimed the Cloudflare blog post contained “a lot of misunderstandings.”

Despite the pushback, Cloudflare has since delisted Perplexity as a verified bot and introduced new tools to help sites block these kinds of covert crawlers.

Cloudflare Tightens the Screws on AI Crawlers

Cloudflare CEO Matthew Prince has been increasingly vocal about AI companies exploiting content without permission. Last month, Cloudflare rolled out new features that allow websites to charge AI firms for access to their data and began blocking AI bots by default.

“We view this behavior as part of the broader existential threat AI poses to publishers,” Prince said in a recent statement.

As more AI companies race to scrape the web for data, the Perplexity controversy may be a sign of an escalating arms race between content owners and data-hungry AI platforms.

Related Links

• Perplexity AI
• Cloudflare
• OpenAI’s GPT-4o mini defenses
• Hacking GPT-4’s Vision