FOLLOW US

America August 9, 2023 3 mins read

OpenAI Web Crawler GPTBot Privacy Controversy, Here’s How to Block It

America ı By Jackie Allen

0 Comments

GPTBot

Artificial intelligence (AI) model GPT-4, through OpenAI, has unveiled GPTBot, a web crawler designed to scour the Internet for data to improve AI performance. While this development holds promise for refining AI accuracy and safety, it also raises significant privacy concerns and prompts questions about data security.

OpenAI's GPTBot functions as a program that accesses publicly available online data to enhance future iterations of AI models. This process, often referred to as web crawling or web scraping, has the potential to empower AI models with real-world information, making them more adept at understanding and generating human-like text.

Website owners who want to prevent their content from being accessed by GPTBot have two options: they can either block the bot's IP address or incorporate a disallow command into their site's code. OpenAI has provided guidance on how to implement these measures, giving web administrators a degree of control over whether their content is included in GPTBot's training dataset.

In a blog post, OpenAI highlighted that the web pages collected by GPTBot are filtered to eliminate sources requiring paywall access, those collecting personally identifiable information (PII) or containing content that violates OpenAI's policies. This approach is aimed at addressing privacy concerns that often arise when AI systems interact with sensitive data.

Open AI controversy

This launch comes amid legal battles and controversies surrounding OpenAI's data collection practices. 

AI Chatbot Security Showdown: Google's Bard and ChatGPT Vulnerable to Hackers! – USA Herald

Microsoft is all-in on its ChatGPT strategy for Bing Chatbot – USA Herald

Just weeks before the Bot was introduced, OpenAI filed a trademark application for "GPT-5" software, indicating the possibility of a forthcoming AI model. This move coincides with the ongoing lawsuits against OpenAI, including allegations of libel, copyright infringement, and a class action lawsuit regarding data confidentiality.

One lawsuit claimed that OpenAI and Microsoft amassed private information, conversations, medical data, and even details about minors from the internet without proper consent or notice to data owners. 

These accusations have prompted the Federal Trade Commission to initiate an investigation into OpenAI's compliance with consumer protection laws. This investigation aims to assess OpenAI's efforts to rectify false or personal information revealed by its AI models, including ChatGPT.

While OpenAI is offering a potential avenue for improving AI models, it isn’t clear how the company trained its previous ChatGPT versions. The lack of transparency regarding AI model development methods and the potential privacy implications of data collection remains a significant concern.

As AI evolves, it's crucial to establish robust safeguards, transparent practices, and ethical data usage to ensure both progress and privacy are maintained. And it’s important to understand that you can block it on your site.

How to Block OpenAI web crawler GPTBot

To execute the block, add GPTBot in your site’s robots.txt file and disallow it:

  • User-agent: GPTBot
  • Disallow: /

GPTBot can be identified with the following token:

  • User agent token: GPTBot
  • Full user-agent string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)

You can also customize access by adding the GPTBot Token in select parts of your site’s robot.txt file. Here’s an example:

  • User-agent: GPTBot
  • Allow: /directory-1/
  • Disallow: /directory-2/

 

 

Previous Article

Pattern of Abuse: New York Doctor Accused of Drugging and Raping Patients 

Read More
3480 Posts

Jackie Allen

Jackie is a freelance journalist and technology geek. She worked as a telecom project director for AT&T and BellSouth. Before joining the USA Herald she has written books, articles, blogs and whitepapers. Her clients include Samsung and other technology companies.

Discussion

No comments yet. Be the first to join the discussion!

Don’t Miss It
America November 19, 2026
GTA VI launch spotlights systemic cyber…

The GTA VI insurance risks are not about the game…

By – Tyler Brooks
America November 19, 2026
Walmart raises its outlook for the…

Walmart raises outlook for the third time this year, but…

By – Rachel Moore
America November 19, 2026
Bed bugs force FEMA staff to…

The FEMA bed bugs story adds an unlikely twist to…

By – Rachel Moore
America November 19, 2026
Anthropic shares jump 21% after $107.3B…

The Anthropic $107.3B IPO opened with a bang on Thursday.…

By – Rachel Moore
America November 19, 2026
Binance Wins $6.5B Freezing Order Over…

The Binance $6.5B freezing order may be the biggest legal…

By – Rachel Moore
America November 19, 2026
Polaris $1M Data Privacy Settlements: California…

The Polaris $1M data privacy settlements show what happens when…

By – Rachel Moore
America November 19, 2026
 Anti-Death-Penalty Groups Intensify Fight Against Capital…

  The United States recorded 37 executions in 2026 as…

By – Jackie Allen
America November 19, 2026
43 Soldiers Unlock a GTA 6…

FORT STEWART, Ga. — The 9th Brigade Engineer Battalion offered…

By – Michallie K Harrison
America November 19, 2026
Lawmakers Press Lutnick Family Over Nvidia…

By Samuel Lopez – USA Herald | November 19, 2026…

By – Samuel Lopez
America November 19, 2026
Palmer Luckey Warns AI Hackers Are…

NEW YORK — Palmer Luckey says artificial intelligence is giving…

By – Michallie K Harrison
America November 19, 2026
Iowa Schools Cancel Classes as Army…

MIDDLETOWN, Iowa — Schools in southeast Iowa have canceled Friday…

By – Michallie K Harrison
America November 19, 2026
“A Nuclear War Cannot Be Won”:…

Trump, Xi and Putin signed a four-point Shenzhen declaration on…

By – Derek Johnson
America November 18, 2026
Government Investigators Find AI Platforms Generated…

Federal investigators found that four major artificial intelligence platforms could…

By – Jackie Allen
America November 18, 2026
Yellowstone’ Star Luke Grimes Opens Up…

Key Takeaways Luke Grimes admits he was worried about continuing…

By – Jackie Allen
America November 18, 2026
Florida Flood Threat: Record Downpours Swamp…

The Florida flood threat turned real overnight. Torrential rainfall and…

By – Tyler Brooks
America November 18, 2026
Anthropic BIPA Suit: Illinois User Says…

The Anthropic BIPA suit alleges that a routine identity check…

By – Tyler Brooks
America November 18, 2026
Oregon Accuses ChatGPT Maker of Endangering…

The OpenAI $2B lawsuit has put one of the world’s…

By – Tyler Brooks
America November 18, 2026
Is GTA 6 Out? Rockstar’s Blockbuster…

Is GTA 6 out? The answer depends on where you…

By – Tyler Brooks
Breaking News November 19, 2026
Anthropic Hits the Nasdaq Today as…

Anthropic begins trading as “ANT” on the Nasdaq after selling…

By – Derek Johnson
Business November 18, 2026
GTA VI Is Finally Here: Rockstar’s…

Grand Theft Auto VI began rolling out at midnight in…

By – Derek Johnson
America November 18, 2026
Iran Says It Targeted Kuwaiti Air…

Iran’s military said it targeted Ali Al Salem Air Base…

By – Michallie K Harrison
America November 18, 2026
China to Lift 25% Tariff on…

SHENZHEN, China — Canadian pork processors lost roughly a quarter…

By – Michallie K Harrison
America November 18, 2026
Brazil Probes Whether Vance-Linked Group Aided…

Brazilian federal police are investigating whether Rockbridge Network, a U.S.…

By – Michallie K Harrison
America November 18, 2026
13 U.S. Babies Born With Measles…

U.S. officials documented 13 babies born with measles and 112…

By – Michallie K Harrison
America November 17, 2026
Katie Couric Reveals  Memory-Loss Episode That…

Katie Couric is opening up about a recent frightening medical…

By – Jackie Allen
America November 15, 2026
Hackers Hit America’s Drinking Water Systems…

ATLANTA – Hackers have crossed a dangerous line in America’s…

By – Samuel Lopez
America November 15, 2026
America Loses Its Measles-Free Status for…

PAHO says the U.S. no longer meets the standard for…

By – Derek Johnson
America November 11, 2026
Brooklyn Building Collapse Kills 2 as…

A Brooklyn building collapse killed at least two people and…

By – Michallie K Harrison
America November 11, 2026
A Veterans Day Betrayal: House Advances…

OPINION | The Republican-led House marked Veterans Day by advancing…

By – Michallie K Harrison
America November 11, 2026
AI Godfather Warns of Chernobyl-Style Blackout…

By Samuel López | USA Herald | November 11, 2026…

By – Samuel Lopez
America November 15, 2026
Tower Hill Investment in Gator Athletics:…

The Tower Hill investment in Gator Athletics just rewrote the…

By – Tyler Brooks
America November 14, 2026
Golden State Valkyries Star Gabby Williams…

 Golden State Valkyries forward Gabby Williams believes Indiana Fever star…

By – Jackie Allen
America November 8, 2026
Dana White Reportedly Declines Trump White…

By Samuel Lopez – USA Herald | November 8, 2026…

By – Samuel Lopez
America November 7, 2026
Mike Tyson And Ric Flair Score…

Inside the $50 Million Legal Battle A federal judge rejected…

By – Samuel Lopez
America November 5, 2026
Dallas Cowboys Face  Crisis After Crushing…

By Jackie Allen | USA Herald The Dallas Cowboys are…

By – Jackie Allen
America November 4, 2026
WNBA: DeWanna Bonner Makes Emotional Admission…

ATLANTA — The WNBA Finals ended in heartbreak for the…

By – Jackie Allen

No posts found.

No posts found.

Signup for the USA Herald
exclusive Newsletter