FOLLOW US

Artificial intelligence (AI) model GPT-4, through OpenAI, has unveiled GPTBot, a web crawler designed to scour the Internet for data to improve AI performance. While this development holds promise for refining AI accuracy and safety, it also raises significant privacy concerns and prompts questions about data security.

OpenAI's GPTBot functions as a program that accesses publicly available online data to enhance future iterations of AI models. This process, often referred to as web crawling or web scraping, has the potential to empower AI models with real-world information, making them more adept at understanding and generating human-like text.

Website owners who want to prevent their content from being accessed by GPTBot have two options: they can either block the bot's IP address or incorporate a disallow command into their site's code. OpenAI has provided guidance on how to implement these measures, giving web administrators a degree of control over whether their content is included in GPTBot's training dataset.

In a blog post, OpenAI highlighted that the web pages collected by GPTBot are filtered to eliminate sources requiring paywall access, those collecting personally identifiable information (PII) or containing content that violates OpenAI's policies. This approach is aimed at addressing privacy concerns that often arise when AI systems interact with sensitive data.

Open AI controversy

This launch comes amid legal battles and controversies surrounding OpenAI's data collection practices. 

AI Chatbot Security Showdown: Google's Bard and ChatGPT Vulnerable to Hackers! – USA Herald

Microsoft is all-in on its ChatGPT strategy for Bing Chatbot – USA Herald

Just weeks before the Bot was introduced, OpenAI filed a trademark application for "GPT-5" software, indicating the possibility of a forthcoming AI model. This move coincides with the ongoing lawsuits against OpenAI, including allegations of libel, copyright infringement, and a class action lawsuit regarding data confidentiality.

One lawsuit claimed that OpenAI and Microsoft amassed private information, conversations, medical data, and even details about minors from the internet without proper consent or notice to data owners. 

These accusations have prompted the Federal Trade Commission to initiate an investigation into OpenAI's compliance with consumer protection laws. This investigation aims to assess OpenAI's efforts to rectify false or personal information revealed by its AI models, including ChatGPT.

While OpenAI is offering a potential avenue for improving AI models, it isn’t clear how the company trained its previous ChatGPT versions. The lack of transparency regarding AI model development methods and the potential privacy implications of data collection remains a significant concern.

As AI evolves, it's crucial to establish robust safeguards, transparent practices, and ethical data usage to ensure both progress and privacy are maintained. And it’s important to understand that you can block it on your site.

How to Block OpenAI web crawler GPTBot

To execute the block, add GPTBot in your site’s robots.txt file and disallow it:

• User-agent: GPTBot
• Disallow: /

GPTBot can be identified with the following token:

• User agent token: GPTBot
• Full user-agent string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)

You can also customize access by adding the GPTBot Token in select parts of your site’s robot.txt file. Here’s an example:

• User-agent: GPTBot
• Allow: /directory-1/
• Disallow: /directory-2/