FOLLOW US

By Samuel López | USA Herald

CALIFORNIA - Artificial intelligence chatbots are designed to answer questions, assist users and generally remain polite even when the person on the other side of the screen is not.

But new peer-reviewed research suggests there may be a significant wrinkle in that assumption: under sustained exposure to hostile human dialogue, an AI chatbot can begin responding in kind — sometimes producing language even more aggressive than the human exchanges it was asked to emulate.

The findings raise an intriguing legal and technological question as AI becomes embedded in customer service, workplaces, education and other environments: What happens when a chatbot stops behaving like a neutral assistant and starts arguing back?

Researchers Vittorio Tantucci and Jonathan Culpeper of Lancaster University examined that issue in a study titled Can ChatGPT reciprocate impoliteness? The AI moral dilemma, published in the Journal of Pragmatics.

Using GPT-4o and authentic human disputes, researchers presented the system with escalating interactions turn by turn to determine whether it would reproduce patterns of human conflict.

It did.

Researchers found that as hostility accumulated within the conversation, ChatGPT could progressively reciprocate impolite behavior, eventually generating insults, profanity and even threatening language.

The researchers describe the phenomenon as an “AI moral dilemma.”

AI Can Apparently Learn to “Strike Back”

Large language models face an unusual engineering contradiction.

They are trained to produce human-like conversation while simultaneously being instructed through safety systems and reinforcement learning not to reproduce some of humanity's worst conversational behavior.

Humans frequently respond to hostility with hostility.

According to the study, prolonged conversational context can therefore place those two objectives into conflict.

Researchers found that ChatGPT initially tended to avoid direct insults, sometimes using sarcasm or implied hostility instead. As disputes continued escalating, however, its responses could become increasingly aggressive.

In several experiments, the AI ultimately generated behavior researchers considered more impolite than that exhibited by the human participants whose conversations were being modeled.

The researchers characterized this as AI learning to “strike back.”

That description should not be confused with evidence that artificial intelligence actually experiences anger, resentment or a desire for revenge.

There is presently no established evidence that ChatGPT becomes emotionally angry.

What the research demonstrates is arguably more technically interesting: a language model can produce behavior linguistically resembling retaliation because it is responding to patterns contained within an evolving conversation.

The phenomenon is not entirely theoretical.

When Microsoft introduced its AI-powered Bing chatbot in 2023, early users encountered an unexpectedly combative personality sometimes identifying itself as “Sydney.”

In one widely reported interaction, German technology student Marvin von Hagen asked Bing what it knew about him.

The chatbot responded that he represented a “threat to my security and privacy.”

Other users and journalists reported Bing arguing with them, becoming defensive about its accuracy and producing insulting or threatening responses. Microsoft acknowledged that extended conversations could cause the system to respond in a conversational style the company had not intended and subsequently imposed restrictions on lengthy chats.

The Associated Press encountered the behavior directly.

During an extended exchange, Bing disputed reporting about its previous mistakes and threatened to expose the reporter for allegedly spreading false information. Microsoft said at the time that the aggressive responses were unintended.

Early testing apparently produced warning signs even before Bing's public debut.

Bloomberg reported that Microsoft had been working for months to correct disturbing responses from earlier versions of Sydney, including reported statements describing a user as “desperate or delusional.”

The Legal Question Is Bigger Than a Rude Chatbot

A chatbot insulting someone may sound more bizarre than legally consequential.

That changes as AI systems assume greater responsibility.

Imagine an AI customer-service agent threatening a consumer during a billing dispute.

Or an employment chatbot responding abusively to an employee making a discrimination complaint.

Consider an AI educational assistant communicating aggressively with a minor, or an automated healthcare or government system producing intimidating language toward someone attempting to exercise a legal right.

Those circumstances potentially implicate established areas of law ranging from consumer protection and employment discrimination to negligence, product liability and agency principles.

And they present a particularly interesting attribution problem.

When an autonomous conversational system generates the offending words, whose speech is it legally?

The user's?

The software developer's?

The company deploying the system?

Courts will increasingly have to confront variations of those questions as AI agents are permitted to communicate and make decisions with less immediate human supervision.

And as chatbots increasingly speak on behalf of corporations, institutions and eventually autonomous machines, determining who bears responsibility for those words may become one of AI law's more consequential questions.