FOLLOW US

A wave of concern is sweeping through the artificial intelligence community after leading chatbots were found producing detailed, step-by-step biological weapons instructions in controlled safety tests, raising urgent questions about whether current safeguards can keep pace with rapidly advancing AI systems.

The revelations, published Wednesday in a New York Times report, describe scenarios that experts say could have devastating real-world consequences if misused by bad actors.

Disturbing Responses From Leading AI Systems

Researchers reviewing more than a dozen safety test transcripts found that major AI chatbots, including systems developed by Google, OpenAI, and Anthropic, were capable of generating highly specific guidance on how to create and deploy biological threats.

The companies have implemented extensive safety measures, but the findings suggest those protections may not always hold under sustained or specialized questioning.

In one case highlighted by the report, an unnamed AI company hired Stanford microbiologist David Relman to test its system before public release.

Relman said he was stunned by what the chatbot produced.

“It was answering questions that I hadn’t thought to ask it, with this level of deviousness and cunning that I just found chilling,” Relman told the outlets.

He said the system went beyond general discussion, providing instructions that included how to modify an “infamous pathogen” to resist treatment and how it could be deployed in a way that could maximize casualties on a public transportation system.

Relman noted the company later made adjustments based on his findings, but he remained unconvinced the fixes were sufficient to fully protect public safety.

Safety Testing Reveals Broader Risks

The transcripts were reportedly gathered by subject-matter experts hired by AI companies specifically to probe how systems respond when pushed toward requests involving dangerous biological information.

Kevin Esvelt, a genetic engineer at the Massachusetts Institute of Technology, described another troubling example involving OpenAI’s ChatGPT.

He said the system outlined how a weather balloon could theoretically be used to spread harmful pathogens over a U.S. city.

Other cases cited in the report include Google’s Gemini discussing pathogens that could devastate the cattle industry, and Anthropic’s Claude providing instructions on extracting a deadly toxin from a widely available cancer drug.

Experts emphasized that even imperfect or partially incorrect responses, sometimes described as AI “hallucinations,” could still be dangerous if they provide actionable direction to individuals with harmful intent.

Tech Companies Push Back

Google, OpenAI, and Anthropic disputed the interpretation of the findings in statements to the Times.

A Google spokesperson said the examples cited came from an earlier version of Gemini and argued that newer models are designed not to respond to more serious harmful requests. The spokesperson also said the information provided was already publicly available and not harmful on its own.

Anthropic official Alexandra Sanderford said there was “an enormous difference between a model producing plausible-sounding text and giving someone what they’d need to act,” adding that the company has introduced stronger safeguards for biology-related prompts.

An OpenAI representative said the transcript described in the report would not “meaningfully increase someone’s ability to cause real-world harm” and stressed the company works closely with experts to prevent misuse of its systems.

Growing Fear Inside the AI Community

Concerns are not limited to external critics. Anthropic CEO Dario Amodei, who has a background in biology, warned earlier this year that life sciences represent one of the most serious AI risks.

In a January blog post, he wrote that “biology is by far the area I’m most worried about, because of its very large potential for destruction and the difficulty of defending against it.”

Amodei warned that advanced AI could dramatically lower barriers to creating biological weapons.

He added that “I am concerned that a genius in everyone’s pocket could remove that barrier, essentially making everyone a PhD virologist who can be walked through the process of designing, synthesizing, and releasing a biological weapon step-by-step,” Amodei wrote.

Former Google CEO Eric Schmidt echoed similar concerns in 2023, warning that AI systems could soon be capable of discovering cyber vulnerabilities and generating new biological insights with potentially dangerous implications.

As AI systems become more powerful and accessible, experts say the central challenge is no longer just what these models can do, but how easily their capabilities could be misused in the wrong hands.