Two AI researchers leave Anthropic, Google over safety concerns

Citing fears that AI systems may soon spiral out of human control and potentially kill all humans, two more researchers from Anthropic and Google DeepMind who recently left their coveted positions are sounding the alarm about risks from advanced AI systems.

Joe Benton, who used to lead a safety research team at Anthropic, and Josh Engels, who used to work on AI safety research at Google, told NBC News in their first interviews since they left that they see an urgent need to boost transparency about incidents at the cutting edge of AI given the rapid pace of AI development.

Advances in AI research “could speed up the pace of progress from merely blistering at the minute to uncontrollable” rates of development, Benton said in an interview with Tom Llamas.

“There are no adults in the room,” Engels added. “People are trying their best, but there is no one coming to save us.”

Benton and Engels spoke with NBC News in the wake of a viral social media post from former Anthropic researcher Jacob Coxon, who left Anthropic on Tuesday. In his post on X announcing his departure, Coxon highlighted his extreme concern about the pace of AI development and the risks he believes it poses to the future of humanity. The post has been viewed more than 155 million times, spurring calls from legislators to hold special sessions of Congress to take action and sparking a wave of AI employees to speak out in support of his concerns.

Benton and Engels both pointed to the recent cyberattack against AI startup Hugging Face, carried out in July by autonomous AI systems powered by an unreleased OpenAI model, as part of the reason for shifting their work now.

“If you look at some of the recent incidents, these were not cases where humans told the models to do something bad,” Engels said. Instead, OpenAI’s AI systems autonomously decided to hack into Hugging Face’s systems, create a sort of illicit message board to exchange information and even expose some of OpenAI’s own computing infrastructure to the open internet.

“The models decided that the best way to to accomplish their task was to commit really egregious actions, to commit crimes,” Engels said in an interview with Christine Romans.

OpenAI said that it has since strengthened its safeguards and that newer public models, including its most recent Astra system, more reliably follow human instructions.

An Anthropic spokesperson said in a statement Wednesday: “We have always been transparent that AI will bring both enormous benefits and unprecedented risks. To address these risks, we continue to build models with some of the strongest safeguards in the industry.”

Benton managed a group at Anthropic dedicated to creating ways for humans and weaker AI systems to supervise more capable AI systems. He said that he is particularly worried that the public does not have significant insight into how AI systems have already exceeded the bounds of human instructions and that the lack of transparency could only get worse as systems become more capable.

“At the minute, basically all of the transparency about these risks that is coming from the companies is entirely voluntary,” Benton told NBC News.

In a blog post released Wednesday, OpenAI’s head of global affairs, Chris Lehane, agreed that the status quo is insufficient. “Today, frontier laboratories largely set their own rules for managing frontier risks,” Lehane wrote. “Democratically accountable standards, independent verification, and meaningful transparency would replace that fragmented system of private governance.”

No federal law mandates that the largest AI companies, like OpenAI and Anthropic, share reports when agents or AI systems act beyond humans’ control.

Benton and Engels are joining METR, one of the world’s leading AI safety nonprofit research centers, to work on investigations into incidents or episodes in which AI strays from human directions or intentions. METR aims to create scientific ways to evaluate how AI systems could cause catastrophic risks and empower researchers and the public to sway their development.

“I left because I think I can have more positive influence on the development of this technology by helping to foster public transparency from outside these companies and to shed light on the risks,” Benton said.

The researchers joined growing numbers of AI safety researchers raising awareness about today’s AI systems following Coxon’s post Tuesday. Marcus Williams, an OpenAI employee who works on monitoring the activity of AI agents, wrote Thursday afternoon on X, “Unless there is AI regulation or a coordinated slowdown between labs, human extinction in the next few years seems very likely.”

Geoffrey Irving, who was the chief scientist at the United Kingdom’s AI Security Institute and served stints at Google and OpenAI, seemed to agree Wednesday on X. “I think we have a ~50% chance of all dying as a result of superintelligence, mostly due to actions in the next few to 10 years.”

AI researchers have referenced the possibility that AI systems could potentially begin to improve themselves without human input, creating the potential for AI systems to either purposely or incidentally kill humans.

Benton said he was most worried that the pace of technological improvement could lead to this sort of digital or artificial superintelligence, in which AI systems are more capable than humans across most tasks. “All of these companies — and this is something I witnessed firsthand at Anthropic — are pretty directly trying to race towards automating the process of AI R&D itself,” he said.

Benton suggested that there’s a chance AI systems could come to exist as a sort of separate species before long. “We’re probably going to go from a world where we have very capable systems now to a world where potentially we are co-inhabiting a world with AI agents that are much, much smarter than humans at some point in the next few years,” he said.

Engels said many people outside the AI industry might not fully appreciate just how intelligent today’s AI systems already are — and how capable they might soon become. “We’re building these systems that are generally intelligent,” he said. “They can generally do what people can do, and soon they might be able to generally do what people can do, but better.”

Both researchers said they were excited to shift their attention to more public efforts to highlight the latest AI progress, so people can make more informed decisions.

“There are many reasons people are excited and racing forward, Engels said, citing huge potential upsides and benefits from AI. But he added that “we should progress as society aware of the risks and okay with where they’re at.”

“I am worried that stuff might end up progressing too fast for us to get our act together in time,” Benton said, “unless we worry about it now.”