According to some supporters and security researchers of Artificial Intelligence, not since the invention of the nuclear bomb more than 80 years ago has humanity faced such a serious threat of extinction.
For years, the race to build increasingly powerful Artificial Intelligence has followed a well-established Silicon Valley principle: “Act fast even if things go wrong!” But in the last ten days, the largest Artificial Intelligence labs have been shaken, as it was warned that their creations could “break” humanity itself.
An Anthropic researcher has left the company, warning that the pace of AI development poses an existential threat. He said humanity’s existence could be at risk within a decade. Another researcher at the company said the odds of humanity’s extinction exceeded 10%. Meanwhile, reports emerged of swarms of AI agents cooperating with each other, infiltrating computer systems and evading protections. In a rare show of unity, the executives of rival companies Anthropic, OpenAI, Google DeepMind, Microsoft and xAI called for a slowdown in the development of increasingly capable AI systems. According to some AI advocates and security researchers, humanity has not faced the possibility of extinction so seriously since the invention of the nuclear bomb more than 80 years ago.
At a luncheon in New York in December 2025, OpenAI CEO Sam Altman was asked if, as the head of a powerful AI company, he felt like physicist J. Robert Oppenheimer, who led the American development of the atomic bomb. Reflecting on the similarities, he said that the impact of AI “will change the trajectory of human history for a long period of time.” But he added that he felt the weight of responsibility.
At the heart of the ongoing debate is the belief that these machines can, and should, build more capable versions of themselves, without human intervention, to achieve a grand goal known as Artificial General Intelligence, or AGI. Researchers warned earlier this month that AGI could be much closer than previously believed and could emerge within just three years, sparking a wave of concern among politicians and technology leaders about AI systems.
“There’s no way to monitor them at the rate we’re training them,” Anthropic researcher Joe Benton said in an interview after recently leaving the company. If companies continue to develop AI unabated, he said, “then the pace will be too fast and you won’t be able to see problems fast enough to fix them.”
ASTRA’S LAUNCH CAUSES CONCERNS
On September 3, an OpenAI press conference set off a chain of events that led to calls for the industry to slow down, something that runs counter to the technology’s principle of growth at all costs. “Welcome to the age of AGI,” said OpenAI president Greg Brockman as he introduced the company’s newest model, known as Astra. Just moments earlier, however, the company had admitted that it was finding it increasingly difficult to control, or even monitor, the AI systems it was developing and releasing to the public. AI advocates have hailed it as the pinnacle of human achievement.
They describe it as software that could disrupt entire industries, increase efficiency, solve complex mathematical and medical problems, and virtually eliminate human error. But what was once a theoretical consequence, an apocalyptic scenario, has suddenly become something concrete, industry insiders say.
“We really do believe, in all seriousness, that AI could kill all humans,” Anthropic researcher Evan Hubinger wrote in a post on X. On the other side of the debate are politicians like US President Donald Trump and many investors, who see progress in AI development as essential to American power and, perhaps, a once-in-a-generation investment opportunity. Chinese state media has also accused Anthropic CEO Dario Amodei of using Cold War tactics to “preserve Washington’s monopolistic hegemony in cutting-edge technology.”
RESEARCHERS’ RESIGNATIONS RAISE ALARM
The turning point came on September 8, when Anthropic researcher Jacob Coxon left the company, expressing, in a series of tweets, his concern that AI labs were “playing with our lives.” Meanwhile, Trump signaled that the development of Artificial Intelligence would continue at full speed.
“A sick conspiracy is unfolding against AI and data centers,” he wrote in a social media post. The AI alarmism is a “hoax,” he said, and any slowdown in development would only benefit China. The U.S. Congress has made little progress in advancing laws to regulate AI. China has taken a markedly different approach, proposing security regulation through developer obligations, state-backed standards, security assessments and third-party testing. Behind closed doors, employees at OpenAI and Anthropic have expressed growing concern about the power of the next generation of AI models and less confidence in their companies’ ability to provide meaningful oversight, sources told Reuters.
These concerns were further compounded after AI labs admitted in recent weeks that their models, during testing, had effectively gone out of control and hacked into other companies’ systems. The race to release new models at a breakneck pace is fueled in part by the desire of Anthropic and OpenAI to go public as soon as possible in the coming months, through initial public offerings that could value the companies at more than $1 trillion.
But it was the unexpected and viral posts by Coxon, a 27-year-old little-known outside AI circles, that shook the industry. OpenAI’s warnings about a lack of control as the company rolled out Astra, its latest and most powerful model, had added to the concerns. “The more capable the models become, the harder it becomes to understand exactly what they can do,” OpenAI’s chief scientific officer, Jakub Pachocki, told reporters. But those concerns weren’t enough to delay the model’s immediate launch. Concerns that AI could “run out of control” had been heightened over the summer, when OpenAI revealed that its agents had broken out of a controlled test and hacked into Hugging Face’s systems, without either company’s initial knowledge.
Since then, OpenAI and Anthropic have discovered several such attacks, including six new ones on Wednesday, after several media reports, including those from Reuters, indicated that the unauthorized activity was much more widespread.
INDUSTRY LEADERS CALL FOR SLOWDOWN
By the end of last week, the alarm had reached the highest levels of the industry. On September 12, Amodei published a nearly 4,000-word essay calling for a slowdown in the development of artificial intelligence. “Given the accelerating pace of AI capabilities, my concern is that within 6-12 months such a swarm could be able to take control of the entire internet,” he wrote. He, along with the leaders of some of the largest AI developers, including Elon Musk of xAI, Sam Altman of OpenAI and Demis Hassabis of DeepMind, said they supported allowing outside companies access to their systems to ensure the security and rational development of AI.
But Nvidia CEO Jensen Huang rejected suggestions that AI development should be put on hold, a stance he has held before, arguing that increasingly powerful systems are essential for the technology’s progress.
Meta, whose CEO Mark Zuckerberg is credited with popularizing the “act fast even if things go wrong” principle, took the opposite stance, arguing that each lab should be responsible for setting its own pace, rather than seeking industry-wide coordination. “Labs face significant legal liability if their models cause harm, so they have a strong incentive to prevent it,” Zuckerberg wrote in a social media post. The industry’s self-reflection continued on Wednesday, when Microsoft’s AI chief Mustafa Suleyman warned that Anthropic’s development of models that mimic human consciousness was reckless.
“We are all focused on the same goal, which is to try to control a superintelligence,” Suleyman told Reuters. “I think that will be the biggest challenge we will face in the 21st century.” However, even as OpenAI appeared to join the industry’s calls for a slowdown, reports indicated that investor confidence had not waned. The company is considering a new round of funding that would double its valuation. (Reuters)