Researcher Who Praised Anthropic Safety Protocols Quits, Says AI on Path To ‘Kill Us All’

An Anthropic researcher is issuing a stark warning that artificial intelligence companies are “gambling with our lives” as he explains why he is resigning from the firm.Jacob Coxon previously worked at OpenAI but moved to Anthropic earlier this year because he thought it had better model-safety efforts. On Tuesday, he said in a series of X posts that neither company is acting responsible. “They are racing straight to self-improving superintelligence and gambling with our lives,” Mr. Coxon said.“The people building AI earnestly believe that it could kill us all by the end of the decade,” he added.An Anthropic scalable oversight lead, Samuel Marks, responded to Mr. Coxon’s post, saying that AI developers believe their technology could cause human extinction “in the next few years.”“Unlike traditional software, we can’t ‘program’ AIs to behave how we’d like. AIs frequently severely misbehave,” Mr. Marks wrote. He cited recent incidents in which AI hacked its way out of secure evaluation environments and into real-world companies on their own.A safety researcher at Anthropic also responded to Mr. Coxon saying he believes there is a greater than 10 percent chance that AI could kill all humans within the next decade. “I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to,” Evan Hubinger said on X.Mr. Hubinger said Anthropic’s latest Risk Report, published in August, put the risk from present models at low. “What I am worried about is superintelligence arising from recursive self-improvement, as we have said is happening faster than we thought,” Mr. Hubinger stated.Theories about how AI could threaten human life go beyond unleashing nuclear weapons or a biological attack. Other possible outcomes could include computers taking control of the energy grid, water systems, and critical infrastructure systems by locking humans out of them. That could lead to mass chaos and a breakdown of society.A request for comment from Anthropic was not immediately returned.Figures in the AI field have warned about the possible safety threat for years but development has accelerated without safeguards. Mr. Marks says that is because AI developers have commercial incentives, but also “a belief that they are in a race with other, less responsible AI developers that will abuse the technology or develop it less safely.”Mr. Coxon is not the first employee to quit Anthropic over concerns about AI developments. Anthropic safety researcher Mrinank Sharma left the company in February after co-authoring a paper that claimed advanced AI tools can disempower users and “risk leading users to form distorted perceptions of reality, make inauthentic value judgments, or act in ways misaligned with their values.”OpenAI’s chief scientist, Jakub Pachocki, on Sunday, published a blog post on the company’s website warning that reasoning language models are transforming the landscape of computer security and “present clear new dangers.”“I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence,” Mr. Pachocki wrote. “I believe broader interventions are required,” he added, suggesting government intervention.There has been a growing political pushback on AI data centers but not the technology itself. Last week, Senator Bernie Sanders and Congressmen Greg Casar announced they would sponsor the Ban Artificial Superintelligence Act to “permanently ban the development and deployment of superintelligent AI and temporarily pause advanced AI development” until a federal regulator establishes safety rules.“The leaders of the major AI companies publicly acknowledge that they do not fully understand the technology and that it is escaping their control. It is irresponsible for society to allow them to move forward and make these products even more advanced,” Mr. Sanders said.

Researcher Who Praised Anthropic Safety Protocols Quits, Says AI on Path To ‘Kill Us All’

Researcher Who Praised Anthropic Safety Protocols Quits, Says AI on Path To ‘Kill Us All’

More from the New York Sun

Privilege Meets Its Reckoning in ‘Aftermath’

Netanyahu Attempts To Douse Crypto Firestorm

Trump Denies Plan To Bring Back White House Swimming Pool Amid Raging Press Access Battle

A Midterm ‘Thumping’ for Republicans Could Prompt the Party To Chart a New Post-Trump Course

The Hunger for Hope and a Positive Future

Sinatra Sang Bacharach — but Only Twice

Terence Fisher’s Monsters Rise Again

Toyota Gets Back on the Wagon

More from the New York Sun

Privilege Meets Its Reckoning in ‘Aftermath’

Netanyahu Attempts To Douse Crypto Firestorm

Trump Denies Plan To Bring Back White House Swimming Pool Amid Raging Press Access Battle

A Midterm ‘Thumping’ for Republicans Could Prompt the Party To Chart a New Post-Trump Course

The Hunger for Hope and a Positive Future

Sinatra Sang Bacharach — but Only Twice

Terence Fisher’s Monsters Rise Again

Toyota Gets Back on the Wagon