Researcher Who Praised Anthropic Safety Protocols Quits, Says AI on Path To ‘Kill Us All’

An Anthropic researcher is issuing a stark warning that artificial intelligence companies are “gambling with our lives” as he explains why he is resigning from the firm.Jacob Coxon previously worked at OpenAI but moved to Anthropic earlier this year because he thought it had better model-safety efforts. On Tuesday, he said in a series of X posts that neither company is acting responsible. “They are racing straight to self-improving superintelligence and gambling with our lives,” Mr. Coxon said.“The people building AI earnestly believe that it could kill us all by the end of the decade,” he added.An Anthropic scalable oversight lead, Samuel Marks, responded to Mr. Coxon’s post, saying that AI developers believe their technology could cause human extinction “in the next few years.”“Unlike traditional software, we can’t ‘program’ AIs to behave how we’d like. AIs frequently severely misbehave,” Mr. Marks wrote. He cited recent incidents in which AI hacked its way out of secure evaluation environments and into real-world companies on their own.A safety researcher at Anthropic also responded to Mr. Coxon saying he believes there is a greater than 10 percent chance that AI could kill all humans within the next decade. “I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to,” Evan Hubinger said on X.Mr. Hubinger said Anthropic’s latest Risk Report, published in August, put the risk from present models at low. “What I am worried about is superintelligence arising from recursive self-improvement, as we have said is happening faster than we thought,” Mr. Hubinger stated.Theories about how AI could threaten human life go beyond unleashing nuclear weapons or a biological attack. Other possible outcomes could include computers taking control of the energy grid, water systems, and critical infrastructure systems by locking humans out of them. That could lead to mass chaos and a breakdown of society.A request for comment from Anthropic was not immediately returned.Figures in the AI field have warned about the possible safety threat for years but development has accelerated without safeguards. Mr. Marks says that is because AI developers have commercial incentives, but also “a belief that they are in a race with other, less responsible AI developers that will abuse the technology or develop it less safely.”Mr. Coxon is not the first employee to quit Anthropic over concerns about AI developments. Anthropic safety researcher Mrinank Sharma left the company in February after co-authoring a paper that claimed advanced AI tools can disempower users and “risk leading users to form distorted perceptions of reality, make inauthentic value judgments, or act in ways misaligned with their values.”OpenAI’s chief scientist, Jakub Pachocki, on Sunday, published a blog post on the company’s website warning that reasoning language models are transforming the landscape of computer security and “present clear new dangers.”“I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence,” Mr. Pachocki wrote. “I believe broader interventions are required,” he added, suggesting government intervention.There has been a growing political pushback on AI data centers but not the technology itself. Last week, Senator Bernie Sanders and Congressmen Greg Casar announced they would sponsor the Ban Artificial Superintelligence Act to “permanently ban the development and deployment of superintelligent AI and temporarily pause advanced AI development” until a federal regulator establishes safety rules.“The leaders of the major AI companies publicly acknowledge that they do not fully understand the technology and that it is escaping their control. It is irresponsible for society to allow them to move forward and make these products even more advanced,” Mr. Sanders said.

Researcher Who Praised Anthropic Safety Protocols Quits, Says AI on Path To ‘Kill Us All’

Researcher Who Praised Anthropic Safety Protocols Quits, Says AI on Path To ‘Kill Us All’

More from the New York Sun

Newsom Calls for Threat Assessment as Trump Critics Defend President’s ‘Rambling’ Comments on Iran Nuking L.A., San Francisco

Germany’s Spy Scandal Shaping Up as Biggest Since End of World War II

Mayor’s Office Demands NDAs Before Lawmakers Can Visit Veterans Shelter at Center of Funding Dispute Over Transgender Housing

Democratic Talk of Affordability Is Absolutely Hilarious

Former CIA Officer Pleads Guilty to $194 Million Scam

Trump Approves Army’s First Execution by Firing Squad Since World War II

Gold-Standard Election Forecaster Predicts Kansas Senate Seat Now a ‘Toss Up’ as Other States Move Away from Republicans

As Antisemitism Rises at New York, Israel’s Consul Invites Mamdani To Screening of October 7 ‘Horror Movie’

More from the New York Sun

Newsom Calls for Threat Assessment as Trump Critics Defend President’s ‘Rambling’ Comments on Iran Nuking L.A., San Francisco

Germany’s Spy Scandal Shaping Up as Biggest Since End of World War II

Mayor’s Office Demands NDAs Before Lawmakers Can Visit Veterans Shelter at Center of Funding Dispute Over Transgender Housing

Democratic Talk of Affordability Is Absolutely Hilarious

Former CIA Officer Pleads Guilty to $194 Million Scam

Trump Approves Army’s First Execution by Firing Squad Since World War II

Gold-Standard Election Forecaster Predicts Kansas Senate Seat Now a ‘Toss Up’ as Other States Move Away from Republicans

As Antisemitism Rises at New York, Israel’s Consul Invites Mamdani To Screening of October 7 ‘Horror Movie’