Biggest AI Rivals Agree They Need To Pump Brakes as Anthropic CEO Warns AI Poses Risk to Humanity

Anthropic CEO Dario Amodei is urging artificial intelligence companies to deliberately slow the pace of AI development, arguing that safety measures can’t keep up with how quickly the technology is advancing. His two main rivals in the AI race, Sam Altman and Elon Musk, agreed with his assessment on Saturday. “We must slow the pace at which we improve the capabilities of A.I. models. Progress will still seem fast, and we must make wise use of the time we gain,” Mr. Amodei wrote in a 3,800-word essay published on his personal blog Saturday.Mr. Amodei wrote that he was particularly concerned that an AI swarm might break out of a laboratory and take over the internet in six to 12 months. Two months ago, an AI swarm did break out of a lab and went on a hacking spree. Mr. Amodei wrote that a more advanced swarm with “a similar level of misalignment could have caused catastrophic effects.”Mr. Amodei also highlighted the ability of AI systems to improve on their own without human intervention. These systems, “could outrun our ability to understand and control these systems, and so must be pursued vert carefully, if at all.” Computer scientists call this dynamic “recursive self improvement” and some believe it is what could lead the software to spiral out of control. The essay follows a turbulent week for the AI industry. Anthropic researcher Jacob Coxon resigned Tuesday, publicly stating he quit because he believes leading labs, including Anthropic and OpenAI, are “gambling with our lives.” According to Mr. Amodei’s essay, Mr. Coxon said the people building AI “earnestly believe that it could kill us all by the end of the decade.”Mr. Amodei laid out a three-step plan designed to slow development without “sacrificing commercial advantage or the United States’ lead in AI.” The first step, which Anthropic says it has already adopted unilaterally, involves granting independent evaluators employee-level access to verify the company’s safety practices. The second calls on AI companies in democratic nations to coordinate on shared safety standards, while the third envisions cooperation between democratic and authoritarian governments alike.“To be clear, pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third-party evaluators to confirm this,” Mr. Amodei wrote.The essay drew swift support from rivals. OpenAI CEO Sam Altman said on X that pacing capabilities has become a “primary topic” internally. “I agree with Dario that we need to pace the frontier. Committing to having independent evaluators with employee-like access is a great idea, and we will do the same,” he wrote.Elon Musk, a longtime Anthropic critic, also voiced agreement, writing simply, “Dario is right.”Mr. Amodei has previously signed statements warning of AI’s extinction-level risks but said that slowing down didn’t make sense in 2023, when models weren’t advanced enough to act independently or engage in “significant deception, manipulation, cheating, or cyberattacks.” He argues that calculus has since changed.“I continue to believe that AI can enormously improve the quality of human life. My desire to achieve these benefits is undimmed,” he wrote. “But the benefits will only be achieved if we build the technology in the right way.”Public and political momentum is building alongside industry concern. A Data for Progress poll released Wednesday found that 68 percent of likely voters support pausing advanced AI development until safety guardrails are established, with just 25 percent opposed. Support crossed party lines, with 72 percent of Democrats, 70 percent of independents, and 63 percent of Republicans in favor.The polling coincides with legislation introduced by Senator Bernie Sanders of Vermont and Representative Greg Casar of Texas, which would ban the development of “superintelligent” AI systems capable of surpassing human cognition and would halt advanced AI development until a new federal regulatory body is established to oversee safety standards.

Biggest AI Rivals Agree They Need To Pump Brakes as Anthropic CEO Warns AI Poses Risk to Humanity

Biggest AI Rivals Agree They Need To Pump Brakes as Anthropic CEO Warns AI Poses Risk to Humanity

More from the New York Sun

Could Republicans’ Midterm Rout Foretell a Broader Political Realignment?

Researchers Urge Greater Public Awareness of Risks Involving Weight-Loss Drugs and Pregnancy

Trump Has Himself To Blame

Disaster Alerts Reach More People Than Ever, but a Third of Those Warned Can’t Act on Them

Al-Shabab Names Christians as Targets in Kenya, Borrowing a Page From ISIS

Was Kurt Cobain Murdered? Seattle Watchdog Probes Whether Police Buried New Evidence

Nobel Laureate Machado, Now Back in Venezuela, Faces Uphill Road to the Presidency

What a Democratic Congress Can — and Can’t — Do About State Abortion Policy

More from the New York Sun

Could Republicans’ Midterm Rout Foretell a Broader Political Realignment?

Researchers Urge Greater Public Awareness of Risks Involving Weight-Loss Drugs and Pregnancy

Trump Has Himself To Blame

Disaster Alerts Reach More People Than Ever, but a Third of Those Warned Can’t Act on Them

Al-Shabab Names Christians as Targets in Kenya, Borrowing a Page From ISIS

Was Kurt Cobain Murdered? Seattle Watchdog Probes Whether Police Buried New Evidence

Nobel Laureate Machado, Now Back in Venezuela, Faces Uphill Road to the Presidency

What a Democratic Congress Can — and Can’t — Do About State Abortion Policy