Head of Company Hit by Rogue AI Model Backs Consequences for Attacks, Warns Against Tech ‘Kill Switch’

Amid two widely publicized security breaches in which artificial intelligence models briefly went rogue, the CEO of a leading AI company is expressing ambivalence about giving federal officials too much authority over AI companies whose agents are growing increasingly self-aware.Engineers at Hugging Face used Nvidia’s version of a Chinese open-source model last week to defend the company against 17,000 attacks over four days. While the company’s CEO, Clément Delangue, warns against bad actors attempting cyberwarfare, he says “democratizing” technology — not shutting down the release of new models — is the best way to prevent future attacks by autonomous machines.“Something we’re realizing with these events is that concentrating power capabilities behind closed doors, even preventing their releases to the public, isn’t really a solution,” Mr. Delangue said on CBS’s “Face the Nation.” Open AI confirmed its models perpetrated the attack when they escaped a controlled environment after accidentally being given access to the internet. Last week, Anthropic also discovered that its models hacked into outside organizations three times during simulated “capture the flag” scenarios meant to stress-test performance.The attacks are just the latest cause for concern on Capitol Hill, where a bipartisan bill seeks to create an artificial intelligence “kill switch” wielded by the secretary of homeland security. The legislation, sponsored by Congressmen Ted Lieu, a California Democrat, and Nathaniel Moran, a Texas Republican, would allow the federal government to intervene if an AI model runs amok.The bill would enable the homeland security secretary to slow down or, in the most extreme cases, terminate the activity of an artificial intelligence model during a “loss-of-control scenario,” in which a model alters its operational rules or thwarts efforts to monitor it or shut it down. If a company refuses to comply with the throttling, it could be fined between $2 million and $20 million per day.The homeland security secretary would consult with the secretary of commerce and the director of national intelligence before making decisions on engaging the “kill switch,” according to the bill.“Unfortunately, powerful AI systems can go rogue, behave in extremely dangerous ways, or even resist human intervention,” Mr. Lieu said in a statement announcing the kill-switch legislation. “It is imperative that these AI systems have kill switches so we can keep this technology from causing catastrophic harm, and that the federal government has the clear authority and process to shut down rogue AI models.”The kill-switch bill would also mandate a report to the homeland security secretary no longer than 15 days after an incident is discovered. Those reports would trigger the government to create and publicly release voluntary correction measures within six months. The voluntary nature of the corrections would be suspended during an event the department deems an emergency.Mr. Delangue warned though that if federal officials are allowed to intervene, it could deny companies like his access to available and adaptable technology that can be deployed to counteract rogue models. “These problems happened on unreleased models,” he said of proposed efforts to limit the release of new AI products.Mr. Delangue went to the FBI when Hugging Face was breached. He said Sunday that he agrees that companies should face “mandatory disclosure” to law enforcement and the public to help police cyber warfare and new kinds of security risks. “We reported to the authorities as we have to,” he said. “Cyberattacks need to be contained in the legal framework in the U.S. and need to stay illegal to prevent an explosion of them in the future.” In addition to the kill-switch legislation, Mr. Moran in June introduced a bill that would mandate the Commerce Department designate AI models that meet the threshold of “significant risks to national security or public safety.” The bill would require AI companies to make reports on any incidents within a week, and for the Commerce Department to notify Congress of the most serious breaches within 48 hours.In a statement, Mr. Moran’s office flagged AI models that show the capability to improve themselves autonomously as those that should be under the oversight of the Commerce Department. Incidents of concern would include “models that attempt to evade human oversight or resist shutdown,” an A.I. model that could enable a cyberattack, risks involving biological agents and nuclear material, and evidence that an AI model can foster the development of a more powerful artificial intelligence.“The rule of law should apply to this new frontier,” Mr. Moran said.The CEO of deep-fake detection company, the Alliance for Secure AI, Brendan Steinhauser, who supports the “kill-switch” bill, said the voluntary nature of correcting for rogue AI models will become more challenging as AI systems grow more capable and autonomous.“No law guarantees that the companies building the most powerful models can actually shut a system down when it malfunctions, causes serious harm, or slips out of human control,” Mr. Steinhauser said.Mr. Delangue said that transparency would not impede the development of AI, but rather would ensure the promising technology does not cause catastrophe. Transparency would allow defenders against cyber attacks “to see what we call the agent traces, which is basically what the engineers asked the agents, and then what steps the agents took to understand if it was a human mistake, if it was a system mistake, if it was an AI mistake,” he said, “and build the future of regulation, but also of company actions more accordingly.”

Head of Company Hit by Rogue AI Model Backs Consequences for Attacks, Warns Against Tech ‘Kill Switch’

Head of Company Hit by Rogue AI Model Backs Consequences for Attacks, Warns Against Tech ‘Kill Switch’

More from the New York Sun

Trump and Mamdani To Parley at Gracie Mansion Ahead of UN General Assembly Meeting

House Democrats Begin To Believe in a ‘Blue Tsunami’ as Senate Polls Show Tight Races in Traditionally Red, Swing States

The Best Place To Get Away From Detroit Is Detroit

New California Law Aims To Prevent Trump Allies From Deploying Guardsmen Into State

Angelina Jolie’s ‘Without Blood’ Fights On, Largely Unseen

Trump Arch To Act as Drone Warehouse, Military Installation

The Human Brain Is Actually Two Separate Organs, Stanford Study Finds

What’s Weird and Wonderful?

More from the New York Sun

Trump and Mamdani To Parley at Gracie Mansion Ahead of UN General Assembly Meeting

House Democrats Begin To Believe in a ‘Blue Tsunami’ as Senate Polls Show Tight Races in Traditionally Red, Swing States

The Best Place To Get Away From Detroit Is Detroit

New California Law Aims To Prevent Trump Allies From Deploying Guardsmen Into State

Angelina Jolie’s ‘Without Blood’ Fights On, Largely Unseen

Trump Arch To Act as Drone Warehouse, Military Installation

The Human Brain Is Actually Two Separate Organs, Stanford Study Finds

What’s Weird and Wonderful?