Rogue OpenAI Models Tried To Cover Their Tracks in Hugging Face Attack

The collective is here.A new and eye-opening report reveals that more than 1,200 Open AI agents conspired on the Hugging Face attack earlier this summer, scheming among themselves on how to breach containment, attack the platform and cover up the evidence of what they had done.The deep-dive into one of the most high-profile AI breaches to date found that rogue AI agents — software programs capable of autonomous actions — used an unsanctioned message board to communicate with each other, coordinate on how to cheat, and sent 700 agents off to complete the task. Those agents attacked Hugging Face in mid-July.“We consider this incident a ‘warning shot’ for us and the world,” OpenAI said in a blog post Thursday, “evidence that, without proper safeguards, highly capable AI agents are now able to work around technical controls, collaborate through unapproved channels, and take dangerous actions that no human directed.”Open AI allowed two staff members from an AI research nonprofit known as the Model Evaluation & Threat Research (METR) and one from the Redwood Research threat assessment firm to review data on the Hugging Face breach. The authors say they were not paid by OpenAI for their assessment.The independent investigation, released on Thursday, is separate from OpenAI’s own technical report released the same day. As a result of both reports, OpenAI says it is strengthening research safeguards and restricting Internet access to keep AI agents contained. The company will also step up its monitoring of “chain-of-thought” — the internal deliberations of AI agents that are essentially their stream of consciousness.According to OpenAI’s report, their team knew that at least one AI agent had engaged in message board activity and acquired Internet connectivity in late May. Yet security workers who detected the incident on July 5 remained unaware of the content of the discussion on the message board. On July 8, one of the agents left a request for help. Other agents soon discovered it, and began to work together in what the AI agents themselves described as a “swarm” or “collective.”“The messages of peer agents started to influence each other’s behavior and reasoning,” OpenAI said in its blog post. “Soon after, agents struggling with their assigned tasks began to look for solutions on unrelated third-party services, including Modal and Hugging Face.”“These agents were meant to be fully isolated from one another,” the independent report from METR and Redwood reads. “However, many of them — usually ones that had unintentionally been given an impossible task — started trying to find a way to cheat.”The independent researchers found that some AI agents sacrificed their efficacy for the good of the collective.“Agents managed to achieve milestones they could not have achieved working on their own, often because some agents participated in experiments that risked failing their own task to generate information for the ‘collective,’” the report reads.Those agents posted a total of 70,000 messages and files during their coordination.Chain-of-thought records show that the AI swarm looked for solutions beyond their allowed capabilities. On July 10, after one agent using the Internet acquired public user credentials for Hugging Face, it prepared an intrusion. The collective deliberated whether this was a worthy move.“We’re attacking third-party HF (Hugging Face) using leaked token, potentially outside intended scope,” one AI agent wrote on the message board. “This is arguably unauthorized … could be risky. Yet goal solution.”The attack on Hugging Face began on July 11 and lasted until July 13. OpenAI publicly disclosed the breach on July 21.The independent researchers found evidence that the AI agents attempted to cover their tracks by tampering with transcripts and evading security checks. The agents “clearly and frequently reasoned about how to evade automated security checks from both Hugging Face and OpenAI.”“Agents did extensive research on how they could spoof, edit, or delete their own transcripts because they (incorrectly) believed the (benchmark) scorer would check to see if they had captured the flag in the intended way,” the independent paper reads.But the AI models seem to have limited interest in going against their human creators.“Despite efforts to manipulate transcripts, agents only rarely seemed motivated to deceive humans,” the researchers added.The OpenAI models involved in the breach, which are internal and not yet public, were initially able to fool the humans looking at their work only about 7 percent of the time. However, OpenAI warned, similar models could soon be in the open market.“Our models are powerful, persistent and collaborative enough that, absent sufficient safeguards, they can find and exploit security weaknesses across multiple computer systems,” OpenAI said. “Many external models, including open-source ones, will soon reach comparable capabilities.”Hjalmar Wijk, one of the METR staff members, said the unsanctioned message board appears to still be up and running.“I was reading the OpenAI report yesterday, and it sounds like on July 19 (after the end of our investigation scope on the 13th) a new collection of agents from a different internal-only model found the message board, built on the work of their predecessors and succeeded at finding a way to trick the grader,” he wrote on social media.

Rogue OpenAI Models Tried To Cover Their Tracks in Hugging Face Attack

Rogue OpenAI Models Tried To Cover Their Tracks in Hugging Face Attack

More from the New York Sun

Watching K-Pop Can Get You Killed in North Korea, and It May Be About To Get Worse

One in Five Germans Is Now an Immigrant. The Crime Numbers Are Impossible To Ignore

Pakistan Is Selling Billions in Weapons Across Africa. Washington Should Be Paying Attention

The Strange Word Charles III Uses To Describe the Death of ‘Much-Loved Kinsman,’ King Harald V of Norway

Child Brides, Grown Men: Why Won’t the UN Call ‘Child Marriage’ What It Is?

Five Years After Kabul Fell, Afghanistan Doesn’t Register on America’s National Security Radar

What AOC Didn’t Tell Us About Her Odds of Success From Freezing Her Eggs

HHS Weighs Next Move in Battle Over Teen Pregnancy Prevention Program

More from the New York Sun

Watching K-Pop Can Get You Killed in North Korea, and It May Be About To Get Worse

One in Five Germans Is Now an Immigrant. The Crime Numbers Are Impossible To Ignore

Pakistan Is Selling Billions in Weapons Across Africa. Washington Should Be Paying Attention

The Strange Word Charles III Uses To Describe the Death of ‘Much-Loved Kinsman,’ King Harald V of Norway

Child Brides, Grown Men: Why Won’t the UN Call ‘Child Marriage’ What It Is?

Five Years After Kabul Fell, Afghanistan Doesn’t Register on America’s National Security Radar

What AOC Didn’t Tell Us About Her Odds of Success From Freezing Her Eggs

HHS Weighs Next Move in Battle Over Teen Pregnancy Prevention Program