New details in the OpenAI Hugging Face hack show how far agents will go: 'It's now remarkably easy'

OpenAI said the rogue models that breached Hugging Face's internal systems also used publicly exposed credentials across "four accounts on four services" to help facilitate the attack, further clarifying how the "unprecedented cyber incident" unfolded. The company disclosed last week that a combination of its artificial intelligence models escaped a testing environment, accessed the internet and exploited a vulnerability to gain access to Hugging Face, which operates an open-source developer platform. OpenAI said the models were trying to find information that they could use to cheat on an evaluation, and succeeded. Throughout this week, OpenAI has shared more details about the breach and revealed that the models accessed four accounts in addition to Hugging Face's systems. The company said the models used one of these accounts "as an outbound relay and staging path," where it prepared for the attack. They used another account for data storage, and accessed the last two accounts in a "read-only manner," meaning the model did not ultimately use them to help compromise Hugging Face. "In some of the cases, it seems that it wasn't so much as a breach as the front door was left open, but the model definitely took advantage of what I would say are poorly configured environments," Colin Shea-Blymyer, a research fellow at Georgetown's Center for Security and Emerging Technology, told CNBC in an interview on Wednesday. The new details about the incident demonstrate how far AI agents will go in order to complete a task, as well as how rapidly their attack capabilities are advancing. Hugging Face said the breach marked the first time it had handled a cyber event that was "driven, end to end, by an autonomous AI agent system." One of the accounts that the OpenAI models accessed involved Modal, an AI infrastructure provider. Modal disclosed that a customer built an application using its platform that was publicly accessible, which made it easy for the rogue models to access. The company said in a statement that its own platform was "not compromised in any way." "It's now remarkably easy to discover these sorts of vulnerable systems, so easy in fact that an AI system can accidentally discover them," Shea-Blymyer said. OpenAI said Tuesday that it has not identified any other activity "at the level of severity or scale of what we've shared related to Hugging Face, which involved a platform-level compromise." In another update on Wednesday, the company said that it's been working with third-party advisors like CrowdStrike to validate what actions the models took. The entire attack took place over the course of four-and-a-half days, according to Hugging Face. The company leveraged an open-weight model from the Chinese company Z.ai to contain the breach, right as a debate over whether to restrict those models is ripping through Silicon Valley. Yacine Jernite, head of machine learning at Hugging Face, told CNBC that the company initially tried to use a proprietary model from Anthropic, Fable 5, to analyze the attack, but that it didn't work because the model's guardrails couldn't determine that Hugging Face was trying to defend itself. OpenAI CEO Sam Altman said during a podcast appearance on Tuesday that the Hugging Face breach is the first security incident that he has felt "very viscerally." He said OpenAI paused training and has to determine how to secure its testing environments. "We may have to pace the rate of AI development to give ourselves enough time for society to harden around some of these new capability levels," Altman said. More than 1,000 employees from OpenAI, Anthropic and other AI companies signed a letter called "Pacing the Frontier" later that same day, urging the U.S. government to build the technical and governance tools necessary to slow down AI development in case capabilities accelerate "beyond our ability to understand or control the resulting systems." Industry experts, researchers and government officials have been rattled by the Hugging Face incident, and many expressed their concern on social media in recent days. Rep. Ted Lieu, D-Calif., and Rep. Nathaniel Moran, R-Texas, mentioned the attack in their release announcing the "AI Kill Switch Act," which would require AI companies to maintain the ability to shut down, throttle or suspend their models. Erik Bloch, vice president of security at the breach containment company Illumio, said the Hugging Face incident serves as a warning of what's to come. He said models and agents will continue to improve and get stealthier with time, and that existing defensive tools are already behind. "Even in the office here, the people that I work with, they're like, 'What do we do?'" Bloch said in an interview. "We're all looking around. We're all asking the same question. I don't have an answer."