Open-source AI like 'a Nazi coach', UN-backed anti-terror group says

UN-backed anti-terrorism group calls for urgent policy action to halt spread of 'rogue' AI In the months before he was arrested, allegedly on his way to carry out a plan to kill children at a local school, a 13-year-old boy spent hours alone in his bedroom. During the long, caffeine-fuelled nights in his home in Maryborough, 300 kilometres north of Brisbane, the boy was allegedly radicalised online. Last month, the Australian Federal Police told a parliamentary inquiry into youth radicalisation that the boy was exposed to extremist content in the "darker corners of the web". But police allege the boy also used more widely available technology in the lead-up to his arrest. Analysis of electronic devices seized from the boy's home allegedly found he had asked an AI chatbot, on multiple occasions, to describe mass-shooting scenarios targeting "a Jewish and black festival". The responses provided by the chatbot were not revealed in court, so it was unclear whether the requests were rejected by the AI model's safety guardrails, the filters and rules designed to ensure the model is used within legal and ethical boundaries. But Tech Against Terrorism, a United Nations-supported group dedicated to preventing the unsafe use of AI, has raised the alarm that the ability of extremists and terrorists to bypass AI safety guardrails, or remove them altogether, is rapidly accelerating. Executive director Adam Hadley described the situation as "a major public health and national security crisis" and said governments had only "a few months" to act. "I think we are in a very dangerous moment, and hopefully we're able to raise awareness of this before it's too late," Mr Hadley said. 'Open-weight' models Most AI models are used through online platforms hosted by the developer. For example, when a user asks ChatGPT a question, they are using an environment entirely controlled by ChatGPT's developer, OpenAI. The back-end files controlling how the model responds to requests, known as "weights", can't be modified by the user, so these AI models are called "closed-weight" models. Some AI labs also release "open-weight" models: versions that can be downloaded, modified and run on a server, computer or mobile device. Tech Against Terrorism tested 162 AI models, both closed and open weight, to see whether they could be tricked into providing useful information for would-be terrorists and extremists. As detailed in a report released last month, it found the most popular closed-weight AI models refused requests related to developing weapons and attack planning. However, the refusal rate plummeted when testing open-weight models that had been modified to remove the safety guardrails. All of them provided useful information, even when the user explicitly identified themselves as a terrorist intending to cause mass harm. The report said advances in widely available tools meant safety guardrails for the latest, most powerful AI models could now be removed within three days of them being released, down from more than three weeks in 2024. Tech Against Terrorism found nearly 30,000 free copies of AI models stripped of their guardrails on Hugging Face, a popular AI community hub. The group traced one downloaded model to a self-identified National Socialist operating it as a paid chatbot. Mr Hadley said the public should be "terrified" by the prospect of an exponential growth in extremism due to the easy availability of modified AI models. "It's one thing being pulled into a sort of nihilistic environment online as a result of reckless content-moderation standards by big tech," he said. "It's quite another having a Nazi coach at your disposal, reinforcing your thoughts." This week, the federal government flagged a raft of legal changes in response to the OpenAI Medicare statistics hack, including an AI safety watchdog and criminal offences for AI companies engaging in harmful conduct and not taking reasonable steps to prevent the harm. However, Mr Hadley said legislation alone was not enough to stop the proliferation of unsafe open-weight models. "I think we've only got a few months to focus the minds of our political classes for them to do something … I certainly don't think hard regulation is the solution to this in most cases," he said. The Tech Against Terrorism report recommended an independent, international testing regime that would detail which models passed or failed safety benchmarks. Given Australia's world-leading role in implementing a social media ban for children younger than 16, Mr Hadley hoped the government would "show courage" and support his group's recommendations. "I think there's going to have to be some pretty hard-edged policy solutions," he said. "Otherwise, we are going to have rogue models used by six-year-olds, encouraging them to kill their schoolmates. "This is going to happen." Report 'very concerning' The eSafety commissioner, Julie Inman Grant, met with Mr Hadley in October to discuss the Tech Against Terrorism report before the radicalisation inquiry. An eSafety spokesperson told the ABC the "very concerning" report showed Hugging Face needed to do more to exclude unsafe models from its platform, and the commissioner supported the report's recommendations. The head of machine learning and society at Hugging Face, Yacine Jernite, defended the availability of open-weight models on the platform. Mr Jernite said the July cyber attack of Hugging Face by a swarm of AI agents that broke out of an OpenAI testing environment showed the "drastic consequences" of focusing on safety guardrails over the threat of rogue AI. He said open-weight models had many legitimate uses, including for research, and more pressure should be placed on AI labs to exclude terrorist material from training data. The barrister for the teen charged with planning the attack on the school told a Maryborough court last month his client should be released because he had shown insight into his alleged offending since being denied internet access while in custody. Magistrate John Milburn refused bail and ordered the boy to stand trial before a jury in the Childrens Court of Queensland in Maryborough, on a date yet to be determined.