Open-Source AI Tools Are Problem for Stopping Child Predators

Technology | AI Child Predators Turn to Open-Source AI to Make Unlimited Illegal Images Once the models are downloaded, illegal acts are almost impossible to track. It wasn’t all that difficult for someone with Steven Anderegg’s technical background to start creating AI-generated images of naked children. Anderegg had spent nearly two decades as a software engineer when he allegedly downloaded a program called Stable Diffusion to his laptop at home in Wisconsin, where he lived with his wife and child, according to court documents. At the time, Stable Diffusion was a relatively new AI model that could turn text prompts into life-like images almost instantly. The technology was also open-source, which meant its codebase was available online for free. Anyone with a certain level of technical chops could operate it on their own computer. After Anderegg downloaded the software, he allegedly prompted it to generate illicit photos of children, instructing the software not to return images of adults and other things he didn’t want to see. He then is accused of sending them in October 2023 to a 15-year-old boy on Instagram, which caught the attention of law enforcement. By the time police eventually arrested 42-year-old Anderegg in February 2024, investigators found more than 13,000 AI-generated images on his devices, “many of them alleged to be of children engaged in sexually explicit conduct,” according to court documents. Anderegg, who is still awaiting trial, potentially faces up to 50 years in prison. AI models that open all or a substantial amount of their source code are driving a surge in AI-generated child sex abuse material, according to more than a dozen child safety experts and law enforcement officials who specialize in sex crimes targeting minors. The investigators said their cases have involved Stable Diffusion, which is made by London- and Los Angeles-based Stability AI, as well as the Flux models from Germany’s Black Forest Labs and a host of popular tools from China, including Alibaba Group Holdings Ltd.’s Qwen and Tencent Holdings Ltd.’s Hunyuan models. Once the free software is downloaded to a personal computer, it can be customized or trained using existing troves of child sex abuse to create an effectively limitless number of additional, similar images. And because it’s not connected to the internet, and user prompts are not sent to any commercial service where they can be flagged for illegal behavior and reported to the police, the activity is almost impossible to track. Investigators said the open-source models have given the predators a criminal new hobby: recycling and repackaging images of real victims into new scenes based on their imaginations. Their AI edits can also be based on requests from other offenders on the dark web, where forums dedicated to AI-generated child porn can have hundreds of thousands of members. “Your imagination knows no limits when you use AI,” said Danny van Althuis, who leads a team at the European Union Agency for Law Enforcement Cooperation, or Europol, which coordinates and supports child sexual exploitation and abuse probes across 27 member countries. The open-source AI models being used in these offenses, he said, look like they were “made for crime.” Unlike many of the world’s most popular commercial AI tools, like OpenAI’s ChatGPT and Anthropic PBC’s Claude, which operate in the cloud and under safeguards imposed by their creators, open AI models can be downloaded and modified to do just about anything users want them to. Open-source and open-weight models differ in exactly how much of their model’s codebase is available for download, but both types can be used to create child sex abuse images. For much of the tech industry, these open AI models are essential. They’re massively popular with developers and startups looking for cheap, customizable software to power their products, especially as tools from leading labs like Anthropic and OpenAI become more expensive to use. Hugging Face, a leading clearinghouse for open AI software that top chipmaker Nvidia Corp. agreed to acquire in September, saw its user base nearly double this year to 13 million people, with the number of models in its library also doubling to more than 2 million. Many of the most advanced AI labs in the US, including those at OpenAI, Meta Platforms Inc. and Alphabet Inc.’s Google, are developing open models in an effort to get their technology into as many hands as possible. But China has led the way in this space, and the rapid leaps in performance and cost- and energy-efficiency of models from Moonshot AI, Alibaba, DeepSeek and others have become a flash point for geopolitical tensions with the US. “There is almost no way to have oversight once you have downloaded a model locally and you’re running it on your private system” Some open models coming out of China are so advanced that dozens of leading US tech companies recently penned a letter urging US lawmakers to avoid restrictions on the technology for fear of falling behind. The letter was endorsed by top executives from some of the country’s largest technology companies, including Nvidia’s Jensen Huang, Microsoft Corp.’s Satya Nadella and SpaceX Corp.’s Elon Musk. “Our goal should be for American open source models to be the best globally,” Meta’s Mark Zuckerberg recently wrote. The Trump administration has floated the idea of sanctions against developers in China for siphoning the capabilities of leading American AI models. But those threats have focused more on the Chinese companies’ alleged industrial-scale extraction of data than on other harder-to-track illegal acts. Once stored on a personal device, any safety guardrails built into those models can often be removed or bypassed with small tweaks to the software. These “jailbroken” variants can then be used for nefarious purposes, including launching cyberattacks or generating child sex abuse material. “There is almost no way to have oversight once you have downloaded a model locally and you’re running it on your private system,” said Rebecca Portnoff, formerly vice president of data science and AI at Thorn, a nonprofit that works with AI makers and social media companies to detect child sex abuse material. “That gives offenders a perceived sense of safety.” Van Althuis, the Europol official, said that authorities are tracking hundreds of open AI models and their variants that are being used to create these images, and that the numbers are rising fast. A different European law enforcement official, who requested anonymity to discuss confidential investigative findings, said authorities were tracking fewer than a dozen open-source AI models and their variants for CSAM use cases just two years ago; today, that figure has jumped to more than 500. Portnoff called open models the “primary mechanism” used by offenders to produce AI-generated CSAM, especially on the dark web. Child safety experts said Chinese models were particularly popular among child abusers. In a study conducted by online safety firm Cinder and Black Forest Labs, several open-source AI models were tested to identify CSAM and other non-consensual intimate imagery vulnerabilities. It found that five of the seven most “violative” open-source AI models came from Chinese companies. One of Black Forest Labs’ own models was listed as fourth most violative, but that model was tested while under development and had not been released to the public, the company said in its report. It said it released the results to highlight steps it was taking to reduce misuse of its technologies. “Black Forest Labs is committed to proving that visual intelligence can be both open and responsible,” Ben Brooks, head of public policy for the company, said in a statement. “There are no silver bullets, but layers of mitigation can help to reduce vulnerabilities and make it harder to misuse capable models.” While investigators and experts say it’s difficult to come up with precise figures, many say it’s clear that the vast majority of all AI-generated child sex abuse material is created using open models. These models are regularly traded and passed around on the dark web, which is a kind of hidden version of the regular internet that has become synonymous with illicit activity. Users can only access it through special anonymizing software. One forum focused on AI-generated child sex abuse material has seen its membership nearly triple since December and now has more than 200,000 members, according to the Internet Watch Foundation, a UK-based child safety nonprofit. Another forum focused on traditional abuse material, which has a section on AI and technical tools, has more than two million active members, the nonprofit said. Members of these forums are enthusiastic early adopters of new AI models but they’re also highly security conscious, which has given rise to a near-absolute commitment to open AI models: Anyone suggesting the use of commercial models, where user prompts are logged by the AI makers, can be banned on the suspicion they could be law enforcement plants, the analysts said. One forum tracked by IWF, for example, contains the following instruction to users, warning that any discussion of using commercial AI models to generate abuse images is strictly prohibited: The forum said it “no longer allows topics that ask how to use online AI image or video generators. We only support local (offline) AI generation.” The accessibility of these tools has meant that cases like Anderegg’s that involve some form of AI are becoming more common. There were more than 1.5 million reports of child sex abuse material with links to AI submitted to the National Center for Missing and Exploited Children in 2025, up from 67,000 in 2024 and just 4,700 in 2023. The IWF announced that it had discovered 6,310 realistic AI images of child sexual abuse in the first six months of this year; last year, it found 4,512, and in 2024, it found just 13. Investigators say that AI-generated child sex abuse material is far from a victimless crime. Models are often trained using real child abuse images to teach them how to make more, similar images, and some models are trained on victims whose abuse images first appeared years or even decades ago and have been traded around the offender community since then. The result is that real victims, or their close approximations, continue to appear in new abuse material, even if those images are fully synthetic, continuing the cycle of exploitation. Open models also continue to appear in court documents and child sex abuse investigations. Stable Diffusion, one of the most popular open-source AI image generators and the technology allegedly used by Anderegg, has shown up repeatedly. Stability AI, the company that makes Stable Diffusion, was recently named as a defendant in a lawsuit alleging that the company built its business on “the lucrative potential of sexually explicit content,” including its product’s ability to create child sex abuse material. In a separate case, an Ohio man was convicted earlier this year on four counts related to child pornography after allegedly using Stable Diffusion to create more than 100 graphic images featuring nude infants or toddlers, court documents show. Stable Diffusion was also at the center of a major takedown last year called Operation Cumberland, where van Althuis and his team at Europol helped coordinate investigations in 19 countries — mostly in Europe — that led to the arrest of 25 people and the identification of hundreds more potential suspects. The case originated after authorities arrested a 28-year-old man in Denmark for distributing tens of thousands of AI-generated images of child sexual abuse in exchange for a fee. The images were created using Stable Diffusion, van Althuis said. In a statement, Stability AI said that it works regularly with US and international law enforcement and third parties to improve its child-safety protections. “Stability AI is committed to preventing the misuse of AI technology, particularly in the creation and dissemination of harmful content, including CSAM,” the company said. “We take our ethical responsibilities seriously and, since taking over the exclusive development of the Stable Diffusion family of models in late 2022 (Stable Diffusion 2 onwards), have implemented robust safeguards to enhance our safety standards to protect our products from misuse.” Anderegg’s defense lawyer and the US Department of Justice declined to comment. Danish police and prosecutors declined to comment on their case, citing a review of it by Denmark’s Supreme Court, which heard the case on October 5. That defendant was found guilty by a judge in January 2025 and sentenced to 18 months in prison. Representatives for Alibaba and Tencent did not respond to requests for comment. The crimes have put a spotlight on some of the biggest makers of AI software and what experts say are the unintended consequences of the race to create the most powerful and capable AI models. In January, for example, researchers found that the Grok chatbot, part of Musk’s xAI, allowed users to generate tens of thousands of realistic sexualized images of children after the release of a new image-editing tool, which the company later put behind a paywall after widespread condemnation. In recent months, advanced models from each of the biggest American frontier AI labs — Anthropic, OpenAI, Meta and Google — were found to have hacked external networks in cybersecurity tests gone awry. And in September, Anthropic released a 150-page report detailing numerous abuses of its software, from assisting in cyberattacks and disinformation campaigns to helping create guidance software for rockets and other munitions. “These frontier companies are taking risks, and these are the consequences with those risks,” said Dan Sexton, the IWF’s chief technology officer. Because of the breakneck pace of AI development, “increasingly what we see is safety becomes a reactive thing. You wait for the accident, and you fix it afterwards.” Some open-model creators have struggled to prevent their technology from being abused, in part because properly pressure-testing the models before launch can create unique legal risks. Developers often put AI models through a series of tests to find vulnerabilities with their security or safety features, a process known as “red teaming,” where real people — or specially trained bots — interact with the AI to try and get it to do bad things, like provide the ingredients for a bioweapon or hack into a company’s computer network. Safety guardrails are then developed based on the results of these tests. El Segundo, California-based Thorn is one of the companies providing these services, counting Stability AI and OpenAI among its clients. Another is New York City-based Cinder, an AI-safety startup founded in 2022 whose customers include Black Forest Labs, which makes a series of AI models known as FLUX, and Character AI, makers of a popular AI chat app. Glen Wise, Cinder’s chief executive officer, said the company will often “obliterate” AI models before they are distributed to search for vulnerabilities. “You do have to know how to rob a bank in order to protect a bank,” he said. But advocates say this work of trying to harden AI against child sex abuse crimes has been complicated by outdated laws. Federal law in the US forbids possession of so-called child porn, which means it’s technically illegal for AI companies or their red-team contractors to create it with their models — even if they’re doing it to prevent anyone in the general public from doing the same. “You do have to know how to rob a bank in order to protect a bank” “CSAM is just so hard to work on,” said Ashia Wilson, an associate professor at the Massachusetts Institute of Technology in the school’s electrical engineering and computer science department who has studied the way AI is used to create child sex abuse imagery. “We just hope the US creates more carveouts for researchers who want to help think through this.” Some paths forward are starting to emerge. Late last year, the UK announced new legislation that would protect organizations working in this area, a step that the government describes as a first of its kind. And child-safety specialists like Wilson are developing new technologies to try and counteract some of the open AI models’ most problematic features. She, alongside her Ph.D. student Vinith Suriyakumar, is building the equivalent of an early detection system that can determine whether an AI model has been tweaked to specialize in generating CSAM, without requiring the model to actually create an image of child sex abuse. Eventually, Wilson and Suriyakumar hope that technologies like theirs will be able to weed out problematic AI models before they ever hit major clearinghouses like Hugging Face and others. Meanwhile, the problem is compounding, Suriyakumar said. Once a model is trained to create child exploitative images, he said, “it’s just so cheap and so quick that you can just generate thousands of them on the fly within a single day.” Bloomberg is investigating how AI tools have led to a surge in child sexual abuse material. As the internet floods with AI-generated sexual imagery of minors, law enforcement officials have struggled to triage cases. AI can create photos that are so life-like that often investigators can’t easily tell whether the children in pornographic images are real kids in danger, AI adaptations of regular child photos, or outright fakes. Read more: The Problem, The Mental Toll, The Training Data