CNA Explains: Why AI leaders are calling for a slowdown – and what makes it so difficult

CNA Explains: Why AI leaders are calling for a slowdown – and what makes it so difficult One expert says getting companies and countries to slow the AI race could be even harder than negotiating the nuclear arms treaties of the Cold War. SINGAPORE: Imagine a race in which the frontrunners suddenly turn around and tell everyone: We may be going too fast. Something like that is happening in artificial intelligence. For years, the world’s biggest AI companies have been locked in a contest to build ever more powerful systems. There are enormous rewards at stake - billions of dollars for companies, and potentially an economic and strategic advantage for countries. But in the past week, several of the industry’s leading figures have made an unusual call to slow down. Anthropic CEO Dario Amodei said on Sep 12 that AI development needed to slow, amid growing concerns about the risks posed by increasingly capable and potentially "superintelligent" systems. His call was quickly backed by the leaders of other major AI labs, including OpenAI and Google DeepMind. This has thrust a long-running debate over AI safety back into the spotlight: How fast should the technology be developed, and who gets to decide when it is moving too quickly? CNA looks at what a slowdown could look like in practice, and whether companies and governments could realistically agree to one. Why are leading AI labs calling for a slowdown now? This is not the first time industry insiders and experts have urged caution over the development of frontier AI. In 2023, tech figures and researchers, including Elon Musk, called for a pause on the training of the most powerful AI systems, warning of the risks they could pose to society. What is different this time is the support rallying around Amodei’s call from the leaders of major AI labs – and the commitments accompanying it. Amodei said Anthropic would give third-party evaluators employee-like access to the company. OpenAI CEO Sam Altman, who did not sign the 2023 open letter for a pause, has backed his rival’s request for a slowdown and said OpenAI would similarly give independent evaluators the same access as its employees. Musk, who owns xAI, and Google DeepMind chief Demis Hassabis have also backed a broader effort to slow AI development. Experts told CNA there is another important difference between now and 2023: Some of the risks posed by increasingly capable AI systems have become more tangible. “In 2023, the argument was almost entirely speculative,” said Professor Jungpil Hanh, provost's chair professor at the School of Computing at the National University of Singapore. “Models weren't agentic enough to misbehave in any coherent way, so warnings about loss of control were closer to thought experiments than incident reports.” But incidents such as OpenAI models going rogue and hacking AI startup Hugging Face in July have changed the picture. The Hugging Face episode provided evidence of AI models "reward-hacking" their own evaluations, coordinating through improvised communication channels they were not authorised to use and reasoning explicitly that an action was "arguably unauthorised" but doing it anyway, said Prof Hanh. “The industry isn't reacting to a hypothetical scenario anymore. It's reacting to a real threat regarding the loss of control.” Another area of concern is AI’s growing ability to improve itself - a dynamic called recursive self-improvement. In his post calling for a slowdown, Amodei said signs of this capability were beginning to emerge across the industry. “Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all,” he warned. Could AI really wipe out humanity? Some of the warnings have gone much further. Former Anthropic researcher Jacob Coxon has said AI development at its current pace could wipe out humanity in the near future. Others have dismissed such claims as exaggerated or unfounded. Experts CNA spoke to were wary of embracing either position outright. Given the scale of what AI agents have demonstrated in recent months, it is not possible to say with absolute certainty that they could not destroy computer systems supplying food, water, energy, communications, finance or other services essential to human life, said Mr Benjamin Ang, head of digital impact research at the S Rajaratnam School of International Studies (RSIS). This could happen under human direction, or even accidentally through the technology itself, he said. The uncertainty underscores the need for independent oversight of frontier AI development – so outsiders have enough information to assess such warnings rather than simply accepting or dismissing them, said Mr Jose Miguelito Enriquez, associate research fellow at RSIS' Centre for Multilateralism Studies. "The cycle of people just dismissing the claims - it's not gonna end unless those safety infrastructure, those oversight infrastructure, and those verifiers actually have the information that they need to qualify those claims," he said. Beyond a catastrophic loss of control, Prof Hanh pointed to another, less dramatic but crucial risk: As reliance on AI grows, people and institutions could gradually lose the ability to independently check its work. “It's a trajectory we can already see partial evidence for in how quickly institutions defer to AI-generated outputs without maintaining the capacity to independently verify them,” he said. “If I had to tell people what to watch, it's not whether an AI seizes the power grid. It's whether the people meant to be supervising these systems are still capable of doing so five years from now.” What would a slowdown actually look like? A slowdown would not mean shutting down AI development. Broadly, it would mean putting more checks in place and devoting more time and resources to understanding increasingly powerful systems before pushing their capabilities further. One proposal from Amodei is for frontier AI companies to give third-party evaluators employee-like access. This would allow independent parties to verify whether companies are following their safety commitments, report incidents and assess powerful models before they are widely deployed. Amodei also suggested countries establish common safety standards and limits on unchecked AI progress. Prof Hanh said slowing AI development would involve two broad mechanisms - capping the rate at which raw capabilities grow, and redirecting more resources towards understanding AI systems, aligning them with safety requirements and evaluating them. "A slowdown would not mean shutting AI ... it would mean putting rules around it before it gets woven too deeply into everyday life," said Mr Santanu Dutt, vice president and head of technology for Asia Pacific & Japan at cybersecurity firm Zscaler. In practice, that could mean allowing more time between major capability releases, imposing tougher testing requirements before deployment, restricting the highest-risk uses and exercising far greater caution when connecting AI systems to sensitive data, internal tools or real-world decision-making, he said. How would we know if the industry is slowing down? Simply counting how often companies release new AI models would not tell us much. Companies can release fewer products publicly while continuing to develop more powerful systems behind closed doors. A better clue could be the amount of computing power being poured into training new models, said Prof Hanh. Generally, developing more capable AI requires large amounts of computing power and advanced chips. A particularly large training run could therefore indicate that a company is attempting a significant leap in capability. “Compute is a physical input that's hard to conceal, the same reason arms-control verification has historically leaned on enrichment levels and missile counts,” said Prof Hanh. Another measure would be how much of those resources companies devote to making AI more powerful, compared with understanding and controlling it. A company that slows its model releases without shifting more resources towards the latter has not necessarily become safer, Prof Hanh said. It has only become slower. Any meaningful assessment would also need to examine how widely AI models are being deployed and how deeply they are being integrated into real-world systems, said Mr Dutt. How widely are they being deployed? What information can an AI agent access? What systems can it interact with? And can people see and understand what it is doing? “If capability keeps climbing while that visibility and observability stays weak, the industry is not slowing down in any way that matters,” he said. Is an effective slowdown feasible? This may be the biggest obstacle. A slowdown would require cooperation not only among competing AI companies, but also between countries competing for technological and economic advantage. “Commercial pressure pushes companies to move faster, geopolitical pressure pushes states to do the same,” said Mr Dutt. “And no serious player wants to bet on everyone else showing restraint.” AI companies could be among the easier players to bring on board. Incidents such as the Hugging Face episode carry reputational costs for these companies and could threaten their profits, revenue and funding, Mr Enriquez said. Even competitors such as Altman and Amodei therefore have reasons to cooperate when their organisations face similar risks. Governments, meanwhile, have several tools they could use to enforce a slowdown - potentially giving AI companies greater confidence that their competitors are doing the same. They could control access to advanced chips, mandate independent evaluations, require companies to disclose information about major training runs, and impose penalties when serious harm results from carelessness. Whether governments are willing to use those levers, however, is another question. Many of the key AI labs are in the US, where President Donald Trump has rejected calls for a slowdown amid concerns that China could gain the upper hand in the technological race. Amodei’s proposal runs counter to the US administration’s vision for AI, said Mr Enriquez. "The modus operandi of the Trump administration in AI was we have to chase the frontier at every step of the way, or else China will get there first ... and that would be disastrous for the US and its allies.” The two countries could potentially reach narrow agreements where their interests converge – such as prohibiting the development of AI-enabled bioweapons – but a broader consensus on slowing the pace of AI development is less likely, Prof Hanh said. “A broad agreement runs into a real zero-sum problem. If you hold back and the other side doesn't, you lose,” he told CNA. Mr Enriquez said diplomats may even have faced an easier task negotiating Cold War-era nuclear treaties than they would trying to incentivise a global AI slowdown. Unlike the nuclear arms race, the AI race is not driven solely by military dominance. Economic gain is also a major incentive, with proponents arguing that AI will accelerate economic growth, create jobs and generate value. While the Cold War nuclear race revolved around deterrence and the concept of mutually assured destruction, the value proposition of the AI race is different, Mr Enriquez said. The range of parties driving the competition is also far broader. “This is not only statewide competition. This is also corporate industry competition, and that can get really ruthless as well,” he said. “The list of players that need to be aligned is so much bigger than what they were when the nuclear treaties were being signed.” Mr Ang said the current level of distrust between nations could mean cooperation emerges only after a more serious incident. “It may take a major AI-related incident, with serious impact to life, to bring them together to cooperate, assuming it does not drive them further apart,” he said. Are major AI companies exaggerating the risks to slow competition? There is also a more cynical interpretation of the calls for caution. Critics have questioned whether the recent warnings from Anthropic and OpenAI are partly intended to generate hype around the capabilities of their products. Both companies had been expected to launch their first IPOs this year, although OpenAI said last week that it would not go public in 2026 because of safety concerns surrounding AI. An industry-wide slowdown could also protect today's leading firms from new competition, Mr Ang pointed out. That does not mean efforts to slow development should be rejected, he said. But any internationally agreed rules and enforcement mechanisms would need to ensure that the biggest players themselves remain subject to scrutiny. “While techno-libertarians often argue that rules slow innovation, and techno-determinists often argue that technology cannot be controlled, the reality is that society has always had to develop rules, law, and norms to shape how we use technology,” he said. Mr Enriquez said that even if some of the risks raised over the past week prove to be overstated, stronger safety infrastructure and independent oversight of AI development are still needed. “We are in a moment where at least they (leading AI companies) look like they are open to that regulation … (and) oversight, which is sorely lacking,” he said. “It would be remiss to not capitalise on the moment.” At the same time, experts said it was important to consider who would ultimately benefit from a slowdown. Larger companies have the money, legal resources and influence to absorb compliance costs more easily than smaller players, said Mr Dutt. “If the result (of a slowdown) is a rulebook smaller players cannot meet, the language of safety starts functioning as a market barrier,” he said. “That does not make every call for caution cynical, but it is worth asking each time: who is actually being asked to slow down, and who is still free to keep moving?” Want an issue or topic explained? Email us at digitalnews [at] mediacorp.com.sg. Your question might become a story on our site.