AI leaders urge caution at UN, with Anthropic chief pledging to slow down

AI leaders urge caution at UN, with Anthropic chief pledging to slow down Leaders of some of the world's biggest AI companies urged caution at the United Nations on Wednesday, warning that increasingly autonomous systems could outpace human oversight. Anthropic CEO Dario Amodei pledged to slow development to ensure new AI systems are safe, while OpenAI chief Sam Altman called for 'extreme care'. Leaders of major AI firms told the UN Security Council on Wednesday that the technology calls for heightened vigilance, with Anthropic's CEO pledging to slow down where needed to ensure the safety of new releases. "This moment calls for extreme care," OpenAI CEO Sam Altman said in New York at an emergency meeting of the 15-member council. He warned that as AI systems become more capable and autonomous, they could move faster than institutions or concentrate power into too few hands. Altman was among industry leaders briefing the council following weeks of global headlines about the risks of uncontrollable artificial intelligence and calls for a slowdown, which have been resisted by US President Donald Trump. "The industry must not accept too much technological risk just because the benefits are too great and important to slow down," Altman told the United Nations' top body. He also urged governments to work with each other and the industry on decisions shaping the technology's future. "In our history, there have been times where countries who compete and don't always like each other very much still come together for shared interests and the collective good in the face of a powerful new technology," he said. "We believe this must be one of those times," he added. Slowing down Speaking after Altman, Anthropic CEO Dario Amodei said his company understands that as AI models become more powerful, "even more stringent standards of safety will be required." He said Anthropic aims to improve its ability to prevent misuse of the technology. "We will slow down as much as necessary in order to make sure that every successive AI technology that we release is actually safe," Amodei said in a video feed. "If managed poorly, I even believe that AI could be a risk to humanity as a whole," Amodei added. Other speakers included Clement Delangue of Hugging Face, which was the target of a recent cyberattack by autonomous OpenAI software. This marked the first in a series of incidents that have raised alarm bells over the rapidly evolving technology. In early September, Jacob Coxon, a researcher who worked at both OpenAI and Anthropic, decided to leave the industry and accused both companies of "gambling with our lives" in the race to develop AI models capable of self-improvement. 'Middle path' The executives' remarks come shortly before Trump is due to welcome Chinese leader Xi Jinping, hosting him for a summit where AI will be on the agenda. On Wednesday, Altman said his company would try to walk a "middle path." "We don't want to fall into the trap of blind optimism," but he cautioned against falling into "doomerism" too. Yoshua Bengio, the Canadian computer scientist considered one of the founders of AI, told the Security Council that "more and more people are rightly worried about recent advances." "We must channel that concern into productive action," he said. Delangue of Hugging Face noted that while his company was targeted by AI, it also used the technology to fend off the attacks. "The world needs open-source AI more than ever to defend itself." Altman, Amodei and several other leading AI executives have been calling for global oversight of the technology. OpenAI has urged the United States to lead a global effort on establishing standards for AI, noting that its industry is at the technical frontier. Prime Minister Andy Burnham said Tuesday that Britain would launch an effort to set up global AI standards during its 2027 G20 presidency. France's Foreign Minister Jean-Noel Barrot similarly urged joint action within the UN framework and with companies to set rules for a new era. But diplomatic efforts could clash with the US position. Michael Kratsios, director of the White House Office of Science and Technology Policy, said rapid advancement is not a reason to pause AI's development or constrain it with new governance structures. Trump on Tuesday rejected international regulation of AI, calling such proposals a "globalist scheme." (FRANCE 24 with AFP)