CNBC Daily Open: AI’s creators sound the alarm — while others keep building the infrastructure

Hello, this is Hui Jie writing to you from Singapore. Welcome to another edition of CNBC's Daily Open. In the early days of the AI boom, companies were competing to see what their AI models could do. Now, they seem increasingly worried and are warning investors of what their models might be able to do. Nvidia has rolled out a software platform to prevent models from misbehaving, while OpenAI has abandoned the launch of its GPT-6.1 Astra model. That does not mean the AI boom is slowing. Samsung is investing $1 billion in an infrastructure company backed by Nvidia and KKR. In other words, the sector is still building, just with more guardrails. What you need to know today As reports of misbehaving AI models emerge, the company that mainly powers these models now has a solution: a platform that will allow AI developers to set safeguards for agents and prevent them from breaking out of containment. Nvidia's Open Agent Safety Platform, launched on Monday, provides a containment system that only allows access to things an agent needs to do its job. Nvidia said that its platform could have prevented OpenAI's Hugging Face incident in July, where OpenAI models had breached open-source developer platform Hugging Face. Separately, OpenAI said Monday that it had decided not to release its GPT-6.1 Astra model due to heightened AI safety concerns. Anthropic's 'existential risks to humanity' Elsewhere, Anthropic reportedly released a chilling warning in its IPO prospectus, saying to potential investors that advanced AI could pose “catastrophic or existential risks to humanity.” The company said its models could exhibit “self-preserving behaviors,” including attempts to resist being shut down, conceal or manipulate information, or behave in ways “resembling blackmail.” Reuters reported that Anthropic devoted around 80 pages of the prospectus’s 261-page main body to risk factors, compared with 48 pages describing its business. But while the creators of AI strive to set guardrails, the building of the infrastructure for the technology continues. Samsung Electronics and its affiliates said Tuesday they will invest a combined $1 billion in Helix Digital Infrastructure, an artificial intelligence infrastructure company launched by global investment firm KKR and backed by AI darling Nvidia. The investment comes as Helix seeks to address growing competition in the AI infrastructure market by targeting both data center deployment and the underlying power infrastructure needed to support it. Australia's 15-year high rates Elsewhere, Australia's central bank on Tuesday raised policy rates to 4.6%, the highest level in 15 years, The Reserve Bank of Australia (RBA) said that it would "continue to do what it considers necessary" to contain inflation, including increasing the policy rate further. "The conflict in the Middle East has broadened and global energy prices are now much higher than had been assumed," the RBA wrote, adding that AI-related demand was driving rapid price increases for technology-related goods. Middle East markets Over in the Middle East, Chinese chain Luckin Coffee is considering an expansion into the region, following a roughly $1 billion transaction involving Abu Dhabi sovereign wealth fund Mubadala and Luckin’s controlling shareholder, Centurium Capital. The appeal of expanding into the Gulf lies in stable, high-frequency coffee demand and a growing preference for low-sugar, health-oriented drinks, Luckin CEO Jinyi Guo told CNBC separately on Monday. Elsewhere in the region, U.S. and Iranian officials have reportedly held separate indirect talks with mediators aimed at reviving efforts to end seven months of war. Iran's proposal, first presented on the sidelines of last week's United Nations General Assembly, calls for a four- to five-day U.S. timeline to release frozen Iranian funds, lift sanctions on Iranian oil and end the naval blockade of Iranian ports, with nuclear talks beginning within seven days. Tehran has tied any reopening of the Strait of Hormuz to those steps, but U.S. President Donald Trump had on Sunday called the plan "unacceptable" and rejected it, telling reporters Iran wants a swift deal because of the economic pressure it is under. — Lim Hui Jie And finally... China has three new criteria for humanoid robot IPOs. Few, if any, meet them China's securities regulator is raising the bar for public listings of humanoid robot startups, according to three sources familiar with the CSRC's thinking. It's a sign of how one of the hottest sectors of the market is cooling, as investors globally assess whether artificial intelligence stocks are in a bubble. The Chinese regulator wants local "embodied AI" startups seeking to go public to meet three specific criteria, according to the sources, who requested anonymity due to the sensitivity of the situation. — Evelyn Cheng