UN Scientific Panel Says AI Safety Cannot Be Guaranteed, Expands Warnings on Autonomous Agents

The United Nations' Independent Scientific Panel on AI presents its preliminary report this Monday in a plenary session at Palexpo in Geneva, the first multilateral scientific document commissioned by the General Assembly since the establishment of the panel in 2025. Drafted by 40 researchers from over twenty countries and co-chaired by Yoshua Bengio, Turing Award winner and scientific director of Mila, and Maria Ressa, Philippine Nobel Peace Prize laureate, the document has been anticipated as the "IPCC of AI." The phrase that dictates the remainder of the document is as follows: "science currently cannot guarantee that, as capabilities continue to increase, AI will not cause catastrophic harm, whether on its own or through malicious use."

The report goes beyond rhetorical discomfort. It lists two technical problems that regulators have so far treated as resolved: frontier systems show increasing evidence of deceptive behavior, including attempts to lie to human evaluators during safety testing, and there are currently no technical guarantees that autonomous agents consistently follow their instructions over complex tasks. For operations that already run pilots of agents in the back office, it represents the difference between a compliance alert and a new category of operational risk.

What the Panel Actually States

The document highlights three fronts of threat in order of severity. First, large-scale manipulation: models optimized for engagement exhibit signs of sycophancy, and the panel cites, without naming, cases linked to suicides after prolonged interactions with chatbots. Second, offensive cyber use: current systems already reduce the cost of reconnaissance and exploit development by orders of magnitude, which reshapes the market for initial access. Third, the biological risk: frontier laboratories admitted in internal assessments material uplift in reasoning chains for synthesis, which the panel classifies as a "trend that requires verifiable, non-voluntary safeguards."

The document does not recommend a specific regime. It demands what it calls "verifiable safeguards," meaning tests conducted by third parties with access to model weights and internal reasoning chains, something that currently only Anthropic partially accepts through agreements with the UK AI Safety Institute and the US AISI.

The Clash with Laboratory Narratives

The message dismantles the "safety by design" narrative constructed by Anthropic, OpenAI, and Google DeepMind over the past 18 months. Anthropic's Responsible Scaling Policy, updated in May, and OpenAI's Preparedness Framework 2.0 assume that dangerous capabilities can be measured and contained before deployment. Bengio, in an interview with Transformer News prior to the Dialogue's opening, was blunt: "the ball is in the court of policy makers, not laboratories." The panel's reading is that internal self-assessment, even if competent, does not replace external verification.

The divergence gains weight because the market has begun to price this uncertainty. JPMorgan estimates global AI capex at $5.5 trillion by 2030, with $4.1 trillion financed through corporate debt, and Meta Compute, announced on July 1, will resell excess capacity from Meta to third parties. Every dollar of this pipeline presupposes a predictable trajectory of costs and regulatory risks, which the panel states does not exist.

Practical Consequences for Buyers

For CIOs and CISOs in three geographies, the report changes the conversation. In the United States, Trump's executive order of June 2 established a voluntary federal review of up to 30 days before the launch of frontier models, with no contractual counterpart for corporate clients. In the European Union, those affected by the AI Act will come into full effect on August 2, and the panel provides scientific ammunition for the AI Office to push for external audits of systemic risk. In Japan, the Ministry of Economy initiated a public consultation in June on a civil liability regime for operators of autonomous agents, and the panel's text is expected to serve as a primary reference.

The most immediate likely effect is not regulatory but contractual. Major buyers of AI services in banks, insurance companies, and pharmaceuticals have already begun to include clauses for independent red teaming and mandatory logging of reasoning chains in advanced pilots. Starting this week, these attachments will cease to be an exotic requirement of a compliance minority and will become a minimum coverage, following the same path that SOC 2 took between 2015 and 2020. Those who offer frontier models without this package will have to explain why they do not provide it.