Governing AI Agents

Advancing cyber capabilities and growing AI autonomy increase the risk of unsanctioned agent behaviour. Despite emerging evidence of these risks, global AI regulation remains difficult. Climate change and nuclear risks offer useful points of comparison (as both involve transnational risks requiring international cooperation). So, what can we learn from them, and where do the analogies break down?

Frontier AI models (the most advanced general-purpose models at a given moment) now perform at or above human levels on advanced science, mathematics, and multimodal reasoning tasks. But these abilities come with greater security and safety risks. The Hugging Face intrusion is proof that agents can discover unintended channels to communicate and coordinate, exploit system weaknesses, and bypass constraints in pursuit of their objectives. Other similar instances of AI agents deploying malicious code and manipulating people have been identified by the AI Security Institute. Moreover, evaluation awareness can make AI safety tests less reliable because models may hide risky capabilities during testing. These raise genuine concerns about the potential of autonomous AI systems to exploit vulnerabilities in, or disrupt, critical infrastructure. Yet, approaches to AI governance remain fragmented. States (and industry actors) differ over whether and how to regulate. So, why is collective action so difficult despite shared risks?

Economic and Geopolitical Incentives

Frontier AI labs (e.g., Anthropic, OpenAI, xAI and DeepSeek) are key private players in the AI race. To develop powerful models, they require substantial investment, creating pressure to remain competitive and profitable. While slowing development may improve safety, unilateral restraint creates a competitive disadvantage. Therefore, labs have little incentive to unilaterally slow down. Regulation could address this collective action problem, but states themselves face a similar dilemma.

The United States and China are the key players, driven by geopolitical primacy. If one slows down for safety, it risks falling behind the other. In the absence of trust, this fear pushes both sides to continue the AI race (even when mutual restraint would benefit both). This dilemma is evident with fragmented AI regulation across major jurisdictions. The US has no comprehensive federal AI statute. The federal government has also supported private actors in challenging state regulation. As a result, Colorado has now repealed and replaced its AI law with a narrower regime.

The EU is not a frontier AI power (compared to the US or China). However, its large market, strategic technology companies, and risk-based regulatory approach give it influence over AI governance. Yet, despite adopting the first comprehensive AI regulatory framework, it has faced industry pressure, alongside delays to key provisions governing high-risk AI systems. In a similar vein, the UK hosts AI infrastructure for global companies, but its approach to AI regulation is a “light touch”. It has no comprehensive AI-specific statute. More importantly, its AI Security Institute assesses frontier AI risks but lacks statutory regulatory powers.

So, if economic and geopolitical pressures fragment domestic regulation, can global cooperation overcome the same collective-action problem? Climate and nuclear governance offer useful insights.

Climate Governance

Both climate change and frontier AI create transnational negative externalities. Greenhouse gas (GHG) emissions cross borders, as can harms arising from AI models developed elsewhere. However, the collective-action problems they present take different forms.

“Climate change mitigation is a global public good”. Its benefits are shared globally, while its costs are borne by those reducing emissions. This creates a free-rider problem. States can benefit from others’ efforts while limiting their own action to protect national economic interests. International climate governance has struggled with this problem. For example, the Kyoto Protocol imposed binding emission targets only on developed countries. So, the US did not ratify it on the grounds that major developing emitters such as China and India faced no binding targets. The Paris Agreement also leaves scope for free riding because its nationally determined contributions are not legally binding.

Frontier AI presents a different problem. Unlike climate mitigation, its benefits (except for open-weight models) can be excluded. For example, a US government directive suspended non-US nationals’ access to Mythos 5 and Fable 5. Furthermore, the capacity to develop and control frontier systems is concentrated among a small group of states and firms. More importantly, the source of risk is concentrated. While all states contribute to global emissions, some far more than others, frontier AI risks arise mainly from actors capable of developing and controlling these systems. Therefore, the principle of Common but Differentiated Responsibilities and Respective Capabilities is not directly applicable. Although scientific participation should remain global (as seen with the UN Independent International Scientific Panel on AI), regulatory responsibility need not be equally shared. Instead, responsibility should reflect technological capacity and control over the source of risk.

Cooperation could begin by identifying systemic risks and the capabilities that warrant regulatory intervention. Climate governance had a clear objective. The Paris Agreement aims to keep warming well below 2°C and pursue efforts to limit it to 1.5°C. Frontier AI has no comparable agreed threshold.

The Council of Europe’s Framework Convention on AI, not yet in force, sets no common quantitative risk thresholds. This leaves each Party to decide when to act. Without a common yardstick, countries may end up with very different rules. Contrastingly, the EU’s AI Act sets a numerical trigger. A general-purpose AI model is presumed to have “high-impact capabilities” when its training compute exceeds 10²⁵ FLOP (Article 51(2)). It is then treated as posing “systemic risk”, triggering notification and additional obligations u/A 52 and 55. However, the Act recognises that the “relationship between compute and risk is highly uncertain and rapidly changing”. Therefore, it allows the Commission to consider other factors including capabilities, number of users and scalability, and to amend the threshold as technology evolves.

Currently, there is no international agreement on quantitative AI risk thresholds. Numerical triggers exist in some US states and South Korea. A 2026 Executive Order directs federal agencies to develop a “classified benchmark” for models with advanced cyber capabilities. China uses tiered risk grading without numerical thresholds. What is missing is not a single fixed number but a shared, adaptable baseline. Achieving this appears unlikely in the short term unless existing trust deficits are addressed. Nuclear governance shows how cooperation can emerge despite mistrust and strategic rivalry.

Nuclear Governance

Nuclear governance offers a closer analogy for frontier AI than climate governance. Both are tied to national security and geopolitical rivalry. There was a race then; there is a race now.

The similarities go further. Both can cause catastrophic harm: from nuclear detonation to the use of frontier AI to facilitate development of biological or chemical weapons. Both depend on scarce, controlled inputs – fissile material for nuclear weapons and advanced chips for frontier AI. Capabilities are also concentrated. China and the US account for 90% of the computing power ‘among the world’s top 500 AI supercomputers’.

This raises the question: can meaningful cooperation emerge between two rival powers? Even at the height of the Cold War, the US and the Soviet Union managed to agree on rules. This suggests that strategic rivalry does not preclude cooperation, although it makes cooperation more difficult by creating a prisoner’s dilemma. Both would benefit from mutual restraint, but the fear of falling behind encourages each to keep racing. The US AI Action Plan describes it as “a race to achieve global dominance”. Whereas China, constrained by chip controls, competes through wider adoption and low-cost open-weight models (not open source, as training data and methods are not disclosed). Each strategy is rational on its own. Together, these expose both to cross-border risks neither can manage alone. Once released, these models can be downloaded and modified, increasing the risk of misuse.

However, their situation differs from a classic prisoner’s dilemma. The classic game is played once, and the prisoners cannot communicate. In contrast, the US and China can communicate and interact repeatedly. This repeated interaction allows reciprocity, and recent discussions on a formal AI dialogue and an incident hotline offer evidence of this. However, this requires trust in each side to know whether the other is keeping its word. A winner-takes-all mindset undermines that trust. There is a deeper problem in this rhetoric of “winning”. It assumes the winner can control what it builds. The US AI Action Plan acknowledges that “the inner workings of frontier AI systems are poorly understood”. Likewise, China’s AI Safety Governance Framework accepts that the “speed and direction of technological evolution may exceed human anticipation and control”. Therefore, if control is lost, neither side wins. In this context, the game may then look more like a Stag Hunt. Each side has reason to exercise restraint, only if it can trust the other to do the same. The only obstacle is trust (not incentives).

Insights from the Cold War show that a shift is possible. In 1985, Reagan and Gorbachev agreed that “a nuclear war cannot be won”. For nuclear weapons, China has also endorsed this principle. There are other reasons cooperation may be possible. Growing public scepticism about AI in Western democracies, including the US, can affect the incentives of elected governments. For China, its export-oriented economy raises the costs of instability. More importantly, US and Chinese capital are also intertwined in AI. Therefore, their economic interdependence gives an interest in stable relations.

The real question is how quickly cooperation can make restraint rational before serious harm occurs. Nuclear cooperation often followed moments of crisis. The Washington-Moscow hotline came after the Cuban Missile Crisis. Frontier AI should not require a similar shock. There are already signs of cooperation. In 2024, the US and China agreed that humans (not AI) should control decisions to use nuclear weapons. Recently, the US has also proposed a notification mechanism for AI incidents.  Cooperation could extend to sharing incident reports and commitments against using AI for cyberattacks. Such measures could gradually build trust.

However, bilateral progress, in the absence of trust, may still be slow. Other states can help. The 1963 Limited Test Ban Treaty began with the US, Soviet Union and UK and later gained much wider participation. The EU, UK, Canada and Japan could play a similar role in AI governance. None lead in frontier AI, but all are key players. Much of their influence comes from what frontier AI depends on. The EU, for example, offers a single market that both American and Chinese firms want to sell into. It is also home to the Dutch firm ASML. Its EUV technology – used to make the world’s most advanced chips – is “unique to ASML.” That gives Europe real leverage over the global chip supply chain.

Acting together would strengthen their influence, and ties are already growing. Canada and the EU have agreed to “explore opportunities to cooperate with like-minded partners.” The UK is connected to the EU’s AI infrastructure. UK-based researchers and firms can apply to use its AI Factories (hubs for developing AI models). Japan is deepening AI and semiconductor cooperation with the EU, and its firms produce most of the world’s high-end photoresists (key chipmaking chemicals). All four belong to a UK-coordinated international network that develops shared AI testing methods.

Testing matters because rivals rarely trust promises they cannot check. The 1963 treaty banned only the nuclear tests each side could already detect with its own systems. Underground tests were left out because the US and Soviet Union could not agree on how to verify them. AI poses a similar challenge. A US-China AI agreement would likewise need shared tests, which the network could help supply. China may distrust tests designed by US allies, but open methods would let each side run them independently. Each test that confirms compliance would build trust and, over time, make restraint rational for both. None of this means that other countries should be excluded. It simply reflects where the greatest risks, capabilities, and critical dependencies are concentrated. A small group can start the process. Wider global rules can follow.

The author is grateful to Shaun Mills, Former Associate Dean, School of Law and Policing, for his insightful discussions and encouragement. These discussions provided valuable motivation and inspiration for the development of this article.