AI leaders urge global pause on self-improving models

Some of the world's most prominent artificial intelligence researchers are urging governments to impose a coordinated global slowdown on AI development, warning that safety measures are failing to keep pace with systems now capable of improving themselves without human input.

The calls come amid growing alarm within the industry itself, after a whistleblower came forward with claims that leading AI laboratories harbour serious internal fears — including the possibility that unchecked development could pose an existential threat to humanity.

At the centre of the concern are so-called self-improving models: AI systems that can modify and retrain themselves, potentially accelerating their own capabilities far beyond what their creators originally designed. Researchers argue that once a model can meaningfully enhance its own architecture, traditional oversight frameworks become difficult to apply and even harder to enforce.

The whistleblower's account, which has circulated among policymakers and AI safety advocates, reportedly describes a culture inside certain technology firms where competitive pressure to ship more powerful products is overriding caution. Staff with reservations about safety, the account suggests, have faced pressure to remain quiet.

AI leaders who have gone public with their concerns are not calling for an end to development outright. Instead, they want binding international agreements that would require safety evaluations to be completed and independently verified before any organisation deploys a new generation of self-improving technology. The analogy drawn by several researchers is to nuclear non-proliferation treaties — frameworks designed not to ban a technology but to ensure it cannot be weaponised or misused without global accountability.

Governments have so far moved at uneven speeds. The European Union's AI Act, which came into force this year, sets baseline requirements for high-risk systems, but critics say it was written before self-improving architectures became a practical concern. The United States and United Kingdom have each established AI safety institutes, though neither has the authority to halt development unilaterally.

Australia has yet to introduce binding AI legislation, though the federal government has signalled it is watching international developments closely. Tech policy advocates in Canberra have pointed out that Australian agencies and businesses are already deploying AI tools built on the very model families now under scrutiny abroad.

The debate is intensifying as the gap between what AI systems can do and what regulators understand about them continues to widen. For many researchers, that gap — not any single capability — is the most urgent problem. A system does not need to be malicious to cause serious harm; it only needs to pursue objectives that humans failed to specify correctly, at a speed and scale that makes correction difficult.

Whether governments will act collectively before the next generation of self-improving models reaches deployment remains the defining question of the moment.