Pressure Mounts for Controls on Superintelligent AI as Insider Warnings Intensify

A series of high-profile resignations, legislative proposals and international lobbying efforts has sharpened the debate over whether artificial intelligence systems capable of self-improvement should be subject to binding controls – placing questions of safety, pacing and governance at the centre of the industry’s agenda.

The most recent flashpoint followed the resignation of Jacob Coxon, an AI researcher who had worked at both Anthropic and OpenAI, who left the industry this week and published a public warning on the social platform X. “Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives,” Coxon wrote. He further stated that “the people building AI earnestly believe that it could kill us all by the end of the decade,” and called for pacing agreements between laboratories developing frontier models.

Coxon’s departure adds to a growing pattern of dissent within the sector. Anthropic has separately suggested a global pause on building the most powerful AI systems, citing signs that the latest models could begin to escape human control. The company has also warned that full recursive self-improvement – AI systems enhancing their own capabilities – may increase the risk of “humans losing control” of the technology.

Legislative Responses on Both Sides of the Atlantic

In the United States, Senator Bernie Sanders and Representative Greg Casar announced the Ban Artificial Superintelligence Act on 3 September. The proposed legislation would permanently prohibit the development and deployment of superintelligent AI and temporarily pause advanced AI development until a federal regulator is in place to monitor for risks. The proposal has drawn criticism from organisations such as the Information Technology and Innovation Foundation, which argues that a sweeping ban would sacrifice American innovation and hand China a strategic advantage without eliminating underlying risks.

Comparable measures are under discussion in the United Kingdom, while Control AI – a lobbying group advocating international regulation of the technology – convened a session of scientists and politicians earlier this week to press for curbs on artificial superintelligence.

Industry Leaders Push Back

The accelerationist position has not gone quiet. Meta increased its capital spending plans for the year by 73% in pursuit of superintelligence, and Chief Executive Mark Zuckerberg published a roughly 6,500-word manifesto in August defending continued development. In it, he argued that the most significant risk is not the technology itself, but the concentration of power over it in the hands of a single government or individual.

OpenAI has also put forward a comprehensive policy vision, with Chief Executive Sam Altman describing superintelligence as a development significant enough to warrant a new social contract. Critics, however, have characterised the proposal as lacking a clear path to implementation.

A Question of Definition and Timing

For industry professionals, the debate underscores a persistent practical problem: there is no agreed technical definition of superintelligence, which complicates both regulation and enforcement. Proposals range from international treaties and federal oversight to voluntary pacing commitments and iterative, proportional risk-governance frameworks of the kind Anthropic has adopted in its Responsible Scaling Policy.

What remains clear is that the window for establishing governance norms is narrowing. With front-line researchers now publicly questioning the trajectory of their own employers, and legislators moving from discussion to draft law, companies developing frontier systems face mounting pressure to demonstrate that safety commitments can keep pace with capability gains. The outcome of this contest between acceleration and control is likely to define the operating environment for the AI industry well beyond this decade.