Anthropic urges AI industry to build emergency pause for powerful systems

Anthropic is pushing the clearest public test yet of whether the AI industry is willing to build an emergency brake for itself. The company wants the world’s biggest labs to coordinate a way to slow down or temporarily pause development if advanced systems start to look too dangerous, a proposal that puts one question above all others: who decides when the line has been crossed?

The call lands as Anthropic argues that AI capabilities are advancing so quickly that human control could eventually slip, especially as models get better at software tasks such as coding on their own. Reuters described the proposal as a coordinated and verifiable pause if systems begin improving themselves faster than society can manage the risks. AP-linked coverage said Anthropic warned that task performance could double about every four months, a pace that would leave little room for error if safety systems fall behind.

Anthropic is not arguing for a permanent shutdown. Instead, it wants a credible mechanism that can be used if the technology becomes destabilizing, with research and governance tools in place before a crisis hits. The company said its internal research institute will examine the issue with others and take actions meant to help create that slowdown system, though it did not spell out what those actions would be. That makes the practical challenge even sharper: what evidence would trigger a pause, and how would competing companies trust one another to comply?

OpenAI has offered a different answer. Reuters and AP-linked coverage said the company argued that democratic governments should define the safeguards and accountability rules, not private labs acting alone. That puts the debate at the center of a larger struggle over AI governance, with one side pressing for voluntary industry coordination and the other insisting that elected governments must set the guardrails.

The urgency has grown alongside cybersecurity fears. On June 2, University of Toronto researchers said they had built a self-adapting worm in a secure digital lab using publicly accessible AI models. The team said every online device is a potential target. The arXiv paper, AI Agents Enable Adaptive Computer Worms, says the worm can tailor attack strategies to each target and use stolen compute from infected machines, creating what the authors describe as a dangerous economic asymmetry for defenders.

Anthropic’s own policy work has already been moving in that direction. On June 4, the company published What we learned mapping a year’s worth of AI-enabled cyber threats, underscoring how central cyber risk has become to its case for a pause mechanism. Its internal research institute says it exists to understand and shape the consequences of powerful AI systems, and its newsroom has recently paired policy and safety work with product launches and expansion announcements. For policymakers, the issue is no longer whether AI needs oversight. It is whether the industry and governments can agree on a real brake before systems outpace the people trying to contain them.