Why AI's own builders are suddenly begging you to slow down

If you've noticed AI safety researchers sounding more alarmed than usual this week, you're not imagining it. A resignation at Anthropic has set off a wave of public warnings from staff at both Anthropic and OpenAI, and the language they're using is blunter than anything you've seen from inside these labs before.

An Anthropic researcher announced his departure this week, accusing both his former employer and OpenAI of gambling with people's lives and warning that those building the technology privately believe it could kill us all by the end of the decade.

Anthropic's alignment lead responded publicly that he personally estimates the odds of that outcome at over 10%. Within days, staff at both companies, including members of OpenAI's safety team, said they now support slowing development down.

In this regard, what comes into focus is recursive self-improvement, the point at which the AI system has reached the level of competence that allows it to enhance itself without much human intervention.

It is only natural to ask the question as to why this particular risk of AI is more worrisome than any other risk associated with artificial intelligence systems.

You've likely already seen some of the warning signs building. Recent months have brought several publicised cybersecurity incidents involving AI models from both companies, and roughly 1,400 researchers across OpenAI, Anthropic, Meta and Google DeepMind signed an open letter in July asking the US government to help pace frontier AI development deliberately.

Lawmakers have introduced competing responses, from the FRONTIER Act's regulatory framework to a more drastic proposal that would pause advanced AI development outright until safety rules exist. None have gained clear consensus yet.