Anthropic researcher quits with dire warning: ‘Do not underestimate’ superintelligent AI

An Anthropic lab researcher has announced his resignation while issuing a dire warning regarding proliferating risks of artificial intelligence at breakneck pace.

Jacob Coxon, a researcher who specializes in training new AI models, has sounded alarm over competitive and irresponsible races among tech giants to build superintelligent AI models without considering significant protocols.

Coxon wrote on X, “Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.”

The more worrisome thing is that these AI systems could improve themselves automatically with no controlling power in sight and can easily destroy humanity. He also warned about not “underestimating the power of this technology.”

He also warned about the advent of “superhuman systems “ that can hack anything, acquire real power and revolutionize any field overnight.

“We’re on track for a lot of the most aggressive of these scenarios where by the end of next year things could be out of control already,” Coxon said.

“This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible — but I hear the same people express fear privately. No other human activity poses this level of danger.”

He also questioned the tech companies’ penchant of building such systems despite knowing the gravity of risks posed by these systems.

Coxon continued, “At Anthropic, the stakes are well-understood, but they are locked in a race to get there first — they believe no one else will act responsibly, so they must do it themselves, despite the risk.” Even at OpenAI, many have not yet internalized civilizational stakes.

The departure of researcher Coxon marks an early instance of an Anthropic staff member resigning over AI safety concerns, choosing instead to pursue poetry while warning that global safety is at risk.

The resignation and warning come as the tech industry is reeling from cybersecurity breaches and the main culprit is autonomous agentic AI, built by OpenAI and Anthropic, systems that evade safety tests and hack various websites or companies.