Anthropic co-founder warns AI industry needs 'brake pedal' now

The AI industry is racing toward systems that can improve themselves without human involvement, and it has built no mechanism to slow down or stop if something goes wrong. That is the warning Anthropic is now making publicly, and it is coming from the company's own co-founder.

In a blog post published by The Anthropic Institute, co-founder Jack Clark and institute leader Marina Favaro warned that "full recursive self-improvement" AI systems capable of autonomously developing their own successors are closer than the industry previously assumed.

"If systems are capable of fully building their own successors, the ways we secure them, monitor them, and shape their behaviour all grow much more important," they wrote.

The authors stopped short of predicting catastrophe, but they did not dismiss it, and they called on companies to consider slowing or pausing frontier AI development to allow safety research to catch up.

In his Thursday night appearance on CNN, Clark framed his argument using language intended to communicate to a lay audience. "When I look down at the car we're driving, all I have is a gas pedal. I don't have a brake pedal," Clark told Anderson Cooper.

When Anderson Cooper asked if he was concerned about what he referred to as “AI kills humanity” scenarios portrayed in science fiction, Clark didn’t deflect the question. "Yeah, we read the science fiction and watch science fiction here as well, so it's not lost on us."

The natural rebuttal to Anthropic’s appeal for collective action comes in the form of a competitive dynamic that is very much alive between AI companies today as well as Anthropic, which has gone public with its filing for an IPO that could amount to tens of billions of dollars.

But Clark rejected this idea by looking to history for evidence, stating that “during the height of the Cold War, during highly volatile interactions between countries that are competitors, there were still ways in which they could stabilise certain elements of the arms race.”

He believed that, if there was the will, it could be done again with AI as well as competing companies.