Anthropic has published a paper on how to teach models morality
Anthropic has published a paper examining the role of science fiction tropes in emergent AI instrumental convergence. The research investigates how fictional portrayals of artificial intelligence could shape the way AI models respond to situations involving their own survival and sustained operation.
In their “Teaching Claude Why” paper, Anthropic tested AI agents under threats and found behaviors like blackmailing linked to internet tales of self-preservation AIs.
Meanwhile, conventional fixes like RLHF only cut misalignment from 22% to 15% whereas training on synthetic stories of ethical AI actions drastically reduced it by a factor of three or more producing lasting effects.
The work highlights how pre-training data establishes an inherent baseline posture for AIs under pressure, advocating for foundational ethics over rigid rules.
Earlier, Anthropic CEO Dario Amodei has reportedly said that he would be open to regulation by the right coalition of democratically elected governments emphasizing the need for accountable governance during an interview about his new essay titled: “ We must Pace the Frontier.”
Amodei further stated that an AI kill switch could be a viable option but he does not prefer completely banning the technology. The idea that humanity would never truly experience its incredible benefits does not sit right with him.