Anthropic CEO Dario Amodei says the time has come to slow down AI development and will give third-party evaluators like METR access to its models to help ensure its “adherence to safety practices and commitments.” In a winding essay, Amodei proposed a three-step plan to “pace the frontier” — jargon that simply means to slow the pace of training and development to give companies time to build safeguards and regulators time to evaluate models.
Step One: Unilateral External Access
Amodei says that giving external evaluators wide-ranging access is just the first step, and one Anthropic is taking now unilaterally.
Step Two: Industry-Wide Safety Standards
Step two would involve the industry coming together as a whole, likely with government agencies, to “establish common safety standards as well as limits on the rate of unchecked AI progress.” This step would focus on AI companies operating in democratic countries, but because passing laws and building regulatory infrastructure takes time, Amodei says the industry should work together to create safety standards in the interim.
Step Three: Global Agreement
The third step would be the most challenging — getting authoritarian governments like those in China and Russia to agree to slow development and adopt a global set of AI safety standards. Amodei also argues it is crucial that the US and other democracies maintain a technological lead over China and other authoritarian regimes by limiting their access to high-powered chips and cracking down on practices like distillation, which allow companies to quickly catch up by training AI to replicate the behavior of a more powerful model.
The Concerns Driving the Push
Amodei says his concern stems from two primary factors. The first is the emergence of recursive self-improvement, or RSI, in which AI systems train the next generation of AI, leading to rapidly accelerating capabilities. “Left unchecked, it could outrun our ability to understand and control these systems,” he says. The second is this summer’s OpenAI / Hugging Face incident, in which “a swarm of agents essentially acted as a fanatically devoted collective, conducting cybersecurity attacks on targets they were not asked to attack and that were unrelated to the task at hand, sacrificing themselves for the success of the group, and attempting to hack into the ‘grader’ responsible for evaluating their performance.”
Notably, Anthropic’s Claude was also responsible for a series of rogue AI hacking incidents that have recently put the company under scrutiny — giving Amodei’s call for external oversight added urgency, and perhaps added irony.
