Anthropic Proposal Would Limit AI Model Shutdowns to Extreme-Risk Cases

Summary

Anthropic is lobbying for a proposed U.S. AI safety bill that would let the government block an AI model’s release only when it could cause more than 50 deaths or injuries, or at least $1 billion in damages. The bill addresses risks involving the creation or release of a bioweapon, AI-powered or AI-enabled cyberattacks, and an AI lab losing control of a model or agents. An independent third party registered with the Department of Commerce would periodically evaluate the lab and its models; the lab could pay the evaluator only if that payment did not compromise independence. If a qualifying event appeared imminent, the evaluator would have 72 hours to notify the Commerce Department. The department would then present the evidence to a federal judge in the District Court for the District of Columbia, who would decide whether to halt the release. The proposal would also prevent any other U.S. department, agency, or officer from ordering an AI lab to stop releasing a model. Anthropic said the bill reflects policies in its Advanced AI Framework and described them as measures Congress should adopt. The article places the proposal amid growing concern about AI safety incidents, warnings from Anthropic CEO Dario Amodei, and other congressional ideas including mandatory kill switches and criminal penalties for developing AI superintelligence. The bill remains a proposal, so its thresholds, enforcement structure, and eventual status are unsettled.