Anthropic Proposes Court Orders for Risky AI Models
In a letter dated October 15, Anthropic told California Governor Gavin Newsom that the state should be able to ask a court to block AI models that pose a serious risk. Under the proposal, independent evaluators would report risks to California’s attorney general, who could seek a court order to pause a model if they saw a near-term risk of catastrophic harm. A court order could stop a model’s release, internal use, or further work. Anthropic also proposed outside safety auditors at leading AI companies, risk reports at least every six months from large developers, and targeted measures such as disabling an individual agent or suspending one customer’s access. An Anthropic spokesperson confirmed the letter was genuine; Newsom’s office had not immediately responded.
The letter is a policy proposal from an AI developer that would give California a new mechanism for seeking court orders against risky models. It is not an enacted court order or enforcement measure.