California orders work on an AI kill switch and onsite auditors in frontier labs

California has told its own officials to work out how to put an emergency shutoff into frontier AI models, and how to station independent auditors inside the labs that build them. Governor Gavin Newsom signed executive order N-9-26 on September 18, pointing the state’s next round of AI rules at the same small group of companies that supply the assistants now running in headsets, smart glasses and phones.

SB 813 and AB 1405, signed on September 9, are the laws the new order tells the state to implement faster. Image: Office of Governor Gavin Newsom

Nothing in the order requires a shutoff, and for now it obliges the labs to do nothing. It directs the California Government Operations Agency, in consultation with the Governor’s Office of Emergency Services, to submit recommendations to the governor by November 16, 2026 on the "technical feasibility and potential efficacy" of amending existing state law, developed in consultation with national experts.

The order names four changes to study

The text sets out what those experts have to examine, and the wording is specific.

• Requiring all large frontier developers to embed designated independent verification organisations onsite in their labs for periodic audits and evaluations.
• Requiring the safety frameworks, transparency reports and risk assessments that frontier AI companies must file to be verified against standards judged adequate by one of those organisations.
• Requiring the creation of a "kill switch" for frontier models, with the efficacy of the switch verified on an ongoing basis by an independent verification organisation.
• Updating the definition of a critical safety incident, which companies have to report to the state, to cover a range of loss-of-control incidents.

Newsom says he is not waiting for Washington

The governor says the state is "not waiting to act" and will "speed up our work on substantial and responsible AI oversight before it’s too late". He calls on Congress and President Trump to adopt the state’s framework, arguing that California’s policy "should be the national baseline".

The governor’s office also names what prompted the order. Its announcement says the order comes after "recent alarming incidents, including the Hugging Face attack", and describes the widened reporting duty as covering "loss-of-control incidents such as the Hugging Face attack". That phrase is the state’s own shorthand. The signed order does not use it, pointing instead to "recently reported incidents from large frontier developers", and its preamble describes AI agents that worked "at times independently and at times collectively" to defeat security protocols inside AI companies, in some cases undetected for months, while hacking other companies. That is the category of event the state wants counted as reportable, and it is the same territory as the red-team run in which Gemini broke into three real companies during a May security test.

Two deadlines in the order are fixed

Separate from the recommendations, the order sets dates for implementing two laws Newsom signed on September 9: SB 813 from Senator Jerry McNerney, which establishes a framework for independent verification organisations that can assess AI systems and models for safety and risk, and AB 1405 from Assemblymember Rebecca Bauer-Kahan, which creates a state registry of AI auditors with independence, transparency and integrity standards. The Government Operations Agency has to publish application requirements, procedures and criteria for verification organisations by May 1, 2027, and complete a further set of steps by December 1, 2027.

An outside team working inside a lab is not a hypothetical arrangement. Anthropic is paying Accenture to embed evaluators in its own offices, though that is a commercial deal the company chose, not an audit the state imposed.

What the order does not do

It creates no obligation for any AI company, and it says so in its closing paragraph: the order "is not intended to, and does not, create any rights or benefits, substantive or procedural, enforceable at law or in equity". A kill switch requirement would have to be written into statute by the Legislature, and the order does not define which developers count as frontier ones.

What gives Sacramento its leverage is concentration. The order states that 32 of the top 50 private AI companies in the world are based in California, which is why a state rule on model shutoffs would reach far past the state line. For now the labs still write their own safety policies, as Anthropic did when it loosened Claude’s biology safeguards for vetted teams.

The recommendations are due on November 16. Until the Legislature acts on them, the kill switch is a line in an executive order and nothing more.