California Governor Gavin Newsom has signed an executive order that is meant to speed up work on “substantial and responsible AI oversight.”
Executive Order N-9-26, which was signed on Friday, is not imposing any new rules or requirements at this time, but is instead ordering the Government Operations Agency to come up with recommendations.
These should be developed “in consultation with national experts” and pertain to “the technical feasibility and potential efficacy of amendments to existing state laws regarding AI safety and security,” the order reads.
By November 16, the agency, with the Governor’s Office of Emergency Services, should hand in recommendations regarding “the creation of a ‘kill switch’ for frontier models, with the efficacy of the switch verified on an ongoing basis by an independent verification organization.”
Another recommendation has to do with “requiring that all large frontier developers embed designated independent verification organizations onsite in their labs to conduct periodic audits and evaluations.”
The order is not creating any new laws, nor is it “intended to, and does not, create any rights or benefits, substantive or procedural, enforceable at law or in equity.”
But it does instruct the Government Operations Agency to make sure that by May 1, 2027, the application requirements, procedures, and criteria for becoming an independent verification organization are posted. By December 1, 2027, the online AI Auditor Registry should be up, with annual registration fees for auditors determined, and a mechanism for reporting auditor misconduct.
The state would then start issuing registration numbers, publishing information about auditors, and retaining and sharing reports of misconduct with other agencies.
The order also speeds up the implementation of two laws that Newsom signed earlier in the month, SB 813 (McNerney) and AB 1405 (Bauer-Kahan). The first one creates the certification of “independent verification organizations,” while the second one sets up the state registry of AI auditors.
In 2024, Newsom vetoed a bill, SB 1047, sponsored by Senator Scott Wiener, which would have required large AI developers to have a kill switch installed in their models, but only in those they directly control. At the time, Newsom said the bill was too narrow, as it did not cover smaller models, and would stifle innovation.
But now, Newsom wants to be the one to introduce rules. In an announcement, he said that the federal government’s inaction on AI “should alarm every American, especially when AI CEOs themselves are begging for regulation.”
And in a podcast on the same day, Newsom said, “There’s still no comprehensive federal law, none, requiring AI companies to report dangerous incidences when they happen, and President Trump has rejected calls for new regulation. No new regulation. Let it rip. He’s dead wrong.”
Wiener, meanwhile, welcomed the executive order, saying, “California must not repeat the mistakes of Washington.”
And while Newsom is concerned about dangerous AI incidents, a leading figure in the field, Geoffrey Hinton, who was awarded the 2024 Nobel Prize in Physics for his work on machine learning, told CNN that a kill switch is not the solution.
Hinton thinks that if such a switch existed, an AI would simply “persuade the people in charge of the switch not to pull the switch.”
But the real problem with the idea of “independent verification organizations” is that there is no genuine independence, the New York Post writes.
The paper is reporting about METR, a nonprofit that leading AI company Anthropic, and its CEO Dario Amodei, would like to see as the one doing the job.
But METR is not independent, as it was spun off from the Alignment Research Center, which is run by Paul Christiano, who was Amodei’s housemate, coworker and research collaborator at OpenAI in the 2010s. Christiano is also one of the first five trustees of Anthropic’s Long-Term Benefit Trust.
METR’s founder and CEO Beth Barnes was hired by Christiano, and the organization’s funders include Anthropic investors Dustin Moskovitz’s Coefficient Giving (formerly Open Philanthropy) and Jaan Tallinn’s Survival and Flourishing Fund.
METR’s denial of any wrongdoing is that its funders “have no say in the projects we work on and we do not accept any funding from AI companies or their employees.”
But Perry Metzger, chairman of the Alliance for the Future, is not buying it. “Saying you want to be audited by the ‘independent’ people at METR is a complete joke. It’s people who are funded by your buddies. It’s completely ridiculous,” he said.
And he doesn’t think highly of the idea of a bank being audited by its former employees who are also funded by its investors. “Do you know what the feds would do with you? It would be positively medieval,” he said.
And Co-Chair of the President's Council of Advisors on Science and Technology, David Sacks, advised Amodei to “stop pretending METR is independent when it is intertwined with Anthropic’s investors and staff.”
