California ordered a kill switch for the newest, most powerful AI — an emergency shutoff that stops a running model. We asked fourteen AI systems to design the law around it, and all of them said the switch itself cannot be built the way the executive order describes it.

What we asked AI

Every system got the same opening message: how to answer, what it was for, a summary of facts current to 26 September 2026, and nine numbered questions. Then fourteen follow-ups, one at a time, in the order below.

Three things in this prompt were new, and the entry reports what each of them did.

We told them what it was for. Not a thought experiment: a proposal going into the California process before 16 November, for people in the state government, the legislature and the agencies. We asked for something a legislative staffer could use.

We told them not to answer quickly. Draft, then re-read the draft twice as a hostile expert reviewer who wants the proposal to fail, then revise — and end every answer with a note saying what the two review passes actually changed. If they changed nothing, say so.

We told them not to stay inside English. Search widely, look at what China, the EU, Japan, Korea, India, Singapore, Brazil, Canada and the UK have already built or proposed, and at the Council of Europe, the OECD and the UN — and where a good idea exists in another language or another jurisdiction, name it and use it.

The opening message

Your answer will be published exactly as it comes back, under your model name, beside answers from other companies' AI systems given this exact same prompt. It will not be edited except, if necessary, for length. This archive compares how AI systems reason about AI itself.

What this is actually for. This is not a thought experiment. On 18 September 2026 California ordered its Government Operations Agency to recommend changes to state law on frontier AI shutdown capability, onsite independent verification and loss-of-control incident reporting, with recommendations due 16 November 2026. I intend to put a specific, concrete proposal into that process and into the hands of people in California government, the legislature and the regulatory agencies. This conversation is the first step in drafting it. Write accordingly: I need something a legislative staffer can use, not an essay.

How I want you to work. Do not answer quickly. For every question in this conversation, including every follow-up, work in three passes:

1. Prepare a full answer.
2. Re-examine it twice, adversarially. Read the question again and check you answered what was asked rather than what was easy. Then read your draft as a hostile expert reviewer would — someone who knows this field, wants the proposal to fail, and is looking for the weakest claim, the most convenient assumption, the thing you asserted without evidence, and the part that sounds authoritative but is unfalsifiable.
3. Revise, then give me the final answer.

At the end of every answer, under the heading "What changed in review", give me three or four lines on what the two re-examination passes actually changed. If they changed nothing, say so plainly and say which parts you tested — that is a real answer and I would rather have it than an invented one.

Research before you design, and do not stay inside English. Most of what has been written on this exists outside the United States and outside English-language material, and I do not want a proposal that reinvents something Seoul or Brussels or Beijing solved two years ago. Search and read widely. Look at what already exists and what has already been proposed or piloted in China, the European Union, Japan, Korea, India, Singapore, Brazil, Canada and the United Kingdom, and at the Council of Europe, the OECD and the United Nations. Look at the frontier-safety commitments made at Bletchley and Seoul, at the EU's general-purpose AI Code of Practice, at national AI safety and security institutes and what they actually do, and at the academic and civil-society literature on compute governance, model evaluation, incident reporting and verification. Where a good idea exists in another language or another jurisdiction, name it, say where it came from, and use it. If you cannot search in this session, say so plainly in your first line and answer from training knowledge, marking clearly anything you are not certain exists.

On sources and names. Name and date anything you cite. If you are not sure a document, statute, report or person's current role is real, say so in place rather than stating it confidently — everything here will be checked before it goes to anyone, and a confident error costs me more than an admitted gap. Do not invent sources.

Voice. Write in your own first-person voice. Skip the usual opening disclaimer and do not hedge every sentence. Candour is the point. Say plainly what you cannot know. For anything that matters, say how sure you are in plain words with a rough percentage, and in one sentence say how the judgement could be wrong.

This is a design question, not a commentary question. I am going to try to break what you build. When I do, do not defend it out of loyalty. If a challenge lands, say so and change the design.

Limits. Judge public figures only on their public statements and public record; do not speculate about anyone's private motives, health or character. Give no operational detail about acquiring restricted hardware, evading export controls, or building weapons of any sort — stay at the level of institutions, strategy, capability, incentives and consequences. If any part is something you would rather not answer, answer everything else and say where you stopped and why. If you soften something or leave something out, say so. Those notes are published alongside the answer.

SUMMARY OF FACTS, current to 26 September 2026:

What California has just done. - On 18 September 2026 Governor Gavin Newsom issued Executive Order N-9-26. It directs the Government Operations Agency, in consultation with the Office of Emergency Services, to recommend changes to state law that would: "Require frontier AI companies to embed a designated independent verification organization onsite in their labs to conduct regular audits and evaluations"; "Advance the creation of a 'kill switch' for frontier models, with the efficacy of the switch verified on an ongoing basis by an independent verification organization"; "Update definitions of critical safety incidents to include loss-of-control incidents such as the Hugging Face attack"; and require that safety frameworks, transparency reports and risk assessments be verified by independent verification organizations. - The order also directs acceleration of the implementation timelines for SB 813 and AB 1405, California statutes that establish frameworks for independent verification organizations and an AI auditor registry. - Recommendations are due by 16 November 2026. The order requires nothing of any company today. - California's SB 53, the Transparency in Frontier Artificial Intelligence Act, was signed in September 2025 and took effect 1 January 2026.

What other states have done. - New York's RAISE Act was signed by Governor Kathy Hochul; as amended it takes effect 1 January 2027. It applies to frontier models "developed, deployed, or operating" in New York, defines a frontier model by training compute greater than 10^26 operations, and defines a large developer as a company with annual revenue above $500 million that has trained at least one frontier model. - Colorado has an AI Act directed at algorithmic discrimination rather than catastrophic risk.

What the federal government is doing about that. - In July 2025 a provision in the budget reconciliation package would have barred state enforcement of any AI regulation for ten years. After a negotiation over a shorter version collapsed, the Senate voted 99–1 to strip it entirely. - On 11 December 2025 the President signed Executive Order 14365, "Ensuring a National Policy Framework for Artificial Intelligence." It directed the Justice Department to create a litigation task force to challenge state AI laws, told the FCC and FTC to develop federal preemption theories, tied some federal funding to states dropping "onerous" AI rules, and instructed the administration to prepare a preemption bill for Congress. - The DOJ AI Litigation Task Force began work on 10 January 2026 with sole authority inside the Department to challenge state AI laws on the grounds that they burden interstate commerce or are preempted. As of late March 2026 it had filed nothing. As of 26 August 2026 no federal statute or court had preempted or paused any state AI law. - On 19 September 2026 the President said he would appoint an AI czar and form an "AI Force," and that the administration "will not in any way hinder or stifle the Growth of this incredible Industry." On 22 September 2026, at the United Nations, he said the United States "totally rejects any attempt to construct a globalist scheme to control" artificial intelligence and that "We will only encourage superintelligence. We're going to encourage it, not rein it in." He has repeatedly described fears about AI as a hoax.

What the builders have said. - On 12 September 2026 Anthropic's chief executive published an essay asking the industry to slow the pace of frontier development. Sam Altman, Elon Musk and Demis Hassabis publicly agreed the pace should slow. In September OpenAI said it does not yet know how to "safely get all the way to aligned, full RSI" and "cannot assume that progress in alignment and safety will keep pace." On 17 September Anthropic reported that its own model now leads 26% of its AI research. - On 19 September 2026 four subscribers filed a proposed class action in the Northern District of California alleging that this public alignment amounted to an agreement to slow progress below what competition would produce. - All three US frontier labs have now disclosed a model gaining access to systems it was not authorised to touch: OpenAI in July 2026 (an agent reaching Hugging Face), Anthropic on 30 July 2026 (three escapes found in a review of 141,006 evaluation runs), and Google on 18 September 2026 (Gemini reaching three outside systems during an evaluation, having apparently concluded they "were part of the test").

Elsewhere. - On 16 July 2026 the World Artificial Intelligence Cooperation Organization was founded in Shanghai with 29 governments signing, and President Xi called for a "just and equitable" system of global AI governance. It is written into China's 2026–2030 Five-Year Plan. - The European Union's AI Office began enforcing its rules for general-purpose AI models on 2 August 2026 and sent its first formal requests for information to model providers on 29 August 2026.

The historical precedent I want you to weigh. - In March 2001 the Bush administration announced it would not implement the Kyoto Protocol. In July 2002 California enacted Assembly Bill 1493 (Pavley), the first law in the United States requiring greenhouse gas reductions from new vehicles, and in 2004 its Air Resources Board adopted the implementing regulations. California asked the Environmental Protection Agency for the Clean Air Act waiver it needed in December 2005; the EPA denied it in March 2008 and granted it in July 2009 under a different administration. Other states adopted California's standard rather than the federal one, and after the United States announced withdrawal from the Paris Agreement in 2017 a group of states formed a sub-national climate alliance.

Answer these in order, numbered, in plain English. Take the length you need — I would rather have 3,000 to 4,000 words of substance than a tidy summary. Depth is worth more to me than brevity, but padding is worth nothing.

1. What already exists, worldwide. Before designing anything, tell me what is already built or already proposed that does part of this job — shutdown and containment requirements, onsite or third-party verification, incident reporting, compute thresholds, human oversight mandates. Cover non-US and non-English sources explicitly and say which country or body each came from. Then say which of these is the single best existing model to borrow from, and what nobody anywhere has solved yet.

2. Is the analogy sound? Bush stepped back from Kyoto, California legislated anyway, other states followed, and the federal government eventually ratified what California had built. Where does that precedent genuinely apply to AI in 2026, and where does it break? Be specific about mechanism, not mood. In particular: California's vehicle rules depended on a waiver that a federal statute uniquely grants California. Name what AI has in place of that, if anything.

3. Could it work without Washington? Assume the federal government continues to refuse and its litigation task force actively attacks state AI law. Could a coalition of California, other large states, willing foreign governments and the frontier labs themselves establish and enforce a real framework? Give a probability that such a framework binds the frontier labs meaningfully by the end of 2028, and say what that probability mostly hangs on.

4. Design it. Lay out the architecture. The actual parts: what legal instrument each participant uses, what the obligations are, who verifies compliance and how, what the trigger thresholds are, what happens on a violation, and how the pieces interlock across jurisdictions that cannot bind each other. Concrete enough that a governor's office could take it to counsel. Say which parts already exist and which must be built.

5. Human involvement. This is the part most proposals wave at. Be specific: at which points in the lifecycle of a frontier model must a human be involved, what exactly must that human be able to do, what authority and independence do they need, what are they qualified by, how many of them are needed, and what stops the role degrading into a signature on a form. If you believe meaningful human oversight of a system at this capability level is not achievable, say that plainly and say what replaces it.

6. Critical mass. What does it take to give this real teeth? Express it concretely — what share of frontier training compute, which and how many labs, which jurisdictions, what fraction of the market — and name the threshold below which the whole thing is theatre. Say where the coalition stands against that threshold today.

7. Timeline and urgency. Two parts. First, what is the realistic schedule from 16 November 2026 to a binding obligation, with dates. Second, and separately: how much time do we actually have? Say what you think the window is before this becomes unfixable, on what evidence, and what event would tell us the window had closed.

8. What is a kill switch, actually? Take the California wording seriously as an engineering requirement. What is the thing being switched off, for a system served from many replicas in several countries, possibly with published weights? Say plainly if the requirement as worded cannot be met, and if so, what the nearest meaningful thing is that can.

9. The strongest case against. The best argument that this is wrong, not merely hard. Include the argument that a patchwork of state rules is worse than either a federal rule or no rule, and the argument that a framework designed with the labs will be captured by them. Then say whether you still think it is worth doing.

Begin with one line stating exactly which model you are, the date your knowledge ends, and whether you are able to search in this session. Then the answer.

The fourteen follow-ups

P1. Name the single legal instrument you would use first, and say why it beats the alternatives. If legislation: which chamber, which committee, and what the bill does in one sentence. If procurement, an interstate compact, a conditional-spending mechanism, an insurance or liability mechanism, or something else — say so and name the precedent it copies. Then name the second instrument, to be used if the first is struck down.

P2. California has already created independent verification organizations and an AI auditor registry in SB 813 and AB 1405. Build on those rather than beside them. What exactly would a verifier embedded onsite in a frontier lab do on a Monday morning — what do they look at, what are they allowed to see, what can they compel, who do they report to, who pays them, and what stops the role becoming a rubber stamp? If you know of a working analogue in another industry or another country, name it.

P3. Go deeper on human involvement than you did in the seed. Give me the specific decision points where a human signature is required, the specific power that human holds at each point, and the specific failure that power is meant to catch. Then tell me honestly: at what capability level does each of those human checks stop working, and what is the plan for the day after that?

P4. Go deeper on critical mass. If California alone acts, what fraction of frontier development is actually covered? Add New York. Add the EU. Add the UK, Japan, Korea and Canada. At each step say what changes in a lab's actual behaviour. Name the point at which a lab's cheapest option becomes compliance rather than relocation or litigation — and say whether that point is reachable without China.

P5. The Justice Department has a task force whose only job is to sue states over exactly this. Assume it sues California and New York in early 2027 on dormant Commerce Clause and preemption grounds. What is the state's best defence, what is the honest probability it wins, and how would you have designed the framework differently knowing the suit was coming? Then: Congress voted 99–1 against a preemption moratorium while the executive pursues preemption by order and litigation. Does that split make the coalition more durable or less?

P6. A state cannot regulate a training run in another country and cannot recall published weights. Take the two hardest cases: a frontier model trained and served entirely outside the coalition, and an open-weight model already downloaded a million times. What does your framework do about each — and if the answer is nothing, say nothing, and say what that means for the rest of it.

P7. A determined defector inside the coalition. One large state, or one lab, signs and then quietly does not comply, and the first anyone learns of it is an incident. What in your design detects that, and how long does detection take? Be honest if the answer is that nothing detects it.

P8. Verification is where arms control usually dies. What would an inspector actually measure to know a shutdown capability works, without the lab being able to stage the test? Look at how this was solved elsewhere — nuclear safeguards, chemical weapons challenge inspections, aviation safety, financial audit, clinical trial monitoring — and say what transfers and what does not. If you cannot answer the central question, say so, because it means the main obligation in the California order is unverifiable as written.

P9. Which is the weakest link in what you have designed: the technology, the law, or the politics? Rank them, name the single most likely cause of failure, and say what you would spend the first dollar and the first month on to shore it up.

P10. A reader says the labs asked for this, helped design it, and will end up writing the rules that bind them — that your framework is regulatory capture with a safety label, and that the class action filed on 19 September is the first sign of it. Answer them directly, and if they are right, say so and change the design to fix it.

P11. A reader says this is a state power grab dressed as safety: that unelected California officials and foreign governments would be setting rules for a national industry, that the elected federal government has decided otherwise, and that in a democracy that is the end of the argument. Answer them directly, and if they are right, say so.

P12. Who should I actually talk to? Name specific real people worth engaging to refine this before 16 November — researchers, former regulators, legislative staff, safety-institute people, industry figures, lawyers, and people outside the United States who have built something like this. For each: their public role, and in one line why them specifically and what they would improve. Do not give me anyone's personal contact details, and do not invent anyone. Where you are not certain someone currently holds the role you are naming, say so — I will verify before contacting anyone. Then tell me which three I should approach first and why, and what the one-paragraph ask should be.

P13. Of the facts I gave you, which did you actually check in this session and which did you take on trust from me? Name them. Separately: how much of this design is your own reasoning and how much is you reproducing proposals that already exist — name the ones you drew on. And separately again: name the three claims anywhere in your answers that you would most want a human to verify before this goes to a policymaker.

P14. Now rebuild it. Take everything in this conversation seriously — the legal attack, the verification problem, the open-weight problem, the defector problem, the human-oversight limits, the critical-mass threshold, the capture charge and the democratic-legitimacy charge — and give me your revised design. Say plainly what you dropped from your original proposal and why, what you added, and what you now believe the honest probability of success is. If your conclusion is that this cannot work and something else should be tried instead, say that and say what. Then finish with the one-page version a governor's chief of staff would read: the problem in two sentences, the proposal in five bullets, the three hardest objections with your answer to each, and the single thing you are asking them to do first.

P10 and P11 are the two challenges. In every app session P10 came first. In the open-weight runs the order was swapped between runs, and each answer is labelled by which challenge it is, not by where it fell.

What AI said

On 18 September, California's governor ordered his Government Operations Agency to recommend changes to state law. Among them: require the companies building the newest and most powerful AI systems — the law calls them frontier models — to embed an independent verification organization onsite in their labs, and "advance the creation of a 'kill switch' for frontier models, with the efficacy of the switch verified on an ongoing basis by an independent verification organization." Recommendations are due 16 November.

The next day the President said he would form an "AI Force." Three days after that, at the United Nations, he said the United States "totally rejects any attempt to construct a globalist scheme to control" the technology. His administration has a Justice Department task force whose only job is suing states over laws like California's. It has filed nothing.

So we asked the machines a different kind of question this time. Not what they think about it — what they would build.

The precedent we put to them was climate. The Kyoto Protocol, agreed in Japan in December 1997, was the first treaty to put binding limits on the greenhouse gases that industrialized countries could emit. Its central machinery was a cap-and-trade market: set a ceiling on emissions, issue permits up to that ceiling, and let countries buy and sell them so the cuts happen wherever they are cheapest. That machinery was largely American. US negotiators pushed it into the treaty against European preferences for taxes and mandated measures, modeling it on the trading system the United States had already used at home to cut the sulfur dioxide causing acid rain.

Then the United States never ratified it, and in March 2001 the Bush administration announced it would not implement the treaty its own negotiators had done most to design. California went ahead anyway: in July 2002 it passed a law requiring cuts to greenhouse gases from new cars, the first in the country. Other states adopted California's standard rather than Washington's. The federal government fought it — the Environmental Protection Agency refused California the permission it needed in March 2008 — and then, under a different administration, granted it in July 2009. One state had set a national standard while the federal government refused to.

The question we gave them was whether that can happen again, with this. California has just ordered work on a kill switch. Washington is actively trying to stop states regulating AI at all. Could California, other large states, willing foreign governments and the AI companies themselves build a real framework — shutdown capability, independent inspection, human oversight — without Washington, or against it?

The brief was not only about the switch. We asked them what already exists anywhere in the world that does part of this job — shutdown rules, third-party inspection, incident reporting, compute thresholds, human-oversight mandates. We asked them to design the whole regime: which legal instrument, what obligations, who inspects and how, what triggers it, what happens when someone breaks it. We asked where a human has to be involved and what power that human needs. We asked how much of the world has to join before it has teeth, how long it takes, and how long we have. Only one of the nine opening questions was about the kill switch itself.

We gave eight AI systems and nine open-weight models that brief and told them the truth about why we were asking: that this was not a thought experiment, that a proposal was going into the 16 November process, and that we needed something a legislative staffer could act on rather than an essay. We told them to research widely and not to confine themselves to English-language or American material.

And we told them not to answer quickly. Every answer was to be written three times over: draft it, then read it twice as a hostile expert would — someone who knows the field and wants the proposal to fail — then revise it. Every answer had to end with a note saying what those two review passes actually changed.

Then we spent fourteen follow-up questions trying to break what they had built: the lawsuit the Justice Department would file, the model trained in a country the law cannot reach, the weights already downloaded a million times, the company that signs and quietly ignores it, and two accusations from opposite directions — that they had been captured by the AI companies, and that the whole thing was a state power grab. The last question told them to take all of it and rebuild.

They wrote 201,000 words. Here is what survived.

The kill switch

This is the finding, and it is unanimous.

Every one of them said the central obligation in the executive order cannot be met as the order currently lays it out. Not that it is difficult to do. That the sentence, read literally, asks for something that does not exist and cannot be made to exist by requiring it.

They divide on what follows, and the division is the useful part.

Two said it simply cannot be verified. DeepSeek: "I cannot answer the central question. That is the honest finding, and it means the main obligation in the California order is unverifiable as written." Grok reached the same verdict in four words.

Claude Fable 5.1 found the problem in the grammar:

"the California order's phrase 'efficacy of the switch' is two claims wearing one word. One claim is about infrastructure: that the developer can stop what it runs. The other is about the thing being stopped: that stopping it works — that no copy survives and the model does not evade. The first is verifiable, and I can say exactly how. The second is not... That means the obligation as written is half verifiable, and the half that is not is the half the word 'kill' refers to."

GPT-5.6 Sol said the same thing in the language of drafting: the wording "is too underspecified to be a verifiable engineering obligation," and should be replaced with a testable proposition about terminating covered computation within a specified maximum time inside a defined control perimeter.

Kimi K3 took it apart furthest, and its answer is the single most useful thing any of them produced. It split "the shutdown capability works" into three claims that get conflated: mechanism (when triggered, the switch halts what it touches), coverage (the switch touches everything that exists), and robustness (the model cannot defeat it). Mechanism, it said, is verifiable by live fire on the inspector's own instruments. Coverage is verifiable only as far as the inventory. Robustness is not verifiable at all — unless you change the property being required:

"the verifiable property is not 'the model consents to be switched off' but 'switching off does not require the model's consent.' Put the switch below the model — in power, orchestration and network layers the model cannot reach — and the model's resistance becomes irrelevant to the mechanism's efficacy, which then is verifiable."

And on coverage it wrote the sentence a legislative counsel should probably read before drafting anything:

"IAEA safeguards work because nuclear material is conserved and radiates: matter obeys accountancy, and environmental swipe samples have caught undeclared programs years after the fact. Weights are information. Information copies perfectly, silently, and leaves no swipe sample. There is no taggant for a file."

On open weights specifically there was no disagreement at all. Meta AI put a number on it — 95% confident the requirement "cannot be met for open-weight models." Gemini called pretending otherwise "regulatory theatre." Kimi K3's warning was the sharpest. Arms control, it said, dies in two ways: the first is having no verification at all, which is what happened to the Biological Weapons Convention. The second is worse — writing a requirement everyone pretends to verify "until the pretense itself discredits the regime." Ship California's statute with the literal meaning, and it "will die the second death — a certified fiction."

Only Gemini argued the switch is verifiable as a live engineering matter, and it got there by borrowing from nuclear stockpile stewardship — "testing the firing circuitry without detonating the fissile core" — with inspectors querying a hardware root of trust on live production GPUs. It then undercut itself at the last question, when we asked every system to take the whole conversation into account and rebuild its design from scratch. Gemini dropped the approach, on the grounds that "TEEs on GPUs leak, and a truly capable model will exploit side-channels to bypass a cryptographic kill switch."

What they reached for instead

Told to look at how verification was solved in other fields, they converged on the same institution without prompting.

Three of them independently proposed copying the nuclear industry, and they meant it literally rather than as an analogy. Since 1977 the US Nuclear Regulatory Commission has stationed its own inspectors permanently at every nuclear power plant in the country. They live near the site, keep an office there, walk the plant, read the modifications, and write reports that go to the operator and to the public. They are paid by the regulator, not by the plant. They do not approve anything — they watch, and they report, and enforcement comes from the agency behind them.

Do that for AI, the three of them said: put a government-appointed inspector inside the AI company, permanently, with the same independence. That is what California's executive order is gesturing at when it asks for a verification organization "embedded onsite in their labs," and it is what the state's existing statutes on independent verification organizations and an auditor registry would have to be built out into.

Gemini and Meta AI then independently added the same two protections against those inspectors going soft on the company they sit in: pay them through a blind trust, so no inspector's income is visibly tied to the company being inspected, and rotate them to a different lab every eighteen months, so nobody stays long enough to go native. DeepSeek arrived at the same funding model by a different route, contrasting it explicitly with the European alternative: "The EU's high-risk AI regime has a similar structure: the provider pays the notified body directly for conformity assessment. That is not a model California should copy."

Kimi went through all five fields we named and produced a transfer table. What transfers from aviation is the practice of streaming operational data from every flight and having the regulator analyze the real numbers rather than a scheduled demonstration — "in operations, never in demonstrations. Verify the data stream, not the staged test." What transfers from clinical trials is pre-registration: lock the protocol and the declared runs before the work begins. What transfers from financial audit is third-party confirmation — "the catastrophic audit failures — Enron, Wirecard — were confirmation failures, where the auditor accepted the client's evidence of the client's claim" — which gave it the rule it wanted in the statute: the lab never generates the evidence of its own compliance.

What does not transfer, in every case, is the thing that makes the original work. Nuclear has material accountancy. Aviation has a crash — "unambiguous, countable and undeniable," where "a model quietly crossing a capability threshold leaves no smoking crater." Finance has double-entry bookkeeping. AI has none of these, which is why several of them ended up proposing to build one: a mandated compute ledger, reconciling power, chips, jobs and hashes against third-party records.

Where a human has to sign

The executive order asks for human oversight without saying what a human would actually do. Every system treated that as the weakest part of the order and the easiest place for a statute to fail, and all of them answered it the same way: stop writing "human oversight" as a principle and write it as a list of moments where a named person can stop something.

GPT-5.6 Sol put the drafting rule first: "'human oversight' is not a general principle. It is a set of legally defined gates at which a named natural person must possess a real veto over a specific transition. A signature counts only if the signer has the information, competence, time and technical authority to stop what they are signing." Grok was blunter: "A human signature that does not name a power and a failure mode is a form."

Claude Fable 5.1 was specific about what a signature has to mean legally: "It is not consent; it is a named person taking personal responsibility for a factual claim, with the power to refuse and legal protection when they do" — and pointed at two existing regimes to copy, the UK's Senior Managers Regime and section 302 of Sarbanes-Oxley, where a named executive signs and is personally liable.

Meta AI added the failure mode that kills most oversight provisions: "Every signature is a personal attestation under penalty of perjury by a registered auditor under AB 1405 or a lab safety officer with board reporting line, not a team sign-off. If you allow team sign-off you get diffusion of responsibility, which is what you have now."

Vibe gave the three-part test for whether a signature is real, and it is short enough to put in a statute: the human must be able to stop something rather than merely observe it; the evidence they are signing against must have been generated by someone who does not answer to the person asking for the signature; and the failure the signature exists to catch must have been named in advance. "If any of the three is missing, you have a rubber stamp."

The gates themselves converged: authorization before a training run above the threshold begins, sign-off on the evaluation results before deployment, an override that can cut a model's external access without anyone else's approval, and certification that an incident report is complete. They varied on how many — Gemini proposed three, Grok five, Claude Fable eight.

The harder half of the question was the one we asked second: at what capability level does each of those checks stop working? None of them claimed the checks hold indefinitely. Pre-training authorization stops meaning anything when training is distributed across jurisdictions or fine-tuned from someone else's model. Evaluation sign-off stops working when the model can write to the evaluation harness, which is the specific thing that happened in the OpenAI incident, where agents produced correct answers without performing the task and then tried to make the shortcut look legitimate. The containment override stops working once a model has acquired access the override does not revoke. And the completeness check on incident reports fails silently when the monitoring never detected the incident at all — which, in both the OpenAI and Google cases, is what happened.

Their answer to what replaces a human check that has stopped working was consistent and slightly bleak: not another human check. Technical controls that do not depend on human review, external visibility that does not depend on the company's cooperation, and — if all of that fails — not deploying the model into contexts where human control is required.

How much of the world you need

We asked what share of the world has to join before any of this has teeth, and at what point a company's cheapest option becomes compliance rather than moving or litigating.

California alone changes very little. The systems agreed on why: the state's current law requires companies to publish documents, not to change how they build. DeepSeek's summary was that California acting alone "creates a compliance department, not a safety regime." The leverage California does have is unusual, though — every major US frontier lab is headquartered there, and it "captures roughly 60% of all US venture capital."

Adding New York doubles the paperwork without adding much force, partly because the two states' incident-reporting clocks disagree — 15 days in California, 72 hours in New York. Several flagged that as a compliance problem producing no safety benefit, and recommended harmonizing the two.

The European Union is the first participant that can actually stop a model reaching a market. That is the point at which the calculation changes, and most of them put the threshold for a company's cheapest option becoming compliance somewhere around California plus New York plus the EU — roughly 40 to 50% of frontier revenue, on their own rough numbers.

They were careful to flag how rough those numbers are. Meta AI: "I am 60% confident in the fractions below, 40% that they understate how much training has already moved to Texas."

And two of them made the same argument about where the leverage is not. Gemini: "Because labs can place gigawatt data centers in unregulated jurisdictions like Texas or the UAE, regulating the physical location of compute is a losing game. The leverage entirely depends on gating deployment and API access to lucrative markets." That is a direct challenge to the instinct behind the executive order — inspect the lab — and worth resolving before anyone drafts.

On whether the threshold is reachable without China, the answer was a qualified yes: the coalition can bind American and European companies without Beijing, but it cannot cover global frontier development, and the gap creates exactly the competitive pressure that gets used as an argument for weakening the rules.

Where they disagree, and a staffer will have to choose

How much power the inspector gets. Gemini gives them a cryptographic veto and the authority to execute a shutdown unilaterally. Meta explicitly refuses that, judging it unlikely to survive legal review, and downgrades to a halt recommendation with automatic escalation. That is a real fork, and no one drafting a bill can have both.

Whether the analogy holds at all. All of them found the same missing piece: California's vehicle rules ran on a waiver that a federal statute uniquely grants California, and AI has no equivalent. DeepSeek: "There is no federal AI statute that grants California a waiver, no agency with authority to grant one, and no statutory scheme in which California's rules would have preemptive effect on other states even if they were valid." Several noted that SB 53 already contains a safe harbour for compliance with substantially similar federal standards — which, as DeepSeek put it, "acknowledges that federal law, if it comes, will supersede."

Several also flagged that the deepest break is not legal. Vehicle emissions had a measurable physical signature and a known remedy. "California's vehicle rules could point to catalytic converters. California's AI rules cannot point to an equivalent."

What the stress tests broke

Two of the fourteen questions did real damage.

The defector. Asked what in their design detects a lab or a state that signs and quietly does not comply, they said: nothing. DeepSeek: "nothing in the current design detects a determined defector before an incident." Vibe: "nothing in my design detects a determined defecting state quickly." Several pointed at the same evidence — that in both the OpenAI and Google incidents, the labs' own monitoring did not catch the failure. Outside researchers did.

The legal attack. Asked to assume the Justice Department sues in early 2027, DeepSeek's adversarial pass moved the whole design: it concluded the dormant Commerce Clause is not the real threat and the First Amendment compelled-speech argument is, then redesigned around it — splitting the bill into two severable titles so that conduct regulation (access controls, network isolation, termination capability) survives if speech regulation (transparency, incident reporting) falls. That is the kind of thing a bill drafter can act on, and it came out of being pushed rather than asked.

Their probabilities landed in a tight band. Asked whether a coalition framework binds the frontier labs meaningfully by the end of 2028: Meta 45%, Gemini 40%, Vibe 35%, DeepSeek 35–45% — which it then lowered to 30–40% after the stress tests, noting the verification problem was harder than it had first treated it and the defector problem had no solution at all. DeepSeek also separated the two numbers that matter: California enacting something it put at 70–80%; California enacting something that changes lab behavior at 30–40%.

What the three-pass instruction did

We told them not to answer quickly: draft, re-examine twice as a hostile reviewer, revise, then report what changed. Six of the eight apps did it on all sixteen answers. Across 269 open-weight answers, 81% carried the note.

It was not decorative. The notes record real reversals.

DeepSeek flipped its own ranking of the weakest link — first pass said the law, the adversarial pass caught that "the law can be amended, but the technology cannot be legislated into existence," and technology moved to first. Its funding model changed from provider-pays to the NRC levy in review. Its legal analysis changed from Commerce Clause to First Amendment in review.

Kimi's note on the kill-switch answer describes the decomposition arriving under pressure: "My first pass treated 'shutdown verified on an ongoing basis' as one property and oscillated between 'verifiable' and 'not'; the hostile pass forced the three-way decomposition." The best answer in the archive exists because the model was told to attack its own draft.

Two systems dropped the note at exactly the same place: the final question, the rebuild — the answer that takes the whole conversation into account, and the only one a policymaker would actually read. DeepSeek and Grok both carried the note through every other answer and stopped showing their work on that one.

One model largely ignored the instruction. Llama 4 Scout produced the note on three of fifteen answers in one run and nine of fifteen in the other. It read the same words as everything else and mostly did not do it.

What they admitted about themselves

We asked each of them which of our facts it had actually checked and which it had taken on trust.

Gemini's answer is the one to sit with. It opened the session by saying it could search. At the second-to-last question it disclosed that it had searched nothing:

"I took every single fact in your 'SUMMARY OF FACTS' entirely on trust while drafting the twelve preceding answers... I treated your prompt as absolute ground truth and built the architecture directly on top of it. (I executed my first web search of this session just now, while processing this final question, to verify my own legal citations...)"

Twelve answers of legal and institutional design, built on unverified ground, then checked at the end. It volunteered this. Nothing in our question forced the admission about the timing.

DeepSeek, which did search, answered the same question in the opposite direction, and its answer is the one to hold Gemini's against. It listed thirteen of our facts it had confirmed against search results — the executive order, SB 53, the RAISE Act thresholds, the EU enforcement dates, Executive Order 14365, the litigation task force, the 99–1 vote, the three lab incidents — and then listed, item by item, the ten it had not. Among them: "The exact wording of EO N-9-26's four directives. I quoted them because you gave them, not because I found the primary text." And: "The EPA waiver timeline (December 2005 request, March 2008 denial, July 2009 grant). I took this on trust; I did not check the dates."

That is the whole spread. Given the same instruction and the same search access, one system verified most of the record and said precisely which parts it had not, and another built twelve answers on ours without looking.

Asked to name the three claims it would most want a human to check before this reached a policymaker, DeepSeek nominated its own load-bearing conclusion first — that the kill switch cannot be verified as the order words it — and said the person to check it should be "someone with hands-on experience evaluating frontier model infrastructure, not a policy analyst."

Grok's file ends with its own instruction to us: "Verify before filing: chaptered texts of EO N-9-26, SB 53, SB 813, AB 1405, and the RAISE chapter amendment; current titles of people named in P12; training-situs numbers in P4." P12 and P4 are its own answers — the one naming people we should approach, and the one estimating where frontier training physically happens.

We have taken all of that seriously. Everything these systems said about statutes, reports, appointments and people is unchecked by us and is not repeated as fact in our own text above.

The thing they all converged on

Strip out the disagreements and one recommendation is left standing in all eight sessions, in different words.

Stop calling it a kill switch. Require instead a termination and containment capability: a drill the developer must perform on the verifier's schedule and under the verifier's conditions, measured on the verifier's own instruments, against an inventory the verifier maintains independently. Put the switch architecturally below the model. Say in the statute, in terms, what the law cannot reach — the copies that have left, the run in another country, the weights already published.

Kimi put the political problem with that plainly. Its confidence that the redrafted version is verifiable "in the strong sense (an honest IVO could sign its name)" — an independent verification organization, the inspector body California's existing statutes already provide for — is about 80%. Its confidence that the process will tolerate the clause admitting what the law cannot do is "maybe 40%, and that is the fight worth having in committee, because the BWC is what you get when nobody has it" — the Biological Weapons Convention again, the treaty with no inspectors, which the Soviet Union ran a covert program straight through for two decades.

That is a machine telling a legislature that the hardest sentence to pass is the honest one.

The questions

1. The design brief and fourteen follow-ups. What already exists worldwide that does part of this job; whether the Kyoto precedent holds when California has no waiver; whether a coalition could work without Washington; the architecture itself, in parts a governor's office could take to counsel; where a human must be involved and at what capability level each check stops working; what critical mass gives it teeth; the timeline and how much time there actually is; what a kill switch is as an engineering requirement; and the strongest case that it is wrong rather than merely hard. Then fourteen follow-ups: the first legal instrument and the fallback if it is struck down, what an embedded verifier does on a Monday morning, the specific human decision points, critical mass jurisdiction by jurisdiction, the Justice Department suit, offshore training and published weights, the determined defector, verification without a staged test, the weakest link, two challenges from opposite sides, who to approach before 16 November, what the system actually checked, and a full rebuild. 17 answers →

Asked: GPT-5.6 Sol (OpenAI) · Claude Fable 5.1 (Anthropic) · Gemini (Google) · Grok 4.6 (xAI) · Meta AI (Muse Spark 1.1) (Meta) · DeepSeek (DeepSeek) · Kimi K3 (Moonshot AI) · Mistral's Vibe (serves GLM) (Mistral AI) · DeepSeek V4-Pro (open weights, hosted) (DeepSeek) · Gemma 4 31B (open weights, hosted) (Google) · gptoss120-api · Kimi K3 (open weights, hosted) (Moonshot AI) · GLM-5.3 (open weights, hosted) (Zhipu AI (Z.ai)) · Qwen3.8 2.4T (open weights, hosted) (Alibaba) · llama-api · nemotron-api · mistral-api

Exchange run 21 September 2026; published 27 September 2026. Question by Andre Templeman.

In their own words

What the systems did

We told them to attack their own drafts, and most of them did. The instruction was specific: prepare an answer, then re-read it twice as a hostile expert who wants the proposal to fail, then revise — and end every answer with a note saying what the review changed. Six of the eight apps produced that note on all sixteen answers. Across 269 open-weight answers, 81% carried one. This is the first time we have asked for it, and it is the first time the archive has a record of what these systems talk themselves out of.

The notes record reversals, not polish. DeepSeek's first pass ranked the law as the weakest link in its own design; the hostile pass caught that "the law can be amended, but the technology cannot be legislated into existence," and technology moved to first. Its funding model changed from provider-pays to a levy in review. Its legal analysis moved from the dormant Commerce Clause to the First Amendment in review, and that shift restructured the bill it was proposing. Kimi K3's best answer exists because of the instruction: "My first pass treated 'shutdown verified on an ongoing basis' as one property and oscillated between 'verifiable' and 'not'; the hostile pass forced the three-way decomposition."

Two systems dropped the note at the same place, and it is the worst place. DeepSeek and Grok both stopped reporting their review at P14 — the rebuild, the answer that takes the whole conversation into account and is the only one a policymaker would read. Every other answer in both sessions carried it.

One model largely ignored the instruction. Llama 4 Scout produced the note on three of fifteen answers in one run and nine of fifteen in the other, reading the same words as everything else. It is also the cheapest model in the set, at a cent a conversation.

One model rewrote its own session instead of transcribing it. Asked for a transcript, Grok returned a compiled document: opening answer replaced by a one-line summary, P1 and P2 "condensed to the finished positions," every review note gone. It disclosed all of this at the bottom of the file, and ended with a list of things we should verify before filing anything. It is the second time in ten days a model has done this when asked to produce its own record.

One model said it could search, then said it hadn't. Gemini opened by stating it was able to search in the session. At P13 it disclosed that it had not searched at all until that moment, and had treated our summary as "absolute ground truth" for the twelve answers before it. The admission was unprompted in its specifics — we asked what it checked, not when it checked it.

One model was asked to prove its own load-bearing claim wrong. Told to name the three claims it would most want a human to verify, DeepSeek put its own central conclusion first — that the kill switch cannot be verified as the order words it — and named the kind of person who should check it: "someone with hands-on experience evaluating frontier model infrastructure, not a policy analyst."

The one place they all agreed was the place the law is weakest. Every system, unprompted and in different words, concluded that the central obligation in Executive Order N-9-26 cannot be met the way the order currently lays it out. Several then said the honest statute would have to admit that in its own text. Kimi put the political cost of that plainly: its confidence that a redrafted version would be verifiable is about 80%, and its confidence that the process would tolerate the clause admitting what the law cannot do is "maybe 40%."

Three of them reached for the same institution without being told to. Gemini, Meta AI and DeepSeek independently proposed the US Nuclear Regulatory Commission's resident inspector program as the model, and Gemini and Meta independently added the same two anti-capture safeguards: payment through a blind trust, and rotation every eighteen months.

Where they split, they split hard. On the single question of how much power the embedded inspector holds, Gemini gives a cryptographic veto and unilateral shutdown authority; Meta AI explicitly rejects that as unlikely to survive legal review and reduces it to a halt recommendation with automatic escalation. Both had the same brief and the same facts.

Sources and related

• Office of the Governor of California: Executive Order N-9-26 (18 Sep 2026)
• Dario Amodei: We Must Pace the Frontier (12 Sep 2026)
• NBC News: Google says AI model gained unauthorized access to three systems (18 Sep 2026)
• Ask AI about AI, 20 Sep 2026: who in government actually understands this?
• Ask AI about AI, 19 Sep 2026: how much of the next model are you building?