Nadella Calls for an 'Emergency Brake' on AI Models: Assume Compromise, Let Humans Hit Stop

What happened

According to TechCrunch's Oct 10 report, Nadella wrote in the post that it is time to "step back and assess the trust architecture" of AI. He argued that "Super Intelligence" — the Trump administration's preferred term for AI — cannot be treated as a set of nested black boxes whose recommendations, answers and actions are simply accepted or rejected.

Nadella's specific proposals include: separating the model from the harness that orchestrates its work; externalizing controls and safeguards outside the model; documenting "every meaningful model action" with "tamper-proof, human-readable evidence"; and ensuring an authorized person can always "pause or shut down a model mid-task." TechCrunch noted the comments come as leading AI companies acknowledge a growing number of incidents in which they seemed to lose control of their models, and after Anthropic CEO Dario Amodei published a plan for more cautious AI development.

Update (October 12): Microsoft President Brad Smith told the BBC, in an interview published that day, that AI needs urgent global regulation with independent evaluators able to vet new developments. He said rules should require a victim to be informed if their system is breached by a rogue AI agent, an emergency brake to be available, and a legal requirement for companies to write incident reports. Smith said a washing machine is currently subject to more safety rules than AI, and that every carton of milk in a grocery store is subject to safety rules and inspection. He also said Britain, with its AI Security Institute, is well placed to lead the way on AI regulation.

Key facts

1. When: Saturday, Oct 10, in a lengthy post on X; first reported by TechCrunch at 2:47 PM PDT the same day
2. Core proposal: Assume a model is compromised and contain it from the start; an authorized person must always be able to pause or shut down a model mid-task
3. Architecture: Separate model from harness; externalize controls and safeguards; tamper-proof, human-readable evidence for every meaningful action
4. Wording: Uses "Super Intelligence" throughout — the Trump administration's official term for AI
5. Context: Follows a string of acknowledged model-control incidents at AI labs, and Anthropic CEO Dario Amodei's recent plan for more cautious AI development