Assume Your AI Is Compromised. Then Build Like It.

Microsoft’s CEO spent a Saturday arguing that the right way to deploy an AI model is to assume someone has already gotten to it. That sounds like a lab problem. It is not — it is a description of what happens when you hand an AI agent your inbox, your calendar, and a payment method, and nobody writes down what it did.

“In a lengthy post on X, Microsoft’s CEO laid out his views on the dangers posed by highly advanced AI models and how to confront those risks. Nadella says we can no longer accept a world where AI is treated as a ‘set of nested black boxes.'”

Terrence O’Brien, The Verge

Our take

Strip out the vocabulary and Nadella proposed four specific controls. TechCrunch’s account of the October 10 post lists them: separate the model from the harness that orchestrates its work; move safeguards outside the model rather than relying on the model to police itself; log every meaningful model action as tamper-proof, human-readable evidence; and guarantee that an authorized person can pause or shut down a model mid-task. He framed the whole thing as time “to step back and assess the trust architecture” and said developers should “assume a model is compromised and contain it from the start.” The emergency brake is his analogy for that last piece.

Here is what is useful about that list: all four translate directly to a ten-person company, and none of them require buying anything.

Separation means the model does not hold your credentials. The surrounding software holds them and decides which requests to pass through. In practice that is the difference between pasting an API key into a chat tool and connecting a service with a scoped integration you can revoke.

Externalized safeguards means the rule that stops a bad action lives outside the thing taking the action. Telling a model in its instructions not to send emails without approval is not a safeguard — it is a suggestion to the same system you are trying to constrain. A confirmation step a human has to click is a safeguard.

Audit logging means a record a person who was not present can read afterward and understand. Most small business AI deployments fail this badly. If an agent posted to your Google Business Profile, replied to a review, or updated a price, can you produce the sequence of what it did and why? If the answer is “it’s in the chat history somewhere,” you do not have a log.

Kill switch means a named person who knows the procedure, not a theoretical capability buried in a settings page. Write down who stops the agent and how. Test it once.

What strikes us about this framework is how old it is. Least privilege, mediated access, immutable logs, revocable credentials, and a documented termination procedure are the insider-threat playbook any business with employee offboarding already half-understands. Nadella’s contribution is insisting that a model is a party you extend trust to, not a feature you switch on. The thing that makes it harder than a human hire is speed and scale: an AI agent acts in seconds, repeats the action a thousand times without fatigue, and has no career incentive to flag its own mistake.

It is also worth being clear about the realistic threat. The failure mode is rarely a model deciding to hurt you. It is a model following instructions that arrived inside content it was asked to read — a customer email, a web page, a PDF, a review. That is why “assume compromised” is the right posture rather than an alarmist one: your agent reads untrusted input every day as part of its job, and anything it reads can try to steer it. Treat incoming text as hostile by default and the containment controls stop sounding paranoid.

One caveat on how much weight to give this. TechCrunch notes the piece was a social media post and reports no accompanying Microsoft product, policy, or investment commitment. So this is framing from a vendor CEO, not a roadmap you can plan a purchase around. Do not wait for a feature labeled “emergency brake.” The controls he describes are things you implement in how you configure what you already have, and the cheapest one by far is reducing scope: most AI agents in small businesses hold far broader permissions than the job requires, usually because broad was the default option during setup.

What this means for your business

• Audit the permissions on every AI tool connected to your accounts this week. List what each one can read and what it can change. Anything with write access it has never used, revoke today.
• Put a human approval step in front of every irreversible action. Sending an email, publishing a post, issuing a refund, deleting a record. Approval belongs outside the model, not in its instructions.
• Create an action log you can hand to someone else. Even a shared sheet the automation writes to — timestamp, tool, action taken, target. You need it the first time a client asks what happened to their listing.
• Name the person who shuts it down, and write the steps. One page, stored where it is findable on a Saturday. Then run the procedure once while nothing is wrong.
• Never let an agent that reads customer-submitted content also hold payment or admin access. Split the roles across two tools with two permission sets. That single separation removes most of the realistic damage.

Does this apply to a small business, or just to AI labs?

It applies the moment an AI tool can change something on your behalf. The controls are configuration decisions, not enterprise purchases.

What does “assume the model is compromised” actually mean?

Design so that a model behaving badly cannot do lasting damage — limited permissions, approval gates outside the model, a readable log, and a way to stop it mid-task.

Nadella’s full argument and the quotes in context are in the original report: Satya Nadella says we should assume all AI models are ‘compromised’, by Terrence O’Brien for The Verge.

---