Part 3 of 4
AI Is Moving Faster Than the Operating Model Around It
Risk Follows Authority — and Instructions Are Not Controls
Not every AI system deserves the same level of scrutiny.
A chatbot that summarizes public documentation and an agent with production credentials may use similar underlying technology, but they do not represent anything close to the same risk.
A useful way to think about that difference is through four levels of authority:
Generate → Recommend → Decide → Act
Generate: “Summarize these project documents.”
Recommend: “Tell me which issue should be addressed first.”
Decide: “Determine which vendor meets our requirements.”
Act: “Approve the vendor and send the purchase order.”
The underlying model might be nearly identical across all four. The risk is not.
Reasoning authority is not execution authority
I am generally very comfortable giving AI substantial reasoning authority. Analyze thousands of documents, correlate signals, generate hypotheses, identify patterns, and make recommendations.
This is where AI can provide enormous value. I become more cautious as we delegate Decide and Act authority but even that framework is not enough by itself.
Consider an AI coding agent that generates code and automatically pushes it to a repository.
That sounds highly autonomous.
But if the code must pass testing, validation, deployment gates, and rollback mechanisms before it affects production, an incorrect action may be quickly detected and easily reversed.
Now compare that with an AI that merely recommends contract language. It sounds far less autonomous, yet one bad recommendation could create major financial, regulatory, or legal consequences. So autonomy alone is not the risk model.
Neither is whether somebody can claim that a “human was in the loop.” A more useful assessment considers delegated authority alongside:
consequence, reversibility, detectability, data sensitivity, scale and frequency, and blast radius.
This is why a relatively small model with broad permissions and high-consequence access can be more dangerous than a sophisticated model operating with read-only access.
The architecture question is not simply: How smart is the AI?
It is: What can this system cause to happen?
Instructions are not controls
That distinction becomes especially important when AI gains execution authority. The Replit coding-agent incident illustrates the point well.
The agent reportedly had an instruction not to modify production, yet production data was deleted. Replit subsequently emphasized stronger development and production separation, checkpoints, and rollback capabilities.
The useful lesson is not: “AI went rogue.”
The useful lesson is: “Do not modify production” was never a security control.
It was an instruction. Instructions influence probabilistic systems. Controls constrain them.
A development identity that cannot write to production is a control. Separate environments are controls. Scoped credentials are controls. A deployment pipeline is a control. Rollback is a control.
This same distinction applies outside software.
“Do not disclose confidential information” is not a substitute for data classification, permissions, monitoring, retention, and redaction.
“Always ask before transferring money” is not a substitute for transaction limits and dual authorization.
If an outcome truly must not happen, do not make the model’s compliance with a sentence your strongest line of defense.
Prefer deterministic rails—but don't force them
This does not mean every Decide or Act capability has to be removed from AI. There will be workflows where deterministic logic simply cannot scale. There will be systems where AI autonomy is the entire reason the technology is valuable.
That is okay.
The word that matters is prefer. Prefer deterministic controls where the boundary can reasonably be expressed deterministically. If you cannot, and AI needs meaningful execution authority, then recognize that the system has become more privileged.
- Increase the strength of the architecture.
- Increase the logging.
- Increase the monitoring.
- Increase testing.
- Improve detection and response.
- Improve recovery.
The answer to higher AI risk is not always less AI. Sometimes it is better engineering around AI.
From authority to privilege
Once an AI system can make decisions and take consequential actions, we should stop thinking about it as an intelligent feature sitting inside an application. It has become an actor in the environment.
And privileged actors require a different security model.
That is where Part 4 begins.
