AI Is Moving Faster Than the Operating Model Around It

The problem is not simply that AI can hallucinate

The 2025 Deloitte Australia government-report controversy is a useful example.

The report was corrected after problems were identified with supporting material, including inaccurate references and a fabricated quotation attributed to a court judgment. Deloitte disclosed the use of OpenAI in the revised report.

The easiest takeaway is: “AI hallucinates.”

But that is not the interesting lesson.

The better question is:

How did unsupported evidence survive a professional review process?

Who owned source validation?
What exactly was the reviewer expected to verify?
Were they reviewing the evidence or evaluating whether the finished document sounded plausible?

Those questions are much more important because they apply far beyond AI-generated reports.

They apply to engineering recommendations, contract review, project forecasting, procurement, financial analysis, and security investigations.

If the process was poorly defined before AI entered it, AI does not magically fix it.

It simply helps the organization move through the ambiguity faster.

AI creates a new problem for reviewers: it sounds good

Modern AI is extraordinarily good at producing polished output.

That creates a human problem.

A response can sound coherent, professional, and confident while still being unsupported.

And that leads to a review experience that looks something like this:

The AI recommends an action.

The human receives a prompt.

Approve?

Technically, we now have a “human in the loop” but do we have governance?

I think that phrase is significantly overused. A human should not be asked to bless an answer simply because it sounds reasonable. The reviewer should have access to the evidence required to challenge it.

If an AI recommends an engineering action, show the records it analyzed, the assumptions it made, the information it could not locate, and contradictory evidence.

If an AI investigates a security incident, show the processes, identities, network connections, queries, timestamps, and telemetry supporting the conclusion.

The human should review the evidence, not the quality of the AI’s prose.

Human oversight should be risk-driven

That does not mean a human should review every AI action. Doing so would destroy much of the scale and efficiency that makes AI attractive in the first place. Instead, human oversight should itself be based on risk. Every AI system should leave an auditable trail.

But the depth of logging, evidence, retention, and human intervention should scale with the system’s access, authority, and potential consequence.

This is not a new concept.

We already apply more scrutiny to privileged administrator activity than ordinary employee behavior.

We already apply stronger controls and longer retention requirements to high-value financial transactions than routine email.

AI should be treated the same way.

Low-consequence systems may need an audit trail and periodic review.

High-consequence systems may require deep evidence capture, stronger monitoring, human authorization, or all three. The important thing is that the control matches the risk.

Evidence is the control surface for human judgment.

And a human clicking Approve is not necessarily governance.

Meaningful oversight requires design

The phrase “human in the loop” tells me almost nothing by itself.

I want to know:

• What does the person see?
• What are they responsible for validating?
• Do they understand the downstream consequence?
• Do they have enough time to make the decision?
• Can they actually stop the action?
• And what happens when there is no human in the loop?

Because there will be plenty of legitimate AI use cases where requiring human review makes no sense.

In those cases, the answer is not to pretend a person is supervising every decision.

The answer is to make the system sufficiently observable that decisions can be reconstructed, reviewed, and investigated afterward.

Good AI governance should enable autonomy where autonomy makes sense.

It should not create ceremonial approval steps simply so that we can say a person was involved.

The next question is authority

Once evidence and review are designed correctly, a larger question emerges:

What is the AI actually allowed to do?

Generating information is not the same thing as recommending an outcome.

Recommending is not the same as deciding.

Deciding is not the same as taking an irreversible action.

That distinction is where AI risk starts to change dramatically.

That is the focus of Part 3.