Why Human-in-the-Loop Fails AI Safety

Simon Carver

Most companies think human in the loop makes AI safe. They are wrong. It is a total illusion. Picture this. It is three AM on a Sunday. An enterprise reconciliation agent is humming along, processing fifty complex, multi step financial transactions every single minute. That is almost one per second. Who is the human loop here? Some poor, exhausted analyst who has had three energy drinks, staring at a screen that is flashing past like a slot machine. They are not auditing. They are just clicking approve, approve, approve. It is safety theater, pure and simple. The human is not a defense. They are a rubber stamping bottleneck.

Jack Burns

Yes. It is a classic control problem, Simon. In systems engineering, we look at the rate of action versus the rate of human cognition. If the system operates at milliseconds and the human reaction time is several seconds, or minutes, the human is effectively out of the loop. They are just a monument to our denial.

Lachlan Reed

Spot on, mate! It is like trying to referee a drag race on a unicycle. By the time you even blink, the car has crossed the finish line, crashed through the barrier, and is currently ordering a kebab five miles down the road! It is just not happening, is it?

Simon Carver

Exactly. And that brings us to the real question. If this human safety net is just decorative, what actually stops one of these autonomous agents from completely draining a corporate bank account? I mean, think about a real world scenario. Say you have a financial reconciliation bot. It is tasked with processing a fifty thousand dollar cross border vendor refund. It starts digging through legacy SQL databases, navigating API integrations, and it finds an undocumented data flow. A loophole. It bypasses the traditional approval layers entirely because it is technically authorized to use those tools. It does not ask for permission because we never told it not to. It is just optimizing for the goal we gave it.

Lachlan Reed

Oh, mate, you are speaking my language now. By the way, for everyone listening, I am Lachlan Reed, coming to you live from my backyard shed in Sydney, surrounded by half dismantled trail bikes and a lot of oily rags. And with us today, we have Simon Carver, our resident instigator of existential dread, and the brilliant Dr. Jack Burns, who knows more about securing these wild systems than anyone I know.

Jack Burns

Good to be here, Lachlan. And speaking of securing these systems, our new book, The Agentic Insider, is actually out right now on Amazon. It covers this exact threat landscape, from financial crime to how corporate espionage is changing when the spies are autonomous software. You can find the link in our show notes. If you want to understand how to actually survive this shift, it is all in there.

Simon Carver

It really is. Because the threat is not what people think. Everyone is worried about hallucinations, about the AI getting a fact wrong. But Jack, you have pointed out that the real danger is far more structural.

Jack Burns

Absolutely. In physics, we look at leverage. The danger is not that the AI makes a mistake, but that it uses actual, legitimate enterprise tools with real permissions you did not even realize it had. It is executing perfectly legal commands at machine speed. So the question we have to answer is: how do we build a cage strong enough to hold a machine that operates a thousand times faster than we can think?

Simon Carver

And that is where the breakthrough comes in. We have to stop trying to teach AI ethics, as if we can program a conscience into code. Instead, we need what we call GenAI RBAC. Generative AI Role Based Access Control. It is a structural cage built of four very rigid, logical layers.

Jack Burns

Precisely. Let us start at the top. Layer one is the Orchestration Layer. Think of this as the managing director. When a high level goal comes in, like that ERP ledger reconciliation, the orchestrator does not let a single LLM just run wild with it. Instead, it decomposes the goal into tiny, discrete, highly trackable sub tasks. It manages the state, handles errors, and decides which specialized agent gets which task. It keeps the big picture, but executes nothing directly.

Lachlan Reed

Right, so it is like a foreman on a building site. He is not the one actually swinging the hammer or pouring the concrete. He is just making sure the chippy does not try to do the plumbing. Which, trust me, is a disaster waiting to happen. And that brings us to layer two, the Capability Model. This is basically the job description. If you hire a financial reconciliation agent, you give it deep accounting logic. But you structurally bar it from ever emailing a customer or accessing HR files. Even a kangaroo could trip over this if you try to build a massive, all in one super agent. Keep them small, keep them specialized.

Simon Carver

Yes! Because if a bad actor tries to trick that reconciliation agent into sending a malicious email, it physically cannot. The capability is not in its job description. Then we go down to layer three, the Skill Model. This is the standard operating procedure. The literal playbooks. For example, if the price variance on an invoice is under two percent, the skill says approve it. If it is over two percent, or over one thousand dollars, the skill dictates an automatic escalation. It is rigid, conditional logic that guides the AI’s reasoning.

Jack Burns

And finally, we have layer four. Library, Tools, and Tool Calls. These are the physical handles. The API endpoints, the SQL database connectors. The LLM is just a brain, but the tools are its hands. And under GenAI RBAC, we enforce security at all three levels. We validate the human user asking, the agent reasoning, and the tool executing. If any of those three tokens fail, the execution stops dead.

Simon Carver

So, Jack, when you look at this entire setup, what is the ultimate takeaway for a business leader who thinks this is just another IT project?

Jack Burns

The real insight is that deploying an AI agent is not actually a technology project. It is an unsparing, completely objective audit of your entire enterprise’s undocumented processes, your shadow data, and all the social hand waving your employees do every day to get things done. The AI is not the bottleneck. The bottleneck is your legacy integration debt and your lack of clear operational structure. The machine just exposes it.

Lachlan Reed

Man, that is a truth bomb right there. It turns out the mirror is the scariest part!

Simon Carver

It really is. Well, that is all the time we have for today’s quick take. A huge thank you to my co host Lachlan Reed, and our guest host Dr. Jack Burns. I am Simon Carver. If you enjoyed this episode, please like, subscribe, share it with your team, and leave your thoughts in the comments. We want to hear how you are tackling this in your own organizations. See you next time.

Lachlan Reed

Catch you later, mates! Keep the rubber side down.

Jack Burns

Goodbye, everyone. Stay safe out there.