AI’s Gray Rhinos and Black Swans

Simon Carver

[warmly] Welcome to the show. I’m Simon Carver, here with Lachlan Reed, and Lachlan, I keep coming back to one line for this episode: the biggest risk in AI is not the mystery over the horizon. It’s the thing already in the road, headlights on, engine loud, and leaders still saying, “Let’s circle back next quarter.”

Lachlan Reed

[matter-of-fact] Yeah, that’s it. It’s not some robot apocalypse caper. It’s the stuff you can see with your own eyes right now. In this conversation, the gray rhino is the obvious beast charging straight at the business. Not sneaky. Not theoretical. Just big, visible trouble that people ignore because changing course is a pain in the neck.

Simon Carver

[curious] And the “gray rhino” part matters because it flips the usual fear story. We spend so much time worrying about the unknown thing AI might do someday, while the known thing is already chewing through operating models today. So let’s name the animals on the floor. What are leaders already facing?

Lachlan Reed

[calm] First one: governance that cannot keep up with agentic AI. And by that I mean systems that don’t just answer questions, but take actions, hand work to other tools, and make little decisions inside a workflow. A lot of companies still govern AI like it’s a chatbot in a sandbox. But if the system can act, approve, route, summarize, escalate, maybe even trigger spending or customer contact... mate, that’s not a toy anymore. That’s an operator.

Simon Carver

[questioning tone] “Act, approve, route” — that trio is the part that sticks with me. Because once a system can route and approve, accountability gets blurry fast. Who owns the outcome then? The product team? The vendor? Legal? The manager who never even saw the step happen?

Lachlan Reed

Exactly. And that blurry patch is where real organizations get bogged. You’ve got overloaded teams, everyone moving flat out, and then AI gets dropped in like a new apprentice with no supervisor. People say “we’re experimenting,” but what they often mean is, “nobody’s fully in charge.” That’s not strategy. That’s a shed full of power tools and no one’s read the manual. I say that as a bloke who records near actual power tools, so... [chuckles] I know the vibe.

Simon Carver

[reflective] And then there’s middle management, which is such an awkward subject because nobody wants to say it too plainly. But a lot of the coordination layer — the people who translate strategy into tasks, catch ambiguity, notice when a team is drifting — that layer is getting squeezed. Not always eliminated outright. Sometimes just hollowed out.

Lachlan Reed

[skeptical] Hollowed out is the right phrase. Because firms look at AI and think, beauty, faster reporting, faster approvals, fewer meetings. Fair enough. But middle managers also do the messy human stuff: context, coaching, judgment, conflict, escalation. If you strip that layer back without redesigning the work, you don’t get a sleek machine. You get confused accountability. Everyone’s busy, nobody owns the weird bits, and the weird bits are where the risk lives.

Simon Carver

[softly] “The weird bits are where the risk lives.” I’m keeping that. Because in enterprise life, failure rarely arrives as one dramatic explosion. It arrives as ten small uncertainties no one feels authorized to stop.

Lachlan Reed

Spot on. And then the third rhino: infrastructure strain. Compute, energy, scalability limits. Leaders can talk a big game about AI everywhere, but the back end has to carry the load. More models, more inference, more data movement, more cost, more pressure on systems that were already creaking. So the front of house says, “AI-first,” and the back of house is quietly muttering, “With what capacity?”

Simon Carver

[responds quickly] “Compute, energy, scalability” — that’s not branding language, that’s physics. And physics does not care about your quarterly narrative. If the infrastructure can’t support the ambition, then “AI strategy” becomes a stack of pilots, demos, and approvals waiting for a machine that never quite arrives.

Lachlan Reed

Yep. Lots of organizations are running three things at once: overloaded teams, unclear accountability, and experimentation without control. That combo feels productive because activity is high. But high activity isn’t the same as operating discipline. You can stir the billy all day — if there’s no recipe, dinner’s still cactus.

Simon Carver

[urgent] And that’s the warning. These are not future risks. They are happening now. Leaders are delaying action not because the danger is invisible, but because the redesign is uncomfortable. Governance is dull. Clarifying ownership is political. Reworking management layers is painful. But ignoring a charging rhino does not make you strategic. It just makes you late.

Simon Carver

[serious] And once you’ve got those visible risks in view, there’s a darker category behind them: the black swans. Not the obvious threats. The rare, systemic shocks — the kind that don’t announce themselves politely and may not be manageable with better dashboards or prettier reporting.

Lachlan Reed

[hesitates] Yeah... and this is where even a kangaroo could trip over the intro a bit, because the hard part is that these risks are fuzzy until they’re not. One example is model collapse: AI learning from AI, over and over, until the quality degrades. The system starts feeding on synthetic leftovers instead of solid human-grounded input. Looks fine for a while. Then the edges go weird.

Simon Carver

[questioning tone] “Synthetic leftovers” is a strong image. So let me try to say it back. If models increasingly train on content produced by other models, the loop can reinforce errors, flatten nuance, and gradually poison the well. Not overnight — more like a photocopy of a photocopy of a photocopy.

Lachlan Reed

[pleased] That’s exactly it. The first copy’s fine. By the fifth, the detail’s gone. And because it degrades gradually, leaders may not spot it early. They’ll just notice outputs getting thinner, stranger, less reliable. Not always broken enough to stop the system — just off enough to matter.

Simon Carver

Then there are algorithmic cascades, which feel much more immediate. One system triggers another, that one updates a third, and suddenly you have interacting automations producing outcomes no single team intended. This is why the comparison to financial flash crashes matters. The speed is the story. By the time a human sees the pattern, the pattern may already have propagated.

Lachlan Reed

[matter-of-fact] Same with cyber incidents that spiral. One machine event becomes a system event before anyone’s had time to make a cup of tea. And if you’ve built for seamless automation without proper brakes, the cascade moves faster than governance can respond. That’s the killer bit. Not just that something fails, but that it fails at machine speed while humans still work at meeting speed.

Simon Carver

[grimly] “Machine speed, meeting speed.” There it is. A board deck is not a control system. A policy PDF is not a live intervention. And emergent behaviors make this harder still, because they don’t always follow the logic the designers expected. The system does something effective enough to continue, but strange enough that nobody can cleanly explain why.

Lachlan Reed

Which brings us to the leadership failure point. Leaders are stuck between ignoring the obvious and being unprepared for the unpredictable. So what do they do? Too often, they lean on vendors for reassurance. “The provider has guardrails.” “The platform handles that.” Maybe. But you cannot outsource accountability just because the sales deck is tidy.

Simon Carver

[firm] And you cannot automate your way out of responsibility. That line should be pinned to every executive agenda. Over-reliance on vendors, weak human-in-the-loop safeguards, and this persistent belief that efficiency equals strategy — that trio is its own risk stack. Efficiency is a metric. Strategy is a choice about what must remain human, what can be delegated, and what must fail safely.

Lachlan Reed

[reflective] Fail safely — that’s the phrase I wish more leaders would sit with. Because the answer isn’t panic. It’s preparation. Move out of the rhino’s path: fix governance now, redesign operating models, stop pretending current structures will somehow hold under new pressure. And prepare for the swan: build redundancy, keep human oversight, design for failure instead of perfection.

Simon Carver

[warmly] That’s the Human Workforce mission, really. Not machine worship. Not machine fear. Clear-eyed responsibility in the age of AI, with human judgment still carrying the weight it should carry.

Lachlan Reed

Thanks for listening, folks. If this gave you something to chew on, follow along for more grounded conversations about AI and work — the real stuff, not the glossy brochure version.

Simon Carver

[softly] This is not about predicting the future. It’s about surviving it — and most organizations are not ready.