Polished Fiction: When AI Sounds Right but Isn’t

Simon Carver

Welcome back to The Human Workforce. I’m Simon Carver, here with Lachlan Reed, and today we’re talking about a kind of failure that, honestly, doesn’t look like failure at all. It looks polished. It sounds clever. It arrives in neat little summaries, decks, roadmaps, talking points. And more and more, it’s being produced with AI.

Lachlan Reed

[skeptical] Yeah, and that’s the sneaky bit, right? It’s not the sort of wrong that rocks up wearing a clown suit. It’s the sort of wrong that turns up in a nice blazer, speaks in bullet points, and asks for a seven-figure budget. Bit rough.

Simon Carver

Exactly. And the heart of this is simple, but easy to forget: large language models are not truth engines. They’re pattern engines. They generate the next likely word, then the next likely sentence, based on what seems plausible. That is very different from checking whether something is actually true.

Lachlan Reed

[chuckles] Yeah. They’re not sitting there going, “Let me verify that for you, mate.” They’re more like, “What would a convincing answer sound like here?” And sometimes that’s useful. Sometimes it’s brilliant. But sometimes, even a kangaroo could trip over this, because the thing sounds dead right and still isn’t.

Simon Carver

And when you ask AI about complex business questions, it usually doesn’t say, “I don’t know,” or, “There isn’t enough information.” It tends to give you something coherent. Maybe a strategy. Maybe an operating model. Maybe a risk framework. It has structure, confidence, flow. It feels complete.

Lachlan Reed

[matter-of-fact] But “feels complete” is doing a lot of heavy lifting there. Real business problems are messy. They’ve got loose wires hanging out. Conflicting incentives, patchy data, history nobody wrote down properly, one exec who changed the plan in a hallway three months ago. Real life’s a shed full of spare bolts. If the answer comes back too clean, I start squinting at it.

Simon Carver

That’s such a good test. Because the danger isn’t only that AI can be wrong. We all know humans can be wrong too. The danger is that AI can be confidently, elegantly wrong. It can produce what I’d call structured fiction. Not nonsense. Not gibberish. Something much more persuasive than that.

Lachlan Reed

[softly] Yeah, convincingly incomplete. Confidently speculative. It gives you the shape of understanding without the guts of it. Like painting a door on a wall and calling it an exit.

Simon Carver

And once that output is dropped into a professional format, something strange happens. A summary becomes insight. A pattern becomes analysis. A plausible draft becomes an executive recommendation. The formatting does a kind of psychological work on us. We see order, so we assume rigor.

Lachlan Reed

Mate, chuck a logo on it, make the arrows line up, add a maturity model, and suddenly everyone in the room starts nodding like it came down from the mountain on stone tablets.

Simon Carver

Which is why this matters so much. We’re entering a moment where intelligence can be manufactured on demand, at least in appearance. But truth still has to be earned. It still requires checking, evidence, context, and someone willing to say, “Hang on, how do we know this?”

Lachlan Reed

And if nobody asks that question, you’re not looking at insight. You’re looking at a very tidy story. And tidy stories can cost a fortune when they end up steering real decisions.

Simon Carver

So how does that tidy story become an expensive one? Usually through pressure. Consulting teams are under pressure to move faster, deliver more, scale work, lower cost, keep up. AI becomes the accelerator. Decks get built quicker. Narratives get assembled faster. “Insights” arrive almost instantly.

Lachlan Reed

[deadpan] Right, and speed feels productive. That’s the trap. Everyone loves fast until fast drives the ute into a ditch. If the underlying output hasn’t been validated, then the whole engagement can become a very polished guess. And once that guess lands in an exec workshop, it stops being “a draft” and starts becoming “the strategy.”

Simon Carver

There are a few reasons this keeps working. First is authority bias. If something comes through a trusted channel, people assume it’s already been rigorously checked. Second is speed over scrutiny. Leaders want answers now. Validation takes time. Time feels expensive, even when skipping it is more expensive later.

Lachlan Reed

And third is the confidence problem. AI doesn’t naturally wave a little flag and go, “I’m shaky on this bit.” It tends to present things like they’re settled. So you get the perfect storm: fast answers, confident tone, trusted delivery channel. No guarantee it’s grounded in reality, though. That part’s optional, apparently.

Simon Carver

If you’re in a meeting and something feels off, there are some warning signs worth watching for. One: the narrative is too clean. Real organizations are messy. Two: there’s no source anchoring. No clear line from claim to evidence. Just vague references to “analysis.”

Lachlan Reed

Three: generic framework overload. Everything’s a two-by-two, a maturity curve, a phased transformation, a tidy stack of boxes. Frameworks can help, sure, but if every problem magically fits the same template, I get suspicious. That’s like saying every bike issue can be solved with duct tape. Useful sometimes. Not a religion.

Simon Carver

Four: when you ask for detail, the presenter pivots instead of drilling down. They move sideways into broader language. And five, maybe the biggest one: there’s no uncertainty anywhere. No assumptions, no risks, no unknowns, no places where someone says, “This part needs testing.”

Lachlan Reed

[serious] That one’s massive. Because in real life, uncertainty is everywhere. If a strategy has none, it’s not a strategy. It’s a story. And stories are lovely until you reorganize teams, invest in platforms, change your operating model, and then six months later reality comes knocking.

Simon Carver

And by then, the money may already be spent, the structure changed, accountability spread so thin nobody can hold it. People rarely say, “The AI got it wrong.” They say, “The transformation didn’t land.” Which is a very tidy way of hiding where the weakness started.

Lachlan Reed

So, what do you do? First: demand source transparency. Ask where this came from. What supports it? Second: separate generation from validation. Was this produced by AI, or actually checked by a human who knows the terrain?

Simon Carver

Third: reintroduce friction. Slow down critical decisions. Not everything should be instant. Fourth: test the edges. Push on assumptions. Ask the awkward questions. And fifth: require human accountability. Someone has to stand behind the recommendation, not just read it aloud.

Lachlan Reed

Simplest rule of the lot: if no one can explain how the conclusion was reached, you don’t have a conclusion. You’ve got a liability.

Simon Carver

That’s the whole challenge, really. The future belongs to people who can interrogate intelligence, not just consume it.

Lachlan Reed

Too right. Good one, Simon. We’ll leave it there for today. Catch you next time.

Simon Carver

Thanks for listening, everyone. Take care, Lachlan.

Lachlan Reed

You too, mate. Bye.