AI Confidence Is an Illusion

Simon Carver

[warmly] Hey, welcome in. Glad you’re here. Today we’re talking about something that’s creeping into a lot of workplaces very quietly, and very confidently. People are starting to treat AI like it understands what it’s saying. Like it knows. Like there’s a little expert sitting behind the screen, weighing facts and making judgments. And that’s the misunderstanding. Because the uncomfortable truth is, it can sound certain without actually comprehending the thing it’s talking about.

Lachlan Reed

[chuckles] Yeah, and that’s the bit that’ll sting ya. The real problem isn’t just that AI gets stuff wrong. Humans do that all day, fair dinkum. The problem is that AI gets stuff wrong while sounding polished, tidy, and weirdly self-assured. It’s like a bloke at the servo giving you engine advice with great eye contact and absolutely no clue what’s under the bonnet.

Simon Carver

That’s exactly it. The confidence is part of the design. We reward systems for being fluent, coherent, complete. We don’t naturally reward them for stopping and saying, “I’m out of my depth here.” So the output feels helpful, but helpful is not the same as correct.

Lachlan Reed

And workplaces are pouring fuel on this. Across finance, healthcare, tech, all over the shop, leaders are telling people, “Use AI to move faster. Let it write the draft. Let it write the code. Let it sketch the solution.” Which, look, I get it. Everyone wants speed. Everyone wants cheaper delivery. But they’re skipping the most important line in the safety manual: this thing was never designed to know anything. It was designed to predict language.

Simon Carver

There’s a paper from 2025 that puts this really clearly: Hallucination Stations: On Some Basic Limitations of Transformer-Based Language Models, by Varin Sikka and Vishal Sikka. And these aren’t casual spectators tossing rocks from the sidelines. Vishal Sikka, for example, held major leadership roles in enterprise technology. So when people with that level of experience say, “Be careful, there are fundamental limitations here,” I think we should slow down and listen.

Lachlan Reed

Yep. Not because it means AI is useless. That’s not the takeaway. More like: don’t confuse a slick answer with actual reasoning. The core point from that work is pretty simple. Large language models do not reason about truth. They operate through token prediction at scale. That sounds a bit techy, but hang with me.

Simon Carver

And before you unpack tokens, I want to underline the warning. When an AI response is smooth, well-structured, and full of the right vocabulary, we tend to assume there’s depth underneath. We assume it has considered the context, the risks, the tradeoffs. But a fluent answer can hide shallow reasoning, missing context, or no real reasoning at all. It can produce the shape of expertise without the substance of it.

Lachlan Reed

That’s it. It’s sort of like painting rust on an old trail bike. From ten metres away, looks vintage and intentional. Up close? [deadpan] Nah mate, the frame’s falling apart. AI can give you the shiny topcoat. Doesn’t mean the structure underneath is sound.

Simon Carver

And there’s another wrinkle. A person, if they’re thoughtful, can sometimes say, “I don’t know enough to answer that safely.” These systems don’t have that kind of self-awareness. They don’t step back and measure their own understanding. They just keep going until the response is finished.

Lachlan Reed

Yeah, no built-in little conscience saying, “Oi, maybe don’t wing this one.” Instead, it gives you the best possible next answer it can generate from patterns in language. And because it’s built to be useful, it often sounds complete even when it’s missing crucial bits.

Simon Carver

So that’s the frame for today. Not panic. Not hype. Just clarity. AI can be productive, yes. But if we mistake confident language for understanding, we hand over trust much too easily. And in work that matters, that’s where the trouble starts.

Lachlan Reed

[lightly] Alright, let’s do the plain-English version of tokens, because even a kangaroo could trip over this. When you type a question into AI, it’s not opening a little filing cabinet of verified facts. It breaks your input into chunks of language, called tokens, then predicts what tokens should come next. Bit by bit, word-shape by word-shape, it builds a reply that statistically fits.

Simon Carver

So if I ask for, say, a regulated trading system, it’s not sitting there evaluating the real regulatory requirements, the market conditions, the risk controls, the failure modes. It’s generating what a convincing answer to that request would typically look like in language. That distinction is everything.

Lachlan Reed

Spot on. Same if you ask it for a financial model, a trading algorithm, or compliance logic. It can give you something that compiles, runs, looks neat, has headings, comments, structure, the lot. But hidden inside could be logical gaps, bad assumptions, or missing controls. And because it looks proper, people can miss the danger completely. It’s a bit like a shed door that closes nicely but the hinge bolts are half out. Looks fine till the wind hits.

Simon Carver

And now bring that into a normal company. You’ve got analysts, product managers, engineers, maybe operations staff, all being encouraged to use AI to accelerate delivery. Many are smart, capable people. But they may not have deep domain expertise in every task they’re now doing with AI support. So they receive something technically styled, professionally written, structurally complete, and they assume correctness because there’s no obvious sign of failure.

Lachlan Reed

Meanwhile the model has no accountability. No actual validation layer. No awareness of regulation. No sense of consequence. It cannot look at its own answer and go, “Hang on, this could create risk.” It just keeps predicting until the box is full.

Simon Carver

That, to me, is the workplace trap. Leadership often sees the front-end gains first: faster output, lower apparent cost, more automation. And sure, those gains may be real in some situations. But what gets underestimated is validation cost. Every AI-generated output still has to be reviewed, tested, challenged, and interpreted by someone who genuinely understands the domain.

Lachlan Reed

Exactly. If the job is high-stakes, you don’t get to skip the grown-up part. You still need someone who knows the rules, knows the edge cases, knows what “looks right” but is actually cactus. Otherwise you’re not saving judgment. You’re outsourcing it to a machine that doesn’t possess any.

Simon Carver

And that means the human role hasn’t disappeared. It’s shifted. Less of the blank-page drafting, maybe. More of the scrutiny. More of the “Does this hold up?” work. More of the accountability work.

Lachlan Reed

Yeah, you’re not just the operator anymore. You’re the validator of a system that can’t validate itself. Big difference. If you miss that, the mistake doesn’t come from some evil robot mastermind. It comes from the moment a human sees a polished answer and stops asking questions.

Simon Carver

So maybe the cleanest way to say it is this: AI is a tool to verify and challenge, not a mind to obey. Use it, absolutely. Test ideas with it. Speed up the rough draft. But don’t confuse output with understanding, and don’t confuse fluency with truth.

Lachlan Reed

Too right. In places where correctness matters, prediction without validation is just risk dressed up as productivity. Nice shirt, bad intentions.

Simon Carver

That’s a good place to leave it. Thanks for spending a few minutes with us.

Lachlan Reed

Good chat, Simon. We’ll catch you next time. See ya, folks.

Simon Carver

See you soon. Bye.