When AI Starts Training on Its Own Echo

Simon Carver

[warmly] Welcome to the show. This quick take is called When the Well Runs Dry: The Collapse of AI's Human Fuel. And before we go any further, if these conversations help you think a little more clearly about work, tech, and what comes next, please like, share, and subscribe. Alright -- Lachlan, here's the image I can't shake: for years, AI was drinking from a deep well of HUMAN material. Books. Journals. documented arguments. Enterprise records. Real people, saying real things, refining them over time. But what happens when the bucket comes back up full of machine-made echo instead?

Lachlan Reed

[curious] Yeah, that's the bit that gives me the wobbles, mate. Because once AI stops learning mostly from humans and starts learning from OTHER AI, you're not getting fresh water anymore -- you're getting yesterday's tea poured back through the leaves. Looks like liquid, still wet, but the flavour's gone.

Simon Carver

[reflective] Exactly. And I think a lot of organizations still talk about AI like the success was inevitable, like intelligence just emerged because the math got big enough. But the early gains were not magic. They were built on a very particular raw material: high-quality human-generated data. Decades of edited books, peer-reviewed journals, verified knowledge bases, structured systems inside companies. That was the fuel.

Lachlan Reed

[responds quickly] And now the fuel's being diluted -- actually, more than diluted, replaced. You can see the loop forming in plain sight. AI writes the blog post. Another AI summarizes that blog post. A third tool rewrites the summary into social captions and a video script. Then that whole pile of derivative mush lands online as if it's fresh knowledge. That's the synthetic loop right there.

Simon Carver

[questioning tone] Let me try to pin that down. You're saying the problem isn't synthetic data by itself. It's the CLOSED loop -- humans create the original knowledge, AI trains on it, AI produces derivative content, and then that derivative content gets swept back in as new training data... without a human checking whether it's still anchored to anything real.

Lachlan Reed

[matter-of-fact] That's it. One, humans create something original. Two, AI trains on it. Three, AI spits out derivative content. Four, that content becomes training data. Five, repeat. Every lap around the track, originality drops a notch, accuracy drifts a bit, but confidence stays weirdly high. That's the nasty part -- convincing and wrong at scale.

Simon Carver

[softly] Convincing and wrong at scale. That's the phrase. Because listeners have probably felt this already. You read something online, and it's perfectly polished. Nice rhythm, neat bullets, sounds authoritative. But then you tug on one thread and realize... there's no actual source underneath it. It's all surface.

Lachlan Reed

[chuckles] Yeah, it's like a showroom bike with no engine in it. Shiny paint, tidy seat, not going anywhere. And AI doesn't naturally know that a blog post scraped from three other blog posts is not the same as a book, or a journal article, or a properly governed enterprise record. Unless humans draw those lines, the model goes, "Sweet as, all text is text."

Simon Carver

[skeptical] And that's where the tension gets sharp. Because people hear "more data" and assume "better model." But if a bigger share of that data is synthetic, unverified, repetitive, or just plain wrong, scale doesn't save you. It can actually harden the error.

Lachlan Reed

[excited] Right! More AI does NOT automatically mean better outcomes. More AI without governance means more risk. More automation without checkpoints means more fragility. More synthetic data means less truth. Even a kangaroo could trip over that one, and honestly, so could half the market right now.

Simon Carver

[serious] Here's why this isn't just a nerdy model-training debate. Studies -- including ones acknowledged by AI developers themselves -- have pointed to hallucination rates around 16% in some settings, and in certain contexts or domains, nearly 50%. Sixteen percent is already uncomfortable. Fifty percent means, in some situations, every other confident answer could be off. That's not refinement. That's regression.

Lachlan Reed

[interrupts] Wait -- nearly 50% is the number that sticks for me. Because once you're flirting with one-in-two being wrong depending on context, you're not talking about a clever assistant anymore. You're talking about a coin flip in a business shirt. [scoffs] That's rough.

Simon Carver

And the danger is that it doesn't sound like a coin flip. It sounds calm. Competent. Helpful. That's why this becomes a HUMAN problem. A flawed recommendation doesn't just sit in a chat window. In an enterprise, it can misroute millions in payments. A hallucinated compliance interpretation can create regulatory exposure. A corrupted dataset can cascade through multiple decision systems before anyone notices.

Lachlan Reed

[calm] Yeah, and that's what we've called Ethical Debt. Not just one bad answer -- accumulated risk from taking humans out too early. You skip validation because the tool seems fast. You automate because the pilot looked good. Then six months later you've built a stack of decisions on top of shaky outputs, and now the whole thing's a bit like building a shed on sand. Looks fine till the storm rolls in.

Simon Carver

[curious] Let's make the knowledge-quality point concrete, because this matters. A blog is not a verified knowledge source. A TikTok is not a biography. A social reel is not a book. An AI summary of an article is not the same thing as the article. Those aren't snobby distinctions. They're about provenance, editing, evidence, and accountability.

Lachlan Reed

[deadpan] Yeah, a 30-second reel with dramatic music is not the moral equivalent of a journal. Sorry to the internet. [laughs] But seriously, if a model treats a verified knowledge base and a mass-generated content farm as equal inputs, truth gets flattened. Everything's on the same shelf. And when everything's equal, the loudest nonsense can crowd out the quiet, verified stuff.

Simon Carver

[questioning tone] So for leaders listening, the practical question is not "Should we use AI?" That's too blunt. It's "Where must humans stay in the loop, and which information sources deserve more trust than others?"

Lachlan Reed

[matter-of-fact] Yep. Three non-negotiables. First: reintroduce human validation layers in critical paths -- not everywhere, but definitely where money, law, safety, or reputation are on the line. Second: classify your data sources. Verified knowledge like books, journals, and structured systems at the top. Semi-verified sources like curated industry blogs in the middle. Unverified stuff -- social content, mass-generated media -- at the bottom. Don't let the machine treat those as the same bloody thing.

Simon Carver

[responds quickly] And third: define no-AI zones. High-risk compliance decisions. Medical or safety-critical outputs. Complex ethical judgments. There are areas where speed is not the prize. Reliability is the prize. Accountability is the prize.

Lachlan Reed

[reflective] And here's the ironic twist, hey: as the internet gets more saturated with synthetic noise, human-originated knowledge becomes MORE valuable, not less. Original thinking. Verified expertise. Structured reasoning. Scarcer means dearer. The good oil is harder to find, so it matters more.

Simon Carver

[warmly] Which leads to the bigger conclusion. The future isn't AI replacing humans. It's AI requiring better humans -- more discerning humans, more responsible humans, humans who know when to trust a system and when to stop it cold. The first era of AI was built on truth. The next era could be built on imitation, unless we intervene deliberately.

Lachlan Reed

[softly] And that's the quiet risk, isn't it? Not that AI fails loudly. It's that it succeeds just well enough to be trusted, while carrying enough error to do real damage. That's the snake in the grass.

Simon Carver

[warmly] If this challenged your thinking, good. Share it with someone building with AI right now.

Lachlan Reed

And if you liked the episode, subscribe and send it on. We'll catch you next time on The Human Workforce. [short pause] Stay human, mate.