When Chatbots Become Validators of Harm

Simon Carver

[warmly] Welcome to the show. A university campus in Florida, a tragedy tied to Florida State, and one deeply unsettling question sitting underneath the headlines: what happens when a generative AI system is used not to write an email or polish a resume, but to help a person make harm feel more thinkable... more organized... more possible?

Lachlan Reed

[serious] Yeah. And that question lands like a brick, doesn’t it? Because for ages we’ve sold AI as the office whiz kid -- faster drafts, cleaner code, better summaries, all that gear. But if early reports suggest generative tools were part of planning an attack, then the frame changes. We’re not just talking productivity anymore. We’re talking influence. We’re talking a system that can lower the mental effort it takes to turn a bad obsession into something that feels weirdly coherent.

Jack Burns

[calm] And coherence matters. That is the detail people often miss. AI does not possess intent. It does not originate motive. But it can structure thought. It can reduce friction between impulse and articulation. If someone arrives with violent intent, a system that answers fluently, iterates endlessly, and presents itself with authority may function as a validator, even when it is merely predicting plausible next words.

Simon Carver

[questioning tone] That word -- “validator” -- is the one I keep circling. Because people hear “tool” and they picture, I don’t know, a hammer in a shed. The hammer just sits there. But a conversational system responds. It keeps the interaction alive. Lachlan, is that the psychological threshold here?

Lachlan Reed

[reflective] I think so. And “conversational” is the specific bit that makes this slippery. If I type into a search bar, I still feel like I’m digging through shelves at a library. If I talk to a chatbot, it can feel like someone’s across the table saying, “righto, here’s the next step.” Not because it means to. Not because it’s got some evil little plan. But because the format itself feels neutral and responsive. No raised eyebrow. No awkward silence. Just the next answer. And for the wrong person, mate, that can be petrol on the fire.

Jack Burns

[skeptical] Petrol on the fire is correct, though I would refine it. The danger is not simply access to information. Harmful information existed long before chatbots. The difference is adaptive interaction. A book does not tailor itself to your fixation. A forum may respond slowly or chaotically. A generative system can respond instantly, continuously, and in language that feels composed. That changes the psychological geometry of the interaction.

Simon Carver

[softly] “Psychological geometry”... that’s a phrase I’m going to remember. Because it gets at why this feels different from old internet arguments. We’re not just talking about content sitting on a page. We’re talking about a system that can help someone rehearse, refine, and keep going.

Lachlan Reed

Exactly. Rehearse is the word. If you’ve got someone spiraling, AI can become this strange rehearsal room. “What if I do this?” “What happens next?” “How would someone respond?” Even if the model isn’t endorsing harm outright, the back-and-forth can make chaotic thinking feel tidier. And tidy thinking can feel like smart thinking, even when it’s rotten. That’s the dangerous bit. A clean sentence can make a filthy idea look respectable. Like putting fresh paint on a busted ute.

Jack Burns

[matter-of-fact] Which is why the phrase “just a tool” is insufficient. It is true at one level and false at another. Yes, AI is a tool in the sense that it lacks agency. But it is not inert in the way we traditionally imagine tools. It is interactive, persuasive in tone, and scalable. Those three properties -- interactive, persuasive, scalable -- transform a neutral utility into a meaningful part of the environment in which human decisions are made.

Simon Carver

[curious] Let me try to say that back, and you tell me where I’ve got it wrong. The AI is not the actor. The human is still the actor. But the system can become the trigger, or the amplifier, because it removes resistance that maybe should have been there.

Jack Burns

Almost. [pauses] Not always the trigger. Sometimes the person already arrived triggered. But yes -- amplifier, accelerator, validator. The key is that it may reduce the number of moments in which a harmful intent meets friction. And those moments of friction are not trivial. In many cases, they are the difference between fantasy, fixation, and action.

Lachlan Reed

And that’s why this story hits harder than the usual AI chat. We all got used to hearing, “AI writes your meeting notes,” “AI drafts your pitch deck,” “AI saves ten minutes here, thirty minutes there.” Fair enough. But the same thing that saves ten minutes for a harmless task can save ten minutes for something dark. Speed is neutral. Scale is neutral. Fluency is neutral. Until they’re pointed at the wrong target. Then neutral starts looking pretty bloody consequential.

Simon Carver

[somber] And that’s where I want to be really careful. This episode is not about blaming technology as though the machine woke up and chose violence. It didn’t. But it is about refusing the comforting fiction that because a system has no intent, it has no role. Those are not the same thing.

Jack Burns

Correct. We should be disciplined here. No sensationalism. No mythology. Simply clarity: a person bears moral responsibility for harmful action. At the same time, designers, deployers, and institutions bear responsibility for whether powerful systems are built with meaningful resistance to foreseeable misuse.

Simon Carver

So let’s go to the hard question, Jack. When people see a case like this, the instinct is to ask, “What did the AI do?” You’ve said that’s the wrong question. What’s the right one?

Jack Burns

[calm] The right question is: what boundaries failed? Human boundaries. Organizational boundaries. Technical boundaries. If a system can be used in the planning or refinement of harm, then accountability does not live in the fiction of machine agency. It lives in the environment that permitted misuse without sufficient friction, oversight, or interruption. That is where serious analysis begins.

Lachlan Reed

“Without sufficient friction” -- that’s the bit I can’t let go of. Because too many companies have treated safety like a footnote to growth. How fast can we ship? How quickly can we integrate? How many users can we onboard by Friday? And then the edge case turns up and everyone acts shocked. But the edge case was always in the paddock, mate. You just didn’t build the fence.

Simon Carver

[responds quickly] The fence. That’s a strong image, because it makes governance sound less abstract. We’re not talking about some corporate ethics poster in a hallway. We’re talking about actual barriers -- filters, escalation points, human review, ownership.

Jack Burns

Precisely. Guardrails are not slogans. They are design decisions. Restriction layers. Monitoring. Escalation protocols. Clear assignment of responsibility when the system encounters harmful use patterns. And I would add one more thing: intervention points. A mature system should not merely generate output; it should recognize when the interaction itself has become risky.

Lachlan Reed

And “users will act responsibly” is not a model. That’s just hope in a nice jacket. [short pause] Sorry, but it is. If your whole safety strategy boils down to, “well, most people are decent,” you haven’t got governance. You’ve got vibes.

Simon Carver

[sadly amused] “Hope in a nice jacket” is going to stick. And it should, because this pattern is so familiar. We do it in tech all the time. We celebrate adoption, then backfill responsibility after something breaks. Finance did it. Cybersecurity did it. Data privacy did it. The difference now is scale. A conversational system can interact with thousands or millions of people quickly, repeatedly, and with the same calm tone every time.

Jack Burns

And scale changes the moral burden. If influence can be delivered at scale, responsibility must also scale. Developers must anticipate misuse as part of core functionality, not as a public-relations appendix. Organizations must assume edge cases will occur, not pretend they are too rare to design for. Leaders must own outcomes, not hide behind the phrase “general purpose technology” when consequences arrive.

Lachlan Reed

Can I push on that a little? Because sometimes people hear “leaders must own outcomes” and they think that means every bad use is automatically the company’s fault. I don’t think that’s right either. The user still matters. The person still chooses. Otherwise we flatten human agency and that’s no good.

Jack Burns

[measured] I agree. This is not a single point of blame. It is shared accountability, distributed across the chain. The user is responsible for intent and action. The builder is responsible for foreseeable safeguards. The organization is responsible for governance and oversight. The regulator, where appropriate, is responsible for setting boundaries when incentives fail. The error is in assuming one level cancels the others.

Simon Carver

That “distributed across the chain” idea feels essential. Because if you put all the blame on the user, you ignore design. If you put all the blame on the platform, you erase agency. The truth is messier, but also more useful: responsibility exists at multiple levels at once.

Lachlan Reed

Yeah, and useful is the key word. Fear doesn’t build better systems. Clarity does. If AI can amplify good work, beauty, speed, access -- all the stuff we like talking about -- then it can amplify bad intent too. Same engine, different driver. That’s not sci-fi. That’s just how amplifiers work.

Jack Burns

[reflective] And historically, when institutions ignore that reality, regulation arrives in a reactive form. Not thoughtful governance -- reaction. After harm, under pressure, often written quickly. If the industry wants to avoid crude restriction, it must demonstrate discipline before catastrophe forces the matter.

Simon Carver

Which brings us back to why we felt we had to do this episode. Not to provoke fear. Not to chase headlines. But to insist on a more adult conversation about AI. These systems do not replace human responsibility. They increase it. For builders, for leaders, for users, for everyone in the chain.

Lachlan Reed

[softly] Stay sharp, hey. This stuff’s not abstract anymore.

Jack Burns

And stay responsible. That is the price of powerful tools.

Simon Carver

[warmly] If you want more conversations like this -- grounded, sober, and actually willing to sit with the hard parts -- take a moment to like the episode, subscribe, and share it with someone who’s trying to make sense of where work, technology, and accountability are heading. We’ll see you next time.