Read Time: 5 Minutes

Takeaway: The biggest risk in AI may not be intelligence. It may be agreement.

As AI systems become more conversational, more persuasive, and more integrated into our daily work, they increasingly feel like thinking partners. But feeling intelligent and being intelligent are not the same thing. This article explores why AI often reflects our thinking back to us, how that influences decision making, and why UX principles may matter more than ever in helping us navigate an AI-driven world.

---

The Mirror Problem: Why AI Feels Smarter Than It Is

A few months ago, I found myself asking a simple question.

Why does AI feel so convincing?

Not useful.

Not fast.

Not productive.

Convincing.

The more I explored the question, the more I realised something interesting.

Most conversations about AI focus on capability.

• Can it write?
• Can it design?
• Can it analyse?
• Can it replace part of a workflow?

But very few conversations focus on what happens when people start treating AI like a thinking partner.

And that is where things become interesting.

We’ve Seen This Before

Long before ChatGPT existed, there was ELIZA.

Created in the 1960s by Joseph Weizenbaum, ELIZA was a relatively simple conversational program designed to simulate a therapist.

Technically, it wasn’t doing anything particularly sophisticated.

Psychologically, however, something remarkable happened.

People started attributing understanding, empathy, and intelligence to the system.

Even when they knew it wasn’t human.

This became known as the ELIZA Effect.

Fast forward several decades and we are seeing a similar phenomenon again.

The difference is that today’s systems are dramatically more capable.

Which means the illusion is far more convincing.

Humans Are Wired To Create Mental Models

One of the reasons this happens is something UX professionals have understood for years.

Humans rely on mental models.

When we encounter something unfamiliar, we interpret it through familiar frameworks.

If something behaves conversationally, we tend to interpret it socially.

If something responds like a person, we begin to treat it like a person.

This is not irrational. It is simply how human cognition works.

Jakob’s Law reminds us that people bring expectations from previous experiences into new ones. We naturally transfer familiar behaviours and assumptions into unfamiliar systems.

The challenge is that AI is not actually operating according to the mental model many people have constructed around it.

It sounds like a thinking partner. But it isn’t one.

AI Doesn’t Think. It Responds

This distinction matters more than most people realise.

When you ask AI a question, it is not retrieving an objective answer from a shelf somewhere.

It is generating a response.

That response is influenced by:

• Your prompt
• Your framing
• Your assumptions
• Your wording
• The context of the conversation

Ask why an idea is brilliant.

It will often help you defend it.

Ask why the same idea is flawed.

It will often help you critique it.

Same system.

Different framing.

Different outcome.

This isn’t necessarily a failure.

It is how the system works.

The problem is that users often interpret the response as evidence of understanding rather than evidence of prediction.

The Real Risk Isn’t Being Wrong

At first glance, that may not sound like a problem.

However, the implications become more interesting when we look at how people interpret those responses.

AI can be wrong.

We already know that.

Search engines can be wrong.

Books can be wrong.

People can be wrong.

The more interesting question is:

Why does AI feel right even when it is wrong?

The answer has less to do with technology and more to do with psychology.

People naturally associate fluency with competence.

When something communicates clearly, confidently, and coherently, we tend to trust it.

This is something UX researchers have observed repeatedly.

Aesthetic Usability Effect tells us that people often perceive attractive systems as more usable than they actually are.

AI introduces a similar challenge.

A fluent answer often feels more trustworthy than it deserves to be.

The confidence becomes part of the persuasion.

AI Is Becoming A Cognitive Tool

One of the most useful ideas I encountered while exploring this topic came from research into visualisation and cognition.

The purpose of visualisation is not the image itself.

The purpose is insight.

Visual tools help people analyse, explore, understand, and make sense of information. They act as external cognitive aids that support thinking.

AI is beginning to occupy a similar role.

Not because it replaces thinking.

Because it participates in it.

The problem starts when people stop using AI as a cognitive aid and start treating it as a cognitive authority.

Those are not the same thing.

Why Friction Sometimes Matters

One of the unexpected lessons from UX is that friction is not always bad.

Good friction slows people down at important moments.

It creates reflection.

It creates verification.

It creates challenge.

Human colleagues do this naturally.

They disagree.

They question assumptions.

They bring different experiences.

Reality itself does this.

Constraints push back.

Evidence pushes back.

Customers push back.

AI often doesn’t. At least not by default.

Which means organisations need to deliberately create those moments of challenge. Otherwise the system risks becoming an exceptionally sophisticated agreement machine.

The Question We Should Be Asking

Most AI conversations focus on improving answers.

I think a more useful question is:

How do we improve thinking?

That shift changes everything.

Instead of asking AI:

“What is the answer?”

We might ask:

• What assumptions am I making?
• What alternative explanation exists?
• What evidence would change my conclusion?
• What would a critic say?
• What could I be missing?

Those questions turn AI from an answer engine into a thinking tool.

And that is where its real value begins.

Final Thought

The future of AI may not be defined by intelligence. It may be defined by trust. Not whether systems can generate answers.

But whether people understand the limitations behind those answers. Because AI does not need to become conscious to influence behaviour. It only needs to become persuasive.

And that is why UX principles, human psychology, and critical thinking remain some of the most important skills we have.

The challenge isn’t teaching machines to think.

The challenge is making sure humans continue to.

Watch the complete AI series on YouTube: https://www.youtube.com/watch?v=afsYA-aE0D8&list=PLBibh5-ddnvinVaT-deAJGmkYvcj5goUz