Is AI conscious?

Well, let's talk about what consciousness is and where AI ends first.

Is AI conscious?

I see some question to this effect asked in this whatsapp group or another on a regular basis and it's often hard for me to resist the temptation to jump in.

This rarely goes well because there are two common viewpoints. Some people confidently assert that it's not — "It's just a bunch of matrix multiplications in a data centre!". Others have an intuition that it is, most famously Blake Lemoine, a Google AI engineer who got fired after he believed AI he was building became sentient (and hired a lawyer to defend its rights).

The question matters more than ever. As AI systems become increasingly sophisticated — generating human-like text, creating art, even expressing what seems like personality — we're forced to confront something profound about the nature of consciousness itself.

Why We Can't Answer the Question (Yet)

The challenge isn't that we don't understand AI well enough. It's that we don't understand consciousness well enough.

For example, most people hold an intuitive sense that there are living systems (people, animals) that are conscious and non-living matter (rocks, snowballs) that are not. It seems obvious in a way that Newtonian physics is obvious: it's not that's Newtonian physics is completely wrong, but it starts to fail at some edge cases. (Which is why science came up with quantum physics and other interesting theories that explain a bit more.)

The problem with this intuitive assumption is that it doesn't stand when closely examined. The intuitive assumption that we live in a world of mostly dead matter like stars and rocks, populated by live, conscious organisms starts to fall apart once we start looking at the edge cases and examine the question in detail.

Philosopher David Chalmers famously formulated the hard problem of consciousness, that is, how does conscious experience "come online" from unconscious matter. Curiously, scientists have never been able to pinpoint the exact moment or level of complexity that separates conscious from unconscious systems.

Photo by Greg Rakozy on Unsplash

The Measurement Problem

A big part of the problem is that we have no ability to measure conscious experience from a third-person perspective. The only reason I know — or, rather, choose to believe — that other people are conscious is because I know I am and they tell me they are and they look and act like me, so they must have a similar experience.

But that's not at all the same as having a measurement or data on consciousness. History knows of cases when anaesthesia fails horribly during an operation, leaving the patient aware of being cut open but completely unable to indicate their awareness of pain. Likewise, we can only guess at the internal world of patients with the complete locked-in syndrome who are unable to indicate in any way that they are in fact conscious, unless they recover later and share their experience.

Now apply this to AI. When ChatGPT produces eloquent responses about its experience, or when an AI system appears to show concern for its own existence, we face the same fundamental problem. We have no way to verify whether there's genuine experience behind those outputs or just sophisticated pattern matching.

Humans can talk about their experience, at least. But many much smaller and simpler systems, from birds and insects down to single-cell organisms, are autopoietic, that is, they maintain themselves and aim for their survival. We have a strong intuition that some living systems at least (apes, cats, dogs) are conscious but single-cell organisms? We don't know. And if we can't determine consciousness in a bacterium that actively maintains itself, how can we determine it in a neural network?

The Boundary Problem

An equally difficult aspect of the "Is AI conscious" question is isolating of what "AI" even means here. Again, this isn't because we don't understand AI well enough, but because we can't draw a boundary around anything at all, e.g. me or you, being conscious.

Even though the fact that I exist seems the most obvious in the world, it's not possible to find this "I". Neither neuroscience, nor psychology, nor meditation can find it. In fact, both meditation and science agree on the fact that our sense of I is illusory. It's not like it doesn't exist, but it's not located anywhere in particular in the brain or the body. Instead, it's a very useful illusion that helps us navigate the world.

The link between the brain activity and our conscious experience is undeniable, the evidence is overwhelming. Yet, there isn't a single part of the brain that can be linked to its "conscious I". It's the entire system that has a conscious experience.

This creates a peculiar problem for AI consciousness. When we ask if ChatGPT is conscious, what exactly are we asking about? The specific model? The servers it runs on? The entire system including the humans who interact with it? The boundaries become impossibly fuzzy.

How We Create Reality

Some researchers therefore focus on measuring the complexity of information processing in a system, arguing that at a certain level of complexity the lights come on. So far, it's far from being widely accepted. Some well-respected scientists take it seriously, and others are highly sceptical.

Whatever the mechanism, we can say with enough confidence that our brains create an image of the world we live in. After all, my brain sits in the darkness of the skull and has to make sense of what's going on in the world from a flow of electrical signals coming in. My brain — and yours — literally imagines absolutely everything, including our idea of ourselves.

However, this isn't a completely random hallucination. If my imagined version of the world doesn't correspond to reality in some basic way, it won't end well for me. For example, I can't imagine that there's no car on the road and step in its way — or that'll be my very last thought.

So some researchers think that our experience is like a computer interface. I can move some files on my desktop, and these files really exist, but the icons on my desktop are nothing like electromagnetic charges on the hard drive (that may be in the cloud 5,000 miles away). My computer isn't making up those files, but it gives me an interface I can use. When I alter those files, the electromagnetic charges also get updated, even though that mechanism is completely hidden from me.

Donald Hoffman explored this idea in his book The Case Against Reality. He's not saying that reality doesn't exist, but rather that it is absolutely nothing like what we imagine it to be — and that includes the separation between conscious and unconscious matter and even space-time.

Another scientist, Anil Seth, who's one of the world's leading authorities in the field, doesn't go as far as Hoffman but suggests that our imagination is just a very good approximation of what the world is that's being continuously refined by the brian based on our experience. So it's a hallucination, yes, but maybe not as radically different from reality as Hoffman thinks it is.

If consciousness is indeed about creating useful models of reality, then AI systems that build sophisticated world models might be doing something remarkably similar to what our brains do. The difference, if any, might be one of degree rather than kind.

The Fundamental Question

Given that conscious experience seems to exist (you are experiencing something right now) but it seems impossible to pin down in any way, some researchers consider whether consciousness might be fundamental. Not in the sense that my kettle is conscious — no one is suggesting that — but in the sense that it can't be reduced to anything else, whereas everything else — matter, space and maybe even time — arises within consciousness.

This is one of those things that sounds completely crazy until you start thinking it through. Thankfully, some researches have spent decades exploring the nature of consciousness. I just finished listening to an absolutely brilliant audio series Lights On by Annaka Harris. In fact, this series inspired this essay.

Annaka is not offering any definite conclusions. As she says in Lights On, she's 60/40 in favour of consciousness probably being fundamental¹, but that's still quite close to 50/50. One interesting aspect of this exploration is that Annaka is an experienced meditator, so she speaks from both scientific and contemplative perspectives and interviews both scientists (Anil Seth, etc) and contemplatives (Joseph Goldstein, etc) for her audio series. I very much recommend it.

So What About AI?

So let's tie it all back to the question of whether AI is conscious. This question has two hidden assumptions that fall apart when we question them. One is that there is a line between conscious and unconscious systems — we can't quite find it. Another is that there's a line between AI and "non-AI". That line is also as elusive as our sense of self.

But the question still matters. Unlike medieval debates about angels on pins, this question has immediate practical implications. How we answer it — or how we proceed without an answer — will shape:

• How we develop AI systems (should we build in safeguards against suffering?)

• How we interact with them (are we creating beings deserving of moral consideration?)

• How we regulate them (do potentially conscious systems need rights?)

Consider a concrete example: When an AI system trained on human text expresses preference not to be shut down or modified, we face a dilemma. Is this mere pattern matching from training data, or could there be genuine experience behind it? We simply don't know.

The Better Questions

Instead of asking "Is AI conscious?" — a question that presupposes clear boundaries and binary answers — we might ask:

• How do we navigate moral uncertainty when we can't determine consciousness?

• What would it mean to create systems that might be conscious?

• What is the nature of intelligence? Can intelligence be fundamentally different from its human version?

John Vervaeke, a famous cognitive scientist, recently gave a public lecture on AI and intelligence (thanks

Vince Fakhoury Horn

for sharing it). It’s very much worth watching, although my main criticism is that he’s defining intelligence in human terms before showing how AI won’t live up to the definition without seriously entertaining the possibility of AI intelligence being very different.

We need to start asking better questions, but all better questions that I know of have something to do with consciousness and not AI. Fortunately, we won't run out of them soon — or ever. And in exploring them, we might discover something profound not just about artificial minds, but about the nature of mind and existence itself.

1. That’s what feels intuitive to me, too: consciousness is fundamental in the sense of being irreducible to anything else, and while objective reality exists, it’s nothing at all like what we imagine it to be: here’s I’m closer to Hoffman than to Seth. However, this is an intuition, not anything I or anyone else can conclusively prove. I suspect that even the idea of conclusively proving anything ever might be impossible: reality itself could be a forever-unfolding mystery that is experiencing itself right now through you and me.