Anthropomorphising AI

LLM-based AI bots are like people in some way, but not in the way that matters.

We should not anthropomorphise AI systems, in particular LLM-based chatbots like ChatGPT or Claude, and it’s important to understand why.

Much has been said about how LLMs don’t really understand what they are saying¹ because they are simply incredibly sophisticated programs² predicting the next word in a block of text³.

Some are saying that this means they are bullshitters⁴ in the sense that they don’t care whether they are telling the truth or not as long as they’re persuasive. Therefore, chatbots are deemed incapable of being honest or moral.

In short, the argument goes, people have a sense of morality, care for the truth and can actually understand things. Since LLMs can’t do any of this, it would be a grave mistake to anthropomorphise them by imagining they are like us.

Well, allow me to present a different argument.

What I’m about to argue⁵ is not that LLMs do much more than predict the next word, or that they understand things, or that they are capable of thinking, let alone being conscious. On the contrary, I will argue that they can’t do any of these things, but so can’t humans, from a particular perspective. This makes us eerily similar to LLMs in some ways but not all. And where it actually matters, we are fundamentally different from AI.

While we don’t fully understand how LLMs work⁶, we do understand a lot since we know how to build them. It’s an incredibly complex set of math operations that feels more magical than it is. However, if an iPhone were invented overnight in the 1950s, it would have looked like magic, too, even though we know how to build one and why it works as it does.

The mistake most people make when they try to think about LLMs lies not in their misunderstanding of how LLMs work. It lies in their misunderstanding of how humans function.

Humans have a sense of self that we refer to when we say “I”. We take its existence for granted. We have a powerful feeling that “I” is capable of thinking, understanding, caring about the truth, being moral, and so on. It’s not exactly wrong, but it’s an incomplete view, in the sense that a two-dimensional image of a three-dimensional object is not wrong but incomplete.

Since we take this sense of self for granted, when we talk to an LLM like ChatGPT and it says “I”, we imagine that this must mean that it also pretends to have a sense of self or could have one. We are sceptical because it’s just a mathematical model, right? Of course, an LLM doesn’t have a sense of self beyond its ability to print the words “I think”, but you, the reader, a human being, also don’t have a sense of self beyond your ability to think and feel that you do.

Wait, what? Bear with me.

Try doing absolutely nothing. Sit or stand and decide that nothing should happen unless you choose it. In a few seconds, you’ll discover that the following is happening: your thoughts, feelings, body sensations and whatever else you usually experience keep on happening on autopilot.

You continue to be aware of the changing contents of consciousness regardless of your intention for it to stop. If you pay close attention to the flow of your thoughts, you’ll discover that you have very little control over them. If you keep paying attention, you’ll find you have no control over them whatsoever. The feeling that you do is a combination of thoughts and body sensations.⁷

In other words, sometimes you’re happily lost in what you’re doing and not even aware of being you. You’re just in a beautiful state of flow. Sometimes, you have lots of self-referential thoughts like “I want this, I must do that, I’m such and such person”. Sometimes, these thoughts are absent. But the bottom line is that you’re just aware of what is happening in your being: thoughts, sensations, emotions, sounds, sights, etc. Most people, most of the time, are lost in thought because they believe that what they’re thinking or experiencing is more real than it is.

Photo by Samuel Austin on Unsplash

From this perspective, LLMs are indeed like us in the sense that they produce the next word in a way that looks like they’re saying something meaningful, just like our consciousness experiences a neverending stream of thoughts, feelings and sensations that isn’t random. What we experience is heavily conditioned by what we have experienced before. I’m experiencing my thoughts and not yours because I had my life experience and not yours. How are thoughts produced? Who knows⁸, but it’s safe to say they are not made by matrix multiplication⁹.

However, there is a crucial difference between living beings and computers. We can experience things. We are conscious in the sense that it is like something to be me, and it is like something to be you.¹⁰ There isn’t any reason to believe that it’s like something to be a calculator, an iPhone or an LLM.¹¹

Not only can we experience things as neutral events, but we also experience them as pleasant or unpleasant, which orients our life towards looking for things we like and avoiding things we don’t like. This goes beyond simple pleasures like a good meal to feelings of love, meaning, or connection to others. And so we all live our lives to the best of our ability in the moment to experience things we want to experience and avoid others.¹² This relatively simple architecture of consciousness that’s being aware of its contents, including various likes and dislikes, is behind all the complexity of life.

From this perspective, it’s not that LLMs are not like us—supposedly independently existing entities with agency—but we are like them in a way. We experience a never-ending stream of thoughts and feelings that aren’t random but intelligent responses to our experience.

I’m digging here into the question of what it means to be human, which deserves not even a book but a library. We don’t have a definite answer, and it will never exist in a way that can be articulated in words. But I think it’s safe to say that what makes us human is not our thoughts, not our bodies and not our sense of self.¹³ Many wisdom traditions and philosophers who went deep into this question over millennia arrived at conclusions that don’t look remotely like an LLM.

So we shouldn’t antropomorphise LLMs not because they “don’t understand” or “can’t think” or “don’t care about the truth” or “just predict the next word”. On some deep level, humans don’t either.¹⁴

In fact, part of our experience looks suspiciously like LLM output. I’m not persuaded that when both ChatGPT and I complete the phrase “The capital of France is…” it’s anything but sophisticated pattern matching.

Instead, we shouldn’t anthropomorphise LLMs because once we look at what makes us human, whatever the answer is, it doesn’t remotely look like an AI system and has more to do with the experience of being conscious than anything we’re conscious of, including our intellect or ability to communicate.

P.S.: Having said all of this, I can see how the current trajectory of AI development can get us to something massively smarter than us, just like we managed to invent machines and weapons infinitely stronger than us. Hence, I think about AI risk in the same terms as nuclear risk: the tech is very powerful. I hope the chain reaction doesn’t go uncontrolled.

1. A correct insult for LLMs here is "a stochastic parrot.”

2. Here’s a brilliant and only moderately technical overview of what’s under the hood.

3. Technically, tokens, which are parts of a word. Plus, the same approach applies not only to text but to other media: sound, images, etc.

4. Here, bullshit is a philosophical term meaning a discourse that is not even concerned with truth. Here’s a good paper on ChatGPT and bullshit and here is a Wikipedia page about the original work on bullshit by G. Frankfurt. His essay is fun to read!

5. If this argument makes you uneasy, I will understand. But if you think I’m wrong, I challenge you to defend your position with something more solid than a belief.

6. For example, we don’t have a good idea of their emergent capabilities. Bigger models surprise us in unexpected ways.

7. Psychology, neuroscience, and contemplative practices have all long ago realised and described the fluid and illusory nature of a sense of self. This is not new, but it doesn’t stop us from feeling like this sense of self is somehow “really real.” It’s not. It’s like a rainbow: it exists in a way that I can see, but not in a way that I can go and touch it or even find where it starts.

8. However this works in the mind, we don’t decide to decide to decide to have a certain thought. Thoughts appear in the same way waves occur in the ocean without anyone “creating” them.

9. LLMs require a tremendous amount of matrix multiplication to do even the most basic things, which is why they require so much power.

10. Thomas Nagel famously argued that an “organism has conscious mental states if and only if there is something that it is like to be that organism—something it is like for the organism."

11. There’s a theory that consciousness arises as a side effect of computational complexity, but at this point, it remains a theory. I find it implausible.

12. The entire Buddhist project of freeing oneself from suffering is based on the idea of seeing through this and realising that the ultimate freedom is achieved by not being bound by our desires and aversions, realising somewhere in the process that there isn’t anyone to be free either, there’s just what is, and it changes all the time. An ego can never find lasting fulfilment; it can only be less miserable for a while.

13. Which is why I’m amused when people sincerely talk about “uploading their consciousness to the cloud”, implying that their thoughts and memories will go there. That’s not what they are. It’s like trying to steal a movie by taking a TV it’s showing on.

14. But on some other level, they do. I’m not making a nihilistic argument here; please don’t make this mistake. However, expanding on that would take a far more extended essay I don’t feel qualified to write.