Découvrez des podcasts liées à Conspicuous Cognition Podcast. Explorez des podcasts avec des thèmes, sujets, et formats similaires. Ces similarités sont calculées grâce à des données tangibles, pas d'extrapolations !
* 00:00 Understanding Misinformation: Definitions and Prevalence
* 04:22 The Complexity of Media Bias and Misinformation
* 14:40 Human Gullibility: Misconceptions and Realities
* 27:28 Selective Exposure and Demand for Misinformation
* 29:49 Political Advertising: Efficacy and Misconceptions
* 35:13 Social Media’s Role in Political Discourse
* 40:50 Evaluating the Impact of Social Media on Society
* 42:44 The Impact of Political Content on Social Media
* 46:57 The Changing Landscape of Political Voices
* 51:41 Generative AI and Its Implications for Misinformation
* 01:03:46 The Liar’s Dividend and Trust in Media
* 01:14:11 Personalization and the Role of Generative AI
Transcript
* Please note that this transcript was edited by AI and may contain mistakes.
Dan Williams: Okay, welcome back. I’m Dan Williams. I’m back with Henry Shevlin. And today we’re going to be talking about one of the most controversial, consequential topics in popular discourse, in academic research, and in politics, which is misinformation. So we’re going to be talking about how widespread is misinformation? Are we living through, as some people claim, a misinformation age, a post-truth era, an epistemic crisis?
How impactful is misinformation and more broadly domestic and foreign influence campaigns? What’s the role of social media platforms like TikTok, YouTube, like Facebook, like X when it comes to the information environment? Is social media a kind of technological wrecking ball which has smashed into democratic societies and created all sorts of havoc? And also what’s the impact of generative AI when it comes to the information environment?
Both when it comes to systems like ChatGPT, but also when it comes to deepfakes, use of generative AI to create hyper-realistic audio, video, and images. Fortunately, we’re joined by Sacha Altay, brilliant heterodox researcher in the misinformation space, who pushes back against what he perceives to be simplistic and alarmist takes concerning misinformation.
So we’re going to be picking Sacha’s brain and just more generally having a chat about misinformation, social media, and the information environment. So Sacha, maybe just to kick things off, in your estimation, if we’re keeping our focus on Western democracies, how prevalent is misinformation?
Sacha Altay: Hi guys, my pleasure to be here. So it’s a very difficult question because we need to define what is misinformation. So we’ll first stick to the empirical literature on misinformation and look at the scientific estimates of misinformation. For that, there are basically two ways or three ways to define misinformation. One of them is to look at fact-checked false news.
So false news that have been fact-checked by fact-checkers as being false or misleading. And by this account, misinformation is quite small on social media, like Facebook or Twitter. It’s in between 1 and 5% of all the content or all the news that people come across. So according to this definition, it’s quite small. There is some variability across country. For instance, it seems to be higher in country like, I don’t know, the US or France than the UK or Germany.
There is another definition which is a bit more expansive because the problem with fact-checked false news is that you rest entirely on the work of fact-checkers and of course fact-checkers cannot fact-check everything and not all misinformation is news. So you see the problems. So another way is to just look at the sources of information and you classify them based on how good they are and how basically how much they share reliable information, how much they have good journalistic practice, et cetera. And the advantage of this technique is that you can have a much broader range because you can have, I don’t know, 3,000 sources of information that share information. And usually it broadly like most of the information that people see. And according to the definitions, misinformation is also quite small. So the definition is just for misleading information that comes from the sources that are judged as unreliable. And by this definition, misinformation is also quite small. Again, it’s like about like one to 5% of all the news that people encounter.
But then of course, the problem is not all the information that people encounter comes in this form. And for instance, some of it can come in terms of like images or all the sorts of things. And so this broadens the definition of misinformation. So some people think that when you broaden this definition, you have much more misinformation. My reading is that when you broaden this definition, you actually include so much more information that you increase the denominator. So of course, there’s going to be more misinformation, but because the denominator is larger, the proportion is going to be pretty much the same. But that’s an empirical question. So let’s say to sum up that it’s smaller than people think, according to the scientific estimates.
Henry Shevlin: If I can just come in here, a point that Dan you’ve emphasized in our conversations to me, and I think Scott Alexander has also emphasized in a great blog post called The Media Very Rarely Lies, is that a lot of what people think of as misinformation is just true information selectively expressed or couched in a way that naturally leads people to maybe form false beliefs but doesn’t involve presentation of falsehoods. Does that sort of feature in any of these sort of more expansive definitions of misinformation? Is it possible to create definitions that can capture this kind of deceptive, intentionally deceptive but not strictly false content?
Sacha Altay: I’d say that when you look at the definitions based on the sources, if a source is systematically biased and systematically misrepresent evidence and stuff, they are going to be classified as misinformation. I think the problem and the more subtle point is that these sources are not very important because people don’t trust them very much. But the bigger problem is when much more trusted sources who have a much larger reach, like I don’t know the BBC or the New York Times, they are accurate like most of the time, but sometimes and on systematic issues like I don’t know, they can be wrong. And that’s the bigger issue because they are right most of the time. So they have a big reach, they have big trust, but they are wrong sometimes. And that’s the problem.
Dan Williams: But I think just to focus on that observation of Henry’s, you might say, well, they’re accurate most of the time, but nevertheless, you can have a media outlet which is strictly speaking accurate most of the time with every single news story that it reports on. But because of the ways in which it selects, omits, frames, packages, contextualizes information, nevertheless end up misinforming audiences, even if every single story that they’re reporting on is on its merits, sort of factual and evidence-based.
I mean, I think the way that I understand what’s happening in this broader debate about the prevalence of misinformation is round about 2016 when we had Brexit in the United Kingdom and then the first election of Donald Trump, there was this massive panic about misinformation because many people thought maybe that’s what’s driving a lot of this support for what gets called like right-wing authoritarian populist politics. And around that time when people were thinking of the term misinformation, they were kind of thinking of fake news in the sort of literal sense of that term. So false outright fabricated information presented in the format of news. And as you pointed out, when researchers then looked at the prevalence of that kind of content, which you don’t really find when it comes to establishment news media for the most part, like there are always gonna be exceptions, that stuff is pretty rare.
And then one of the responses to that is to say, okay, if you’re only looking at like outright fake news, then you’re missing all of these other ways in which communication can be misleading by being selective, by omitting relevant context through framing, through kind of subtle ideological biases.
And then my view on that is, well, once you’ve expanded the term to that extent, and you’ve got this really kind of elastic, amorphous definition, it becomes really kind of analytically useless. Like you’re just bundling together so many different things. And that kind of content is also really pervasive in my view, within many of our establishment institutions, including within the social sciences. But Sacha, it sounds like you don’t necessarily want to endorse that last point. You seem to be thinking, even if you do have this kind of very broad definition of misinformation, we can still say that it’s a pretty fringe or pretty rare feature of the information environment. Is that fair? Am I understanding you right? Or is there something different going on?
Sacha Altay: I think I would agree with you that if the simple fact of framing information or having an opinion, like any scientist, even in the hard sciences, they have some theories that they prefer, they are more familiar with certain frameworks, and so they are going to be biased anyways. Scientists are humans, they are biased, but calling physics or the theory of relativity or whatever misinformation because it omits certain facts that it cannot accommodate or whatever, I think it’s far-fetched. I think it goes too far. So yeah, I would agree that if you use this broad definition of misinformation, then it’s very widespread. But then, yeah, even theories in physics would be misinformation because they cannot be completely objective.
I think science works not because scientific individuals are perfect, etc., or even because one theory is perfect, but because as a whole and as an exercise of arguing, etc., we get better and a little bit closer to the truth. But still, we are not getting at the truth and we cannot avoid the mistakes that you’re pointing.
Henry Shevlin: If I just want to push back a tiny bit, it seems to me, so obviously there’s this point here that, you know, all theory is value laden, the kind of physics point that I think is maybe true, but not very interesting. But I think there is maybe something in the middle here that is what I worry about, which is cases where there might be really quite, quite deliberate pushing of an agenda, a realization by a media provider that they are generating maybe inaccurate views, but they’re doing so just through reporting factual things.
So one example, Dan, that you’ve given before is that most of the kind of what we think of as misleading anti-vax discussion just reports on true factually accurate but rare vaccine deaths, but just reports on them in a very regular fashion. In the same way, you might think that selective reporting of certain kinds of violent incidents, whether it’s terrorism, police shootings, leads systematically to overestimation of the incidence of this kind of phenomena by the public or increased worries about its prevalence in a way that I think is perhaps worrying and politically objectionable, right? I think we might say, hang on, it is bad that we give so much press coverage to event type X rather than event type Y. And we know that this leads the public to overestimate the prevalence of event type X compared to event type Y. So I think there’s something in between the sort of, well, even physics is biased and the view of misinformation as, you know, strictly speaking lies. This kind of third category. I don’t know if that, I defer to you both as misinformation experts, but it seems that that is a worrying category.
Sacha Altay: I think you’re totally correct. And that’s what the field of misinformation has been proposing, like just for instance, classifying headlines based not on whether they are true or false, but whether they will create misperceptions after you have read them. And so researchers are saying, for instance, that we should classify as misinformation headlines such as, “a doctor died a week after getting vaccinated and we are investigating the cause.” And I think I disagree with this. I disagree with this thing that we should classify this.
What you were suggesting, Henry, was a bit different, is that it needs to be systematic. If you systematically misrepresent vaccine side effects, then it becomes problematic. But reporting on vaccine side effects and their possible negative effects is normal. And I think it’s healthy that news outlets are able to talk about and cover negative effects of vaccines, even if after reading the headlines, you have more negative opinions about vaccines, which is not supported by science, et cetera—they should be able to do that and they should do that. But if it’s systematic, as you say, I think it becomes more problematic. But I do think that when the bias is very strong in some of the definitions of misinformation based on the source, they would be classified as misinformation sources like Breitbart, et cetera. They are systematically extremely biased towards, I don’t know, these kind of things. And so they would be classified as misinformation.
Dan Williams: I think sort of one of the worries that I have though is who decides what constitutes systematic bias and bias about what? I think there’s a real kind of epistemological naivety that I often encounter with misinformation researchers where it’s like, you’re reporting accurate but unrepresentative events when it comes to vaccines. So we can call that misinformation. And then it’s like, well, as Henry mentioned, well, what about police killings of unarmed black citizens in the US. There’s a vast amount of media coverage of those sorts of events. Someone might argue that they are, statistically speaking, rare and unrepresentative, and that large segments of the public dramatically overestimate how pervasive those sorts of occurrences are.
And I think you go through many, many examples like that. And for me, the lesson to draw from that is not that, therefore, there are no differences in quality when it comes to the different media outlets in the information environment, like of course there are, but I also think like there’s such a thing as politics and there’s such a thing as science, where you’ve got scientists who attempt to acquire a kind of objective intellectual authority on certain things, and we should be very careful not to kind of blur the distinction between those two things.
I think when we’re talking about media bias in this really expansive way, where we’re not saying, okay, you’re just making s**t up, but we’re saying you’re being selective in terms of which aspects of reality you’re choosing. For me, that’s a really important debate, but it’s a debate that happens within the context of politics and democratic debate and deliberation and argument. And I think sometimes I encounter misinformation researchers who treat that as if it’s just, it’s a simple sort of technocratic scientific question. Like we can quantify the degree to which the New York Times is biased or we can objectively evaluate the degree to which different kinds of outlets approximate the objective truth when it comes to their systematic coverage. And I get a little bit kind of squirmy when we get to that point, because I think that there’s just collapsing the distinction between kind of politics with all of its messiness and complexity and science, which I think should aspire to a kind of objectivity, which gets lost when we start making these really sort of expansive judgments.
I think we’ll probably circle back on this a few times as we go through this debate. But Sacha, you’re also somebody with very interesting views about not just this question of the kind of prevalence of misinformation, but also about human belief formation and the extent to which, in your view, lots of people, both in popular discourse, but also in academia, kind of overestimate the gullibility of human beings when it comes to exposure to false or misleading content. So do you want to say a little bit about your view concerning human gullibility?
Sacha Altay: Yeah, I just wanted to finish the last point on the fact that, you know, we are criticizing definitions of misinformation, but in media and communication studies for a long time, they have been studying kind of media bias, framing, agenda setting. Like they are very old theories of media, how they can misinform in subtle ways and indirect ways the public. And all of that has kind of been ignored by misinformation research. But now I feel like today misinformation research is catching up and be like, actually, we should go back to these theories. And so I think it’s good. But I just wanted to point that out.
And regarding gullibility, yes, I think it’s quite popular, the idea that people and like large complex events like the Brexit, Donald Trump or whatever are caused by people being irrational or gullible in particular. By gullible, I think what people often mean is that they are too quick to accept communicated information, like social information that they see out there in the world, in the news, communicated by others. And I think that the scientific literature shows something very different.
For instance, there is a whole literature on social learning, so how people learn either from their own experiences, their own beliefs, or what they see compared to like communicated information, social information, advice. And the consensus in this literature is that people underuse social information. They do not overuse it, they underuse it. And they would be better off doing many kinds of tasks if they were listening and weighing other people’s opinion and beliefs more than their own. So, I mean, it makes sense. Basically, we trust ourselves, we trust our intuitions, our experiences much more than that of others.
And so that’s kind of a consensus. There are many kinds of tasks, like you ask people, oh, what’s the distance between Paris and London? It’s like, 300 kilometers. Another participant, say 400. And you’re not going to take into account other people’s advice as much as your own intuition, even though you have no reason to be an expert on this kind of geographical distances. But you still trust yourself more.
And there are also many like theories and mechanisms that have been shown in political communication and media studies that I think suggest that people put a lot of weight on their own priors and their own attitudes when they evaluate and choose what to consume, which greatly reduces any kind of media effects or any kind of outside information. Like people are not randomly exposed to Fox News. They turn on the TV and they select Fox News. And then people selectively accept or reject the information they like the most. And so I think when you take all that into account, like selective exposure, selective acceptance, and egocentric discounting, it complicates a little bit the claim that humans are gullible.
Dan Williams: Yeah, so there’s this sort of popular picture of human beings as credulously accepting, you know, whatever content they stumble across on their TikTok feed. Although when I say human beings, it’s always other human beings, right? This is another point that you make with the third-person effect. Nobody really thinks of themselves as being gullible and easily influenced by false and misleading communication. But when it comes to other people, there’s this kind of intuition which is that, yeah, people are just being kind of brainwashed en masse by their lies and falsehoods and absurdities uttered by politicians and that they’re encountering in their media environment.
And your point is, no, actually, if you look at the empirical research, it doesn’t really support that at all. If anything, people put too much weight on their own kind of intuitions, their own priors, their own experientially grounded beliefs relative to the information that they’re getting from other people. So rather than thinking of many of our sort of epistemic problems as being downstream of gullibility, we should think of in some ways there being the opposite problem of people being too mistrustful, too kind of skeptical of the content that they’re coming across. Is that a fair summary of your perspective?
Sacha Altay: Couldn’t have said it better.
Henry Shevlin: If I can just raise one question here. Reading your brilliant paper, you emphasized, so this is a paper with the Knight Columbia School. You go through all these different misconceptions about how easily influenced people are by different sources, by sort of different peers, by the media, by the news. But this sort of does prompt the question, you know, where do people’s beliefs actually come from?
And you mentioned people’s priors, people’s intuitions, but presumably people aren’t born with these intuitions, they are formed from somewhere through certain kinds of processes. So I’m just curious if you have any sort of thoughts on where do people’s views come from? Because obviously that would suggest, well, that’s the place you go then if you want to influence people, you intervene on whatever is causing this fixation.
Sacha Altay: I mean, my view on beliefs, and I mean, much of my views come from Dan Sperber and Hugo Mercier, who have these theories on like reasoning and the roles of beliefs. And so basically, to answer your question, I think a lot of people’s beliefs are downstream of their incentives and intuitions they have about the world. For instance, vaccines. Vaccines are like profoundly counterintuitive. Like it’s very difficult intuitively to like vaccines. Like first there’s a needle that goes into your arm, there’s a little bit of blood, you think that there is some kind of like pathogens inside the vaccines, like it’s not something that’s very intuitive. So first I would say most, like not necessarily the beliefs, but the attitudes people have about vaccines largely comes from these very general intuitions that they have about contagion, about infections and about all these things.
And then the beliefs, well, people need beliefs to justify their attitudes. And so if your doctor is like, do you want to get vaccinated and you don’t really want to get vaccinated, you can say you’re scared of needles. But if there are also some widely available cultural justifications like, vaccines cause autism, maybe you’re going to jump on it. Maybe you’re not going to jump on it because maybe you’re smart and you know it’s false, et cetera. But you need justifications. And so I think a lot of people’s beliefs comes from this need to rationalize some justifications that they have. And I think that’s also why on many topics, people don’t have that many beliefs because often people don’t really need to justify many of their attitudes. And there’s a lot of work, for instance, in political science on how surveys kind of create beliefs in people because people have intuitions and kind of like vague opinions about all sorts of stuff. But when you ask them, they have to fix it and they have, and in some sense, it creates the beliefs.
So yeah, I would say beliefs mostly come from prior attitudes that people have and incentives that they have to act in the world.
Henry Shevlin: Okay, but those... just to push a little bit harder there, so the prior beliefs, I think we’re just still kicking the can down the road a little bit. Incentives I get. Incentives seem genuine and explanatory here, but presumably it’s not the case that you can predict people’s vaccine attitudes from the degree of phobia they have towards needles, right? Or at least, even if that is predictive, I don’t know if it is, it seems like there’s more going on there. I don’t want to give people, and I think that’s the danger of giving people too much credit for saying, oh, people’s beliefs perfectly track their own incentives. I can totally agree that incentives play a role, but I’m sure just when we think about our own sort of peer groups, right? I disagree with the political views of a lot of my peers, despite us being in the same socioeconomic class, despite us working in the same industry, despite us having, you know, broadly similar interests, I would have thought. So, I don’t know, I can see incentives carry us some of the way, but yeah, they don’t completely close the mystery here.
Sacha Altay: No, of course, of course. I think it’s, you take the example of vaccines. I think most people who get vaccinated, they just get vaccinated because they trust institutions, they trust their doctors. Maybe they have seen their doctors for 20 years, their doctors tell them to get vaccinated, they do it. So that’s the main explanatory agent here is just they trust some institutions, some experts who tell them to do something and they do it.
You wanna jump in, Dan?
Dan Williams: Yeah, I was just going to say, I think it seems like it’s possible to think, and as I understand Sacha, your view, this is your view. It’s possible to think that we overestimate the degree to which people are kind of influenced by whatever content they happen to stumble across in their media environment or the viewpoints that they happen to encounter in their social network—that we tend to think people are too gullible when it comes to those things.
It’s possible to think that, but also to accept that, of course, we are going to be influenced in complex ways by the information we get from people that we trust, from sources that we trust, from our upbringing, from our social reference networks and so on. So the idea that we’re not gullible and not credulous shouldn’t be sort of conflated with the idea that we somehow are born with our entire worldview from the start in ways that aren’t influenced by the media environment and by the testimony that we encounter. Like clearly we’re massively influenced by what we hear from other people, but sort of my understanding of the perspective that you’re outlining Sacha is that process whereby we build up beliefs about the world—firstly, there are some things that just everyone kind of finds natural, like maybe like there’s something weird about vaccines when you hear about the concept, most people just have a kind of instinctive aversion to it, but also things like, you know, my group is good, the other group is bad, or like certain kinds of maybe xenophobic tendencies that come naturally to people and so on. So there are certain ways of viewing the world and certain things which are intuitive, maybe as a consequence of our evolutionary history, and that interacts then in very kind of complex ways with our experiences, with our social identities, with our personality, with the people that we trust, the institutions that we trust, those we mistrust, and so on and so forth. So you can accept all of that and the role of social learning within that whilst also thinking people tend to exaggerate how gullible, how credulous people are when it comes to sort of incidental exposure to communication. Is that your view, Sacha? Is that a kind of accurate representation of it?
Sacha Altay: Yes, yes, yes it is. I think a lot of the reason, like when we change our mind drastically, it’s either because like we have a lot of reasons to trust the source. Like if the BBC says that the Queen died and the BBC says it, the Guardian says it, we’re going to update our beliefs immediately. And most people, even the people who distrust the BBC are going to update their beliefs directly.
And it’s the same if like, I don’t know, my wife tells me that there is no more milk in the fridge and I have to buy some. I’m going to update my beliefs about the milk in the fridge and buy some, you know, in some ways, of course we update our beliefs based on the information that’s provided to us. It’s just that we do so I think in ways that is broadly rational in the sense not that it’s perfect, but that it serves our everyday actions and our incentives, like what we want to do in the world, like very well. So I think that’s also the way in which I mean it is that when we update it and when we do it, we do it quite well, not to discover the truth, but at least to get along in the world.
Dan Williams: And could you maybe say a little bit more about this point concerning selective exposure? So the fact that when people are engaging with media, with the viewpoints of pundits and politicians and so on, a lot of that is, quote unquote, demand driven in the sense that people have strong attitudes, they’ve got strong political, cultural allegiances, they identify with a particular in-group, they want to demonize like those people over there or that kind of institution, et cetera. And it’s these sort of pre-existing attitudes, interests, allegiances, which often build up in complex ways over a long period of time, which then causes people to kind of seek out information and often misinformation, which is consistent with their attitudes and their interests, rather than the picture I think sometimes people have, which is—I think the way Joe Uscinski puts it is, you know, they’re walking along and they slip on a banana peel, you know, they encounter some conspiratorial content on social media and now they believe in QAnon or like Holocaust denial. That’s just not the way that it works. Could you say a little bit more about that concerning like selective exposure and the demand side of misinformation?
Sacha Altay: Yeah, we know for instance that misinformation on social media like Facebook or Twitter, which have been the most studied in particular Twitter, you have a very small percentage of individuals who account for most of the misinformation that is consumed and shared on these platforms. And it’s like very small. It’s like 1% or less than 1% that account for most of the misinformation that is consumed and shared on these platforms.
And these people, they are misinformation sharers and consumers, not because they have like special access to misinformation because they have a lot of money or whatever, but simply because they have some traits that make them more likely to seek out such content, such as having low trust in institutions, being politically polarized. And because of these traits, because they don’t trust institutions, they are looking for counter-narratives to like the mainstream narratives they find on mainstream media. Because the thing is that these people who consume and share most of the misinformation on social media, and give us the impression that there is a lot and that many people believe it—these people are also exposed to mainstream narratives. It’s just that they decide to reject the mainstream narratives and instead of trusting what the TV tells them, they go on some Telegram channels, they go on some weird websites to learn about the world and do their own research.
And this is, I think, some of the strongest evidence, at least in the case of misinformation, that the problem is not in the offer of misinformation because it’s actually quite easy, quite free, quite accessible. It’s super easy to find misinformation online, but most people consume very little of it. But you have a small group of people who are very active and very vocal who consume most of it. And they have low trust in institution and are highly polarized. And I think it matters a lot for how we want to tackle the problem of misinformation. The problem is not that you have a majority of the population that’s kind of gullible and so we should avoid them being exposed to misinformation, rather you have some people who have some very strong motivations to do some specific stuff. And I think we should address these motivations. And because addressing the offer is impossible. And I’m not against like content moderation and stuff. I think we should try to be in an information environment where the quality of the information is the highest possible, et cetera. But if you have motivations to look and to pay or to consume some content, then the offer will be met, like people will create such content.
Dan Williams: Could we maybe just before we move on to these issues about kind of social media and AI, because I really want to get to those, there’s another point connected to this issue about gullibility where I think there’s this massive kind of gap between common sense, conventional wisdom, and what the empirical research shows, which you’ve written a lot about, which is like the impact of things like political influence campaigns and commercial advertising and so on. So you go into that in your paper on generative AI and why you think there’s been a lot of unfounded alarmism about that, which we’re going to get to shortly. But even separate from the issue concerning AI, could you say something about what the evidence that we have actually shows when it comes to the impact of political and sort of economic advertising campaigns?
Sacha Altay: So political scientists have been studying that for a while because in the US there is so much money that is being spent on political advertising, especially in presidential elections. And so the best studies, they come from political science. And to give you an example, some of them have up to 2 million participants that are being exposed to hundreds or thousands of ads for long periods of time, like months.
And so these are the kinds of study that are being done in this field, like very large sample, long periods of time, et cetera. And the consensus is that political advertising in presidential elections in the US has very, very small effects. The effects are not zero because of course, with such big sample size, long periods of time, et cetera, you do find significant effects, but the effects are very, very small, like point of percentage. And so that’s the consensus in political science in the US.
So it’s a bit specific because the US you have like Democrats and Republicans and you socialize these identities and these identities are very hard to change. Like if you’re a Democrat, it’s very hard for you to change and vote Republican. And of course, in the US, you often have only two candidates that are very prominent and people hear about them all the time. So it’s difficult to move the needle. But in like other elections in other country, multiparty, you have more room for political advertisement to have an effect. But even in these cases, even when it’s like lower stakes campaigns with less known candidates, the effects are still quite small. Like, I don’t know why we have this idea that advertisement works very well, it influences people, but at least when it comes to political voting, it’s just very hard to influence people’s vote. And it’s the same for like marketing, like online ads, like on social media—are very ineffective, the thing is that they are very cheap as well. So I don’t want to say that they are useless because they’re actually extremely cheap. So that’s why these companies do them a lot, but they’re also extremely ineffective. And so that’s the consensus in political science.
Henry Shevlin: So I had a question about this in relation to your paper again. It really paints quite a dismal view of the power of advertising in general. And yet this is like a vast global industry. Is it all just founded on sand? Is it all just smoke and mirrors? Are people basically wasting hundreds of billions of dollars a year on advertising that doesn’t, largely doesn’t work?
Sacha Altay: That’s the opinion of many people. Yeah. Many people think that at least it’s overblown. I don’t want to say that it’s completely useless, et cetera. Like, of course, if you want to buy like a washing machine, they all look the same. And if they are all about the same price, if you have more information about one and the information is good and the reviews are good, et cetera, you’re probably going to buy it more. But it’s just you already want to buy the washing machine and you have a price range and you have already like so at the margin, advertisement can work and has an effect, it’s just that the effect, like they calculate basically the elasticity. So how much more when you spend on advertisement, how much more will you sell basically? And the elasticity is like super small. It’s like, I forgot exactly, but it’s like very small.
But yeah, some people have written books about how the whole internet and know, products on the internet, like social media, et cetera, are free because we are the product and they sell us advertisement and stuff. And all of that is a bubble. Some people think that it’s completely a bubble. I don’t think it’s completely a bubble, but clearly I think, yeah, it’s overvalued. I think ads are a little bit overvalued. And I don’t think AI is gonna change that much.
Dan Williams: Okay, so just to sort of summarize what we’ve got to so far. So on this question of how prevalent misinformation is, if you’re focusing on fake news, it doesn’t seem to be anywhere near as widespread as many people think it is. Once you start stretching and expanding that definition to encompass more and more things, yes, misinformation so defined is much more widespread and plausibly is much more impactful, but it becomes so kind of amorphous that it’s difficult to apply scientifically.
Then the second thing we talked about was this issue concerning gullibility, where in your view, Sacha, and I agree with you, even though obviously people are influenced by social learning and there is evidence that, you know, persuasion can work, it can influence what people believe, people also tend to dramatically overestimate how gullible people are.
Let’s now turn to technology and where AI is relevant. And let’s start with social media, kind of very broadly construed. Henry, actually, why don’t I bring you in here? Because I think in a few of our previous conversations, you said something like the following, and you can tell me whether I’m remembering correctly. You said, we can contrast two kinds of cases, like video games and social media. In both cases, there was this big societal panic. Video games are going to make people really violent. They’re going to play Call of Duty, and then they’re going to go out and start shooting people in their community.
And your view is, the evidence there is actually incredibly weak and that there’s very little to support that kind of panic. Whereas when it comes to social media, there was a lot of panic, maybe not initially, actually, I think there was a lot of optimism about social media initially. But these days, there’s a lot of kind of concern about social media and how it’s, you know, destroyed democracy and human civilization itself. It’s this awful thing, having all of this sort of awful set of political consequences. And am I right, Henry, in thinking you’re actually quite sympathetic to that view about social media, even though you’re not sympathetic to the violent video games story.
Henry Shevlin: Yeah, yeah, no, great. I’m glad you bring up this example. Two things. One is I think my main point with that example is about sort of the time course of these worries that with violent video games, we had this massive initial panic that sort of died down as the evidence sort of basically didn’t arrive. As we saw that there wasn’t that as much concern as initially there was we thought there was reason to think there was. Whereas in the case of social media, there really wasn’t that much concern at first. It was seen as, if anything, a positive technology and concern has just sort of grown over time. And that sort of point about the time course of sort of the moral panic is sort of separate from the degree to which these are robust.
That said, I do, I am more sympathetic to the idea that social media presents an array of worries. So I’m probably more sympathetic than both of you to sort of Jonathan Haidt’s worries about the impact of social media and mobile phones on teenage mental health, which is a separate point from misinformation. I also worry about the role of social media and things like political polarization. Again, at least a little bit distinguishable from misinformation. But yeah, I guess I’m a little, at least a little bit worried about the role of social media and misinformation as well.
Dan Williams: Okay, I’ve got sort of views that are difficult to summarize about this. Let’s stay away from the teen mental health, because I think that opens up a whole can of worms, et cetera. Let’s focus on kind of the political impacts of social media broadly construed. Sasha, my understanding of your view is you basically think that the panic over social media and its political impacts is unfounded and it’s not well supported by evidence. Is that fair? Care to elaborate?
Sacha Altay: Yeah. So I’m just going to start by mentioning, I think, the scientific literature and what I think is the best evidence that social media have weaker effects than people think. So there have been many Facebook deactivation studies. So basically, you pay some participants to stop using Facebook for a few weeks. And in the control group, the participants are the same, but they are paid either to stop it for one day or to do something else.
And in general, what these studies find is that when you stop using Facebook for a few weeks, you become slightly less informed about the news and current events, suggesting that using Facebook regularly helps you slightly know about the world and what’s going on in the news. But it also makes you slightly more sad. So you’re slightly less happy when you use social media. So participants who deactivate social media, especially Facebook for a few weeks, are slightly happier. It’s not exactly clear why. It could also be because they are less exposed to news and news is sad and makes people less happy, etc. So it could be that. And there are also many other studies on Instagram.
And basically what all these studies suggest is that the effect of social media on stuff like affective polarization, political attitudes, voting behaviors, is either extremely small or no. And so the effects are very small. But now that I’ve mentioned this literature, I want to mention that there are many critics of this literature and of these experimental designs. For instance, even the longest RCTs are like two months. And of course, two months is super small at the scale of social media. They have been here for years. And you could imagine that it takes a few years for the effects of social media to kick in.
You can also imagine that, of course, participants stop using social media for a few months, but the world continues using social media. People around them continue using social media. So you kind of have these network effects that are possible. And of course, the effects of social media are not individual, they are collective. And so these RCTs are kind of missing the point. They cannot capture the collective and more systemic effects that social media could have. So that’s another critique. And there are many other critiques.
But I still think that what these RCTs show is that social media probably has effects. And there are studies like in collaboration with Meta showing that if you change Facebook or Instagram with like a chronological feed, that is instead of showing users the most engaging content, you show them the most recent content. When you do that, they spend much less time on the platform. Like the time they spend on the platform is diminished by one third.
And it has a lot of effects on in-platform behaviors, but very few effects on out-platform behaviors, on attitudes, on et cetera. So we should take these studies with a grain of salt, but I still think they show us that the effects are probably not as big as at least the most alarmist texts suggest.
Dan Williams: Hmm. I think maybe another critique that some people have raised is that these studies, especially that set of Facebook, Instagram studies that you mentioned, were conducted after there had been a lot of adjustments to the platforms and the algorithms in light of concern about things like misinformation and their effect on polarization and so on.
So that just goes to say, as you say, many people have generated lots of different criticisms of what we can really infer from these studies. I mean, my own view is they tell us something, which is that the most simplistic, alarmist stories about social media don’t seem to be supported by the current state of really kind high-quality empirical research. I don’t think they provide very strong evidence that should cause someone who goes into this with a really strong prior that social media is having all of these catastrophic consequences to update that much. And that then suggests that like how you view this topic is going to be shaped by a lot more than just the empirical research itself. So in your case, I assume that you’ve got these general priors about how media doesn’t have like huge effects on people’s attitudes and behaviors and these things are shaped by all sorts of complex factors other than media. And am I right in thinking that’s doing a lot of the work when it comes to your skeptical assessment over and above these studies themselves?
Sacha Altay: Yes, but I would say the strongest argument maybe in favor of my position is descriptive data on what people do on social media and how often they encounter political content. Because to be politically polarized, you need to be exposed to political content. And there are more and more descriptive studies, some of them on the whole US population in the US, showing that it’s less than 3% of all the things that people see on social media.
So less than 3% of all people see on Facebook is either political or civic content. And there are also super nice recent studies that are using a novel methodology, which is basically recording what people see on their phones. So it’s like a lot of participants download an app and the app records what people see on their phones like every two seconds or so. And these studies have shown that in the last US presidential election, for instance, people have been exposed to content about like Donald Trump less than three seconds per day. So during the US presidential election people have seen so little political content on their smartphones that it’s ridiculous and it’s so small that in my opinion it can only have small effects.
Then again a contrary argument could be it’s the average and they do find that you have a small minority who is exposed to a lot of political information but then again who are these people? Again I think they have attitudes, have priors and they have motivations, they are partisans. And yes, misinformation or content on social media can reinforce, exacerbate, radicalize them a little bit. But I think for the mass and for the general public, who’s generally not that interested in politics, etc. I don’t think it can have very strong effects.
Dan Williams: Yeah, I just want to double click on that and then I’ll bring Henry in. One other kind of stylized fact, which we should flag, which I think is surprising to some people, is if you’re the kind of person who cares about politics and follows the news carefully, and you read political commentary and so on, you are extremely unrepresentative of the average person. Most people don’t follow politics. They don’t follow current affairs closely at all.
And if you ask people very, very basic questions about politics, they are shockingly uninformed about things. That is shocking relative to the perspective of someone like us who follows politics very, very closely. And that’s another thing which I think people who are highly kind of politically engaged often get wrong when they’re thinking about this topic. If the picture in your head when you’re thinking about social media and politics and so on is that the person who’s constantly posting on X about politics is representative of ordinary people. You’ve got an incredibly skewed, misleading picture.
Okay, there’s tons, I think, more to say here. Henry, did you want to come in with any kind of pushback or any more articulation of your perspective?
Henry Shevlin: Yeah, this is all really interesting and helpful. I guess the only thing I’d say is that it seems to me social media has also just changed the kinds of voices that get platformed in the first place in a way that’s both positive and negative. But, we think about things like the rise of Tumblr and its contribution to sort of a lot of so-called, you know, woke discourse, particularly in sort of the late 2010s. And we could equally say the same thing about, for example, reactionary bloggers or neo-reactionary bloggers like Curtis Yarvin and so forth. I think these are the kind of voices that probably just wouldn’t have found an outlet in the prior social media ecosystems. Maybe that doesn’t matter, right? If none of this stuff actually impacts people’s views that much. But it does seem like an interesting shift in our broader political media landscape that social media has changed not just the kind of how much time people spend interacting with content or the way in which they do so, but also the kind of content that gets out there in the first place. Does that figure at all in the impact of these things?
Dan Williams: Sacha, before I bring you on, just want to say just one really quick thing about that, which is the reference to Curtis Yarvin there made me laugh because I think like he’s an example where like the overwhelming majority of people won’t be aware of him. But I think he probably is influential within the kind of ideas, intelligentsia space of the political right. But this idea that like social media and the affordances and incentives of social media kind of changes which voices become influential and prestigious. I think that’s such an interesting and important point, but Henry, I thought you were going in the direction of, like someone like Donald Trump can absolutely murder on social media because he’s so good at like tapping into the attention economy dynamics on social media in a way that, you know, he’ll be much less successful if we lived in a kind of Walter Cronkite kind of media environment.
But there’s this other aspect, which is like the decline of elite gatekeeping, which is characteristic of social media and it’s via that route, I think, where people like Curtis Yarvin can enter the conversation in a way in which they probably wouldn’t have been able to if you go back to like the 90s, 2000s. Sorry, I just had to double click and say that. Sacha, did you want to respond to Henry’s point?
Sacha Altay: No, yeah, I agree. I just also want to say we often mention Trump as like the example of like someone we don’t like who benefits from social media, but there are also people who we like who benefit from social media like Barack Obama. Like he used Facebook a lot during his campaign. He’s super charismatic. And if he was president today or if he was running today, he would do great on TikTok. He still does great on TikTok. Like he’s so charismatic, so good. So it doesn’t always benefit the worst actors.
And I want to say, it’s a very important point about how social media may also shape how politicians communicate. There are some studies, for instance, in France on how short format videos like TikTok is changing how parliamentary members are talking at the parliament. And there are studies showing that especially at the extreme and especially the extreme right, they are doing more and more speeches that are like with more emotions and more, I don’t know, buzzwords. And what they say is that then they post this on social media, and the more buzzwords, the more emotions, and the more all of that, the more it’s going to go viral. And so that their goal is not to convince other parliamentary members, but instead just to buzz on social media and reach some parts of the population.
Then it’s a normative question, whether it’s good or bad. Probably using emotions and stuff is bad. But you could also imagine that if they were speaking to the general public in more authentic way and try to reach them because a lot of people are not interested. That could also be good. But of course, because it’s the extreme right and stuff, we don’t like them. And I think we have good reasons not to like them. But I think we should be careful and we should also think of ways in which it could be used to do good stuff. But I agree that in general, it probably hasn’t done very good. And it’s very hard to quantify it.
Dan Williams: Just before we move on to the topic of generative AI, my view is there’s so much uncertainty in this domain when we’re asking these really broad questions like what’s the impact of social media on politics that we can’t really be very confident about any view that we might have. But it does seem like, at least in my view, a lot of the popular discourse and academic research has focused on things like recommendation algorithms and filter bubbles and so on, where I think I’m very close to your view, Sacha, in thinking that there’s just a lot of kind of unfounded alarmism. But there’s this other aspect of social media, which I think probably has been very consequential, which is just its democratizing consequence. The fact that like prior to the emergence of social media, it was a much more elitist media environment. Whereas now, anyone with a phone, a laptop, whatever, can open up a TikTok account, get on X and start posting about their views.
And I don’t think you need to view that through the lens of, well, that means they’re going to start articulating their views and then persuading large numbers of people. But what I think it does is certain views, which were kind of systematically excluded from the media environment before the emergence of social media, can now become much more normalized. And also people can achieve kind of common knowledge that other people share views that used to be much more marginalized and stigmatized. So those sorts of views can end up being more consequential in politics, even though the views themselves aren’t necessarily more widespread.
And I think you find that with things like conspiracy theories. My understanding of the empirical research, again, people like Joe Uscinski, is that the actual number of people who endorse conspiracy theories hasn’t really increased, but they do seem to play a more consequential role within politics because people with really weird conspiratorial views used to be kind of marginalized in media. Whereas now it’s very easy for them just to start expressing those views online, finding people who share similar kinds of views, coordinating with them. And so it can play a bigger role in politics, even though it’s nothing to do with, you know, mass algorithmically mediated brainwashing or anything like that.
Okay, I’m sort of conscious of time and I really want to focus on generative AI. So there was this big panic about how once we’ve got deepfakes and other features of generative AI, this was going to have really disastrous consequences on elections. It’s going to shift people’s voting intentions in all sorts of dangerous ways. Sacha, you’ve written a paper which we’ve already referred to with Felix Simon. Looking into the evidence on this and presenting a kind of framework for thinking about it, what’s your take?
Sacha Altay: I will start by saying there are three main arguments why people are worried about the effect of generative AI on the information environment. The first one is that generative AI will make it easier to create misinformation and basically to kind of flood the zone with misinformation. The second one is that it will increase the quality of misinformation, better, faster misinformation. And the last one is personalization.
Generative AI will facilitate the personalization of misinformation. I think these are the three main ones and I can go quickly over them and argue why I don’t think they are a big deal. So about quantity, I think that quantity does not really matter. Today we are exposed already to so, I mean, there’s already so much information online and we are exposed to a very tiny fraction of that information. So adding more content does not necessarily mean that people will be more exposed to it. And I think it’s particularly true in the case of misinformation, where I think demand plays a very important role. And so it’s not because there is more misinformation that people will necessarily consume more misinformation. Like it’s not because you have more umbrellas in your store that people will buy more umbrellas. There needs to be like factors, like I don’t know, rain. If it rains more, you will sell more umbrellas. But so there needs to be something, there needs to be like incentives for people to demand more misinformation, to consume more of it. And that’s why I don’t think that the quantity argument is very strong.
I think also the cost of producing misinformation are already extremely low. Like we see it with Donald Trump or whatever, they just say something that is false and they say it with confidence and that’s it. Like the costs are very low. Also, we are very imaginative as a species, like humans have come up with like incredible, fascinating, engaging stories. And of course, AI can improve our innovative skills, but still we are very good at making up stories that make us look good, that make our group look good. And so I don’t think generative AI is going to help that much in creating more misinformation. Regarding quality, yeah...
Dan Williams: Just to interrupt you so that we can sort of take these step by step. So the first worry is generative AI both with kind of large language models and the production of text, but also deepfakes. I take it you’re including kind of both of those categories. The worry is, well, this is going to just really reduce the costs of producing misinformation. Therefore you’ll get this explosion and the quantity of misinformation and that’s going to produce all sorts of negative consequences. And your view is, well, the bottleneck that matters isn’t really quantity anyway. It’s like what people are paying attention to. So you can increase as much misinformation, you can increase the amount of misinformation as much as you want. And in and of itself, that’s unlikely to have a big impact on people’s attitudes and behaviors. Do you have any thoughts about that, Henry, before we move on?
Henry Shevlin: I guess one concern would be even though media environments are flooded with content already, and I completely agree attention is the sparse commodity, maybe you could think of generative media as allowing sort of very niche areas to get flooded with content in a way that wouldn’t have been easy before. I’m just thinking here’s a silly little example, maybe an interesting example from recent media. Some of you may have seen the anti-radicalization game that was launched in the UK, featured this character about two weeks ago, featured this character called Amelia, a purple-haired anti-immigration activist in this fictional game, which was quickly seized upon by a lot of the anti-immigration right in the UK. And now there’s a flood of AI-generated content all about Amelia, mostly making her look really cool and some of it kind of playful, some of it kind of silly. But the point is this was just like a niche news story that I think people found amusing, but I think it would have died a lot quicker had it not been for the ability of people to seize upon this and generate huge swathes of content about Amelia in a very, very short time. So maybe there was just pre-existing demand there, but it would have been demand that would have been perhaps hard to meet without the ability of generative AI tools to create the content to meet that, which maybe is a difference.
Sacha Altay: Yeah, no, I mean, that’s possible. But when you look at the memes on the internet, most of them are like very cheap. It’s just like an image with some text and you just change a little bit the text and we’re probably going to go into that. But it’s the same with like deepfakes. Like cheapfakes are much more popular than deepfakes because they are super easy to do. Like you just change the date or change the location of something and boom you have your cheapfake. And that’s why they are super popular. Yeah, I don’t know, anyway.
Dan Williams: What’s the definition of a cheapfake, Sacha?
Sacha Altay: A cheapfake is just a low-tech manipulation of information. Like you have an image and you change the date of the image, or you change the location of the image. So in opposition to deepfakes, which are like high-tech, completely like for instance, generated image where like it’s usually sophisticated, et cetera. Cheapfakes in opposition, like very cheap, like that you can, most people can do with their computer without like requiring any tech skills basically.
Dan Williams: Sorry, I think I cut you off. I just wanted to give some clarity to people who weren’t familiar with that. Okay, so that’s quantity. And the next thing that you mentioned as a sort of worry that many people have is quality. That is generative AI won’t just enable us to increase the amount of misinformation but increase its quality and initially at least you’re understanding quality as being different from personalization. You’re treating that separately, is that right? Okay, so give us like, surely the concern here is just that, okay, quantity in and of itself isn’t gonna make a difference. But once we’ve got the capacity to generate like incredibly persuasive text-based arguments and deepfakes, even if it’s true that you can create these sorts of cheapfakes and they can be influential, in different contexts, surely the quality of the misinformation must make a big difference to how many people get persuaded by it.
Sacha Altay: Yeah, I think quality is the most perhaps intuitive argument because it’s the idea that you’re going to be able to create images or videos that are unrecognizable from real videos or images. And so of course people are like, how am I going to trust images or videos anymore if they are unrecognizable from real ones? So I think that’s like a very fundamental fear that people have. And I think it makes a lot of sense. It’s very intuitive. But I don’t find it very convincing.
I think it raises a lot of challenges, but I don’t think it raises enough challenges to be alarming. For instance, I think we have had this challenge before with photography. We have been able to manipulate photography in ways that we cannot distinguish them from real photography since the beginning of photography. And how did we solve this problem? Not with technical tools or whatever, but just with social norms about the use of images to not mislead others.
And we have been able to create like fake texts or say false stuff forever and we haven’t solved the problem with like some fancy tech innovation but simply by having rules, reputation, social norms and trusting more or less people based on what they have said before based on what our friends think of them based on their past accuracy and I think all of this we will still be able to use it to help us navigate an environment in which videos could be AI generated or could be real.
And I mean, something I’ve mentioned before, but quite fundamental is that, for instance, we trust the BBC or the New York Times to be broadly accurate most of the time. We also trust them to not use AI in misleading ways and not share like deepfake footages of like presidential candidates that mislead us. And I think this trust and the institutions that exist are sufficient to prevent most of the harm from this.
I think this will have effects. For instance, maybe we will be less able to trust people and sources that we don’t know. Because if we don’t have their track records, how can we trust them that the information they are sharing is true or false or AI generated or not? But I think that’s a very old problem and we will manage. It will make it more complex, but I think we’ll manage. Yeah, Henry?
Henry Shevlin: I was going to say though, isn’t there a worry that sort of new technology creates kind of normative gaps that allow for sort of a kind of annealing or a kind of recalibration of norms? I’m thinking about something here like file sharing, for example. Like I’m of the generation where, you know, Napster, the generation where suddenly it became possible to download music for free. And this created a whole bunch, a whole shift in norms where I think for my generation, at least, you know, this form of theft was basically just completely normalized. Hence we had advertising campaigns like you wouldn’t steal a car, therefore why would you download a song or a movie? And basically pirating went from something that was niche and maybe frowned upon to something that was just completely normalized.
In the same way, I think you might worry that the ease, ubiquity of generative AI is gonna shift our norms around creating fake content. And arguably we’re already seeing this. We had just a very recently the White House itself retweeting pictures, I think of a protestor at an anti-ICE rally and they had manipulated the image, right? And you know, I think if called out on that, they’d probably say, yeah, sure, you know, of course, yeah, we play around with images, you know, that’s what generative AI can do. That’s just the way things work these days, which does seem like a normative shift perhaps, one partly occasioned by technology.
Sacha Altay: My intuition is quite the opposite, is that if anything, challenges that AI, these new challenges of AI will instead increase the epistemic norms that we have. And because we want to know the truth, like we don’t want to be biased. We don’t want to be misled. We don’t want to be misinformed. And so the fact that the challenge is becoming harder, that it’s going to become harder to know if a video is true, authentic or not, is going to make us harder and harsher on people who do as the White House did, where they, we don’t know if it’s them who manipulated it or not, but they share the manipulated images that do not portray her accurately. And so I think people are going to be angry at that. And I think it’s just going to increase how people, the level of the expectation, like what people expect. And I think people are going to expect more. They’re going to expect news outlets and people to be better. I mean, it’s just a prediction. I hope I’m right. I’m an optimist, but...
Dan Williams: So can we connect that to the worry many people have about the liar’s dividend? The idea that, once we’ve got deepfakes, I mean, we’ve currently got technology to create hyper-realistic audio and video recordings, which are basically indistinguishable from reality. There’s the kind of initial worry many people have, which is, my God, people are going to become persuaded en masse that this stuff is true. And I think that’s very unsophisticated as a worry.
But then there’s another story people have which is, okay, maybe it won’t persuade people, but now that you’ve got the capacity to create these deepfakes, politicians, elites, other people who do shady things, they can use the possibility of something being a deepfake to just dismiss any kind of recording which is raised against them as evidence of them doing something shady.
And I guess connected to that as well, there’s the worry people have, which is that just as consumers of content, now if we encounter any kind of audio or video which goes against what we want to believe, we can just say, well, it’s a deepfake. I don’t have to believe it. So we’re just gonna end up becoming like more and more cocooned within our own belief system, not having this access to learn about the world via recordings. So it’s a kind of liar’s dividend worry and this general worry that this is just going to just obliterate the kind of epistemic value, the informational value of recordings. What’s your thought about those kinds of worries?
Sacha Altay: First thing, I think the liar’s dividend does not hinge on AI itself, but rather the willingness of politicians and some elites in particular to lie and to evade accountability and responsibility. AI will certainly be a new weapon in the arsenal and we have seen it in the elections in 2024, etc. Many politicians have used AI to their benefit and many politicians and elites are continuing using them. So for sure, it’s something we should be, we should worry about and we should regulate, et cetera. But will it be a particularly good weapon in the arsenal? Will it be a game changer? I’m not sure. I mean, time will tell. So far, I don’t think it has been particularly good. I don’t think it has been used in particularly good ways. I don’t think people particularly buy it. I don’t think when people share something and then they’re like, no, it was just AI or like try to use AI as the excuse. I don’t think it works very well. And I think there are going to be reputational costs for people who try to do that. We are going to remember that they have tried to do that. And so I don’t know. Again, time will tell. It’s an empirical question. I may be wrong. I don’t know. Yeah, Henry.
Henry Shevlin: I was just going to chime in. I’m sure I’m not alone in having seen on Facebook in particular, lots of cases of AI-generated media being mistaken. I don’t want to pick on boomers too much, but it is often boomers who completely seem to buy it. Like you might have seen these examples of people breaking these glass bridges, these videos that went viral and lots of people, particularly I say older respondents who completely seem to believe this is a real video they’re seeing.
But I guess two responses to that that you might push back with, Sacha, one would be like, well, we’re just in a transitional period, right? This is new. This is so new to a lot of people seeing seeing this concept for first time that they just aren’t aware yet that this is possible and they’ll adjust over time. Another would be to say, look, yeah, maybe if I’m producing a cute image of, I don’t know, a rabbit or an image of someone breaking a bridge or something non-political, it’s easier to convince people that that’s real than it would be the case to, for example, change their political views. So mean, either or both of those responses, things you’d like to go with in response to that.
Sacha Altay: Yeah, I mean, my impression is that if a rando shares a video of Macron doing something crazy, people are not going to believe it. They are going to wait for like France Info and like the real media to cover it. Because if I don’t know, Macron is saying, we are starting a new war with this country. People are not going to believe it, even if it’s very high quality, because they know if it happens, all the media are going to cover it. So I think in very in this high case of like the politician saying something absolutely crazy, people are going to be vigilant and are going to wait for the mainstream media to buy it.
I think many of the AI slop that we see a lot on Facebook, but also on TikTok, are humorous ones. I think there is some part of the boomers, but not just the boomers, who want to be entertained. And for entertainment, they don’t really care whether it’s true or not, whether it’s authentic or not. And you have extremely, you can create extremely cute images of like little animals doing cute stuff and you get what you wanted. You have like this super stimuli, super cute, super entertaining, super engaging. You have what you wanted. Like, and whether it’s authentic or not, I do care. I don’t understand how people don’t, but at the same time it’s like, yeah, it’s brain candy. It’s a candy, brain candy that people get and I don’t see why it’s wrong.
And I just want to point out that we as elites, because we have always looked down on the content that the mass population consumes. Now we look down on like short video formats on TikTok, but we have always looked down on their entertainment practices, et cetera, saying that it makes them stupid, et cetera. And so I think we should be careful about that. Careful about saying that kids are stupid because they are on TikTok and are watching short format video or whatever. I think we should be careful. And I think we are falling a bit into that with the AI slop, but the TikTok AI slop are very different from the Facebook AI slop. The TikTok AI slop are very weird and absurd. And I think they work because they are extremely weird and absurd. You know, they’re something weird about them and people are playing with it. They are playing with the fact that it’s AI and that you can do extremely weird stuff, but it’s very different from the AI slop on Facebook that works, I think, among older populations.
Henry Shevlin: Since we’re discussing TikTok, just a quick point that’s been lurking in the back of a lurking worry I’ve had is it seems to me most of the research focuses on adults. And yet a lot of the worries about both social media misinformation and generative AI misinformation concerns teenagers and young people. And I’m curious, A, whether there’s how much specifically targeted research there is looking at that group. And B, I think there probably are some good prior reasons for worrying about that group more than others, just because teenage years—firstly, our political beliefs are less likely to be stabilised at that point. And secondly, it is obviously an important window for the formation of political identities in the first place. So even if the worries about social media and generative AI misinformation are overblown for adults, could there be more to worry about there in the case of teenagers?
Sacha Altay: No, that’s very possible. That’s a point that has at least has been made for social media and mental health that very few studies have looked at adolescents or young adolescents and that’s probably the group that’s like could be the most sensible to these effects. And so that’s a totally fair point. Regarding generative AI, I also think we should acknowledge that they are also probably much better at using the technology and recognizing it, like whether it is ChatGPT, DALL-E or like all the AI technology, I think they are much better.
And that’s why the AI slop I see on TikTok are like very meta, like they are second degree, third degree, like very meta. Whereas on Facebook, they are just like first degree, like, look, I did this amazing thing. Oh look, this cute baby. So I think very different. So to be honest, I’m not so worried about teens and generative AI on TikTok. Regarding mental health, I don’t know and we need more data, but it’s a very fair point.
Dan Williams: Just on this point about quality, so we’ve been talking about deepfakes, but there’s this other aspect of generative AI, which is just producing kind of tailored text-based content. And there has been this flurry of empirical research, so I’m thinking of like Tom Costello’s work on chatbots and conspiracy theories and so on, work by people like Ben Tappin showing that LLMs can be pretty persuasive with the content that they produce, partly because they’re just very good at recruiting evidence and persuasive rational arguments that is tailored to people’s specific pre-existing beliefs and informational situation. What’s your feeling about the impact of generative AI there? Because presumably there, it’s a very different conversation about deepfakes. And it does seem to me at least that generative AI, you might argue, is going to disproportionately benefit people with sort of bad, misinformed views, because that’s often where you’re lacking kind of human capital, right? You don’t have access on tap to the sophisticated intellectual skills of the intelligentsia when it comes to a lot of this kind of lowbrow misinformation. So they can now access, you know, generative AI, at least if it’s not subject to various sorts of like safety and ethical requirements, and that might happen down the line, isn’t there a real risk there that that’s going to kind of asymmetrically benefit people pushing out misinformed conspiratorial narratives?
Sacha Altay: So it’s good you mentioned these studies because they find super large effect sizes on important topics like politics. But all the authors acknowledge that these effects are estimated in experimental settings and it’s unclear how this would translate outside of experimental settings where LLMs are not going to be prompted to convince participants or users of believing something.
So first, they are not going to be prompted to do that. Second, they are not going to be paid to pay attention and use the LLM in that way. And so that’s why also, you know, Ben Tappin has this piece on like for mass persuasion, it matters more like attention. Are people actually going to do that? Are people actually going to be exposed to that rather than how persuasive it is? And that’s why I’m not so worried.
And it’s important, I think you mentioned the symmetry or asymmetry because I don’t see any good reason why bad actors would be more successful in using generative AI to mislead than good actors using generative AI to inform and make society better or citizens more informed, et cetera. I think in general, good actors have more money, have more trust. Like in France, if the French government releases an AI or whatever to inform people, it’s going to be more successful than if it’s the Russian government. And so in many ways, I think good actors have the advantage, but they need to take it seriously. They need to act and they need to proactively use these tools for democracy and for the better. They should not wait, I think, for the bad actors to attack and them to defend. They should already be using them in the best possible ways to improve society.
Dan Williams: Yeah, my thought concerning asymmetry was just take something like Holocaust denial, right? I think to a first approximation, everyone who believes in Holocaust denial is like stupid for the most part. And if you give them access to highly intelligent generative AI tools, well, they’re gonna be able to use the kind of on-tap intelligence to rationalize that false perspective. Whereas when it comes to the truth, namely that the Holocaust actually happened, we can use generative AI maybe to improve the persuasiveness of the arguments that we’re going to generate, but we’ve already got extremely persuasive evidence and arguments, right? Because that’s where all of the intellectual research and so on exists.
In any case, again, I’m conscious of time. Could we end with this point about personalization? So I still meet people who think that Brexit was due to Cambridge Analytica and micro-targeting and things like this. I think it’s a very kind of common belief people have, which is that once you start targeting personalizing messages, you can have like really huge impact on what people believe. And one of the consequences of AI, very broadly construed, is that they’re gonna greatly enhance the personalization of persuasive messages. So what’s your take on that?
Sacha Altay: Maybe the best evidence is actually the papers by Ben Tappin, Tom Costello and stuff who have actually measured what matters more. Is it whether the arguments generated by the LLMs are targeted to the users based on their political identity, etc., or whether they present more facts and the quality of the facts, etc. And in general, what they find is that what matters is facts. So the more you provide people with facts and good arguments, the more they change their mind. And personalization matters very little.
And in political science, there’s a whole literature showing mostly the same thing, that like, of course you need some targeting, like you need to target based on the language or like some basic level of targeting is needed, but like micro-targeting based on like, yeah, political preferences, values, et cetera, broadly ineffective basically, especially compared to the most convincing arguments you can make.
I think also there is a whole literature in like communication showing that people highly dislike targeted messages when they are very targeted, when they feel like it’s very targeted at them, people recognize it and they dislike it. Yeah, the Cambridge Analytica thing is just a scam basically. I still don’t know why people believe it that much. It’s just a company. They are selling influence. They said they influence major elections and all of a sudden people are like, oh yeah, of course I understand why they do that. People have priors about other people being gullible and being swayed by social media. So when a company said that they sway people on social media, people are receptive to it. They’re not being gullible. It’s just on their priors, et cetera. But yeah, no, there is very little evidence that Cambridge Analytica affected the Brexit or the 2016 US presidential election. And it’s better to present people with good arguments and facts rather than to micro-target them.
Henry Shevlin: If I can squeeze just another angle into the personalization discussion, something you talk about in the paper is relational factors, which is sort of related to personalization, but a bit distinct. And I’m curious about whether you think AI could play a role there. We’ve talked on the show previously about social AI and the idea that young people in particular might be forming deeper and more profound relationships with AI systems or AI friends, companions, lovers, which then potentially could be leveraged for changing their views.
And it seems to me just intuitively that these kind of relations, whether they’re sort of direct relations or more like parasocial relations, can be really influential if we think about, for example, something like Logan Paul’s Prime Energy Drinks. You know, this was an influencer who promoted his own brand of energy drinks that then became a massive sensation, hundreds of millions of dollars, if not billions of dollars in sales over a very short period of time. So it seems like these relationships can be powerful. Is that not a worry that AI could leverage them?
Sacha Altay: And to be honest, I’ve been, it’s very hard, it’s a very hard question. I’m being asked that all the time. And I think the best counter-argument I have at the moment is just, there is very little evidence that people change their mind according to their life partner. Like the people they trust the most, they sleep with, et cetera. There is very little change of mind. And when there is, it’s hard to know whether it’s because the incentives are getting more aligned. Like, you know, they get married, so they are sharing their money, they are buying a house together, they live in the same place, etc. So of course when the incentives are getting closer, you could imagine their beliefs, etc. are getting closer. But basically attitude change is very small with your life partner.
And I imagine that if my wife, who I trust a lot, I love, etc. tells me, GMOs are bad, nuclear energy is bad, etc. Why would she convince me? Like I trust her a lot on many things, but I’m not like completely blind to her. And so how would ChatGPT beat my wife at this, like, I don’t see it, I don’t see it. But to be honest, it’s just my opinion, let’s see how it goes, but I don’t find it very convincing.
Dan Williams: I can confirm that my girlfriend would very much like to influence my political attitudes, but is not having much success as of yet. Okay, one thing we didn’t do actually is you’ve given us your kind of analysis and your belief, Sacha, about the impact or lack of impact of generative AI. But we should mention there were all of these sort of alarmist forecasts about the impact of generative AI and deepfakes on the kind of 2024 election cycle.
And one of the things that you do in your paper is you don’t just go through each individual worry, but you actually kind of survey what the empirical research that we have says. So briefly, what does the research that we have actually say about the impact of generative AI on that election cycle?
Sacha Altay: I mean, to be honest, it’s not like a systematic review, like it’s not super reliable. I just went over and looked at what happened in these elections. And basically, in most countries, the consensus is that there have been some problems with elections, but that it’s old problems with elections, such as politicians lying, trying to gain, to change, like basically politicians doing bad stuff. And generative AI has been used a lot to illustrate what politicians want to say. Often they want to say that they are strong and that their opponent is weak or stupid. So they have been using generative AI to do that in the US, in Argentina, in many countries. They have used generative AI a lot to do some kind of like soft propaganda, portraying themselves and their group as good and the others as bad.
In some countries, apparently, generative AI has been used to do some good stuff, like in India, where we have like many languages and where translation is often a problem and takes time. And apparently, generative AI has been used a lot to translate some political campaigns into all the languages and dialects that exist in India. So I think it’s very varied and not as catastrophic, let’s say, as the alarmist tech suggests. But I think it’s just suggestive evidence. And of course, it’s just the beginning of generative AI. So we should see how generative AI will be used in the future, in future elections. But we should not forget that it can be used to do good stuff. Like it’s not necessarily being used to do bad stuff. You can use it to translate to, and even to illustrate, you can use it to do like faithful imitation, illustration. You don’t need to like portray yourself as super strong and the opponent as bad. You can do, I don’t know, some good or artistic stuff.
Dan Williams: Yeah, we didn’t really talk about the positive side of generative AI very much in this conversation. But my view is, at the moment at least, the kind of boring truth about large language models is that they’re basically just improving people’s access to evidence-based kind of factual information. And I think if you compare the kind of like one-shot answer you get from ChatGPT or Claude or Gemini on any political issue to what you get from the average voter or pundit or politician, it’s just of much higher quality. But I think that truth doesn’t really get the attention that it deserves because it’s sort of boring for the most part. It doesn’t fit into these kind of threat narratives. And it’s kind of counterintuitive because like, why would it be that these, you know, profit-seeking companies that everyone despises have just had a really beneficial consequence on the information environment? But that is in fact, what I think the case is.
Sacha Altay: So you’re totally right because another concern I haven’t mentioned is just hallucinations, like individual users using LLM on their own and being misled by an LLM because they confidently say stuff that is false. But as you say, I think it depends compared to what? How often do they hallucinate and how correct are they compared to alternative sources of information like other human beings, social media, TV?
And I think they would do pretty well actually compared to most of these other sources. And so that’s why I’m not so worried. I think the confidence thing is a bit annoying, but I think most people who use AI regularly kind of know that, yeah, sometimes they completely hallucinate and they go completely awry, but we know it. And I think most people who use it often know it. And that’s why I’m not so worried. But again, it would be better if they did not hallucinate and were perfect, but it’s setting the bar a bit high.
Dan Williams: Okay. Okay, fantastic. Well, thank you, Sacha. We’re going to have to bring you back on at some point because I feel like we’ve just barely scratched the surface with many of these issues. Was there anything that we didn’t ask you that you wished we had asked you?
Sacha Altay: No. I mean, as you said, many things to talk about.
Dan Williams: Okay, fantastic. Well, thanks, Sacha, and we’ll see everyone next time
This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.conspicuouscognition.com/subscribe
AI Sessions #7: How Close is "AGI"?
vendredi 9 janvier 2026 • Durée 01:26:51
Keywords
AGI, artificial general intelligence, AI progress, transformative AI, human intelligence, skepticism, economic impact, political implications, cultural shift, predictions
Summary
In this conversation, Dan Williams and Henry Shevlin discuss the multifaceted concept of Artificial General Intelligence (AGI) and various controversies surrounding it, exploring its definitions, measurement, implications, and various sources of scepticism. They discuss the potential for transformative AI, the distinctions between AGI and narrow AI, and the real-world impacts of AI advancements. The conversation also touches on the philosophical debates regarding human intelligence versus AGI, the economic and political ramifications of AI integration, and predictions for the future of AI technology.
Takeaways
AGI is a complex and often vague concept.
There is no consensus on the definition of AGI.
AGI could serve as a shorthand for transformative AI.
Human intelligence is not a perfect model for AGI.
Transformative AI can exist without achieving AGI.
Incremental progress in AI is expected rather than a sudden breakthrough.
Skepticism towards AGI is valid and necessary.
AI's impact on the economy will be significant.
Political backlash against AI is likely to increase.
Cultural shifts regarding AI will continue to evolve.
Chapters
00:00 Understanding AGI: A Controversial Concept
02:21 The Utility and Limitations of AGI
07:10 Defining AGI: Categories and Perspectives
12:01 Transformative AI vs. AGI: A Distinction
16:15 Generality in AI: Beyond Human Intelligence
This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit
AI Sessions #6: AI Companions and Consciousness
samedi 20 décembre 2025 • Durée 01:07:04
In this episode, Henry and I spoke to Rose Guingrich about AI companions, consciousness, and much more. This was a really fun conversation!
Rose is a PhD candidate in Psychology and Social Policy at Princeton University and a National Science Foundation Graduate Research Fellow. She conducts research on the social impacts of conversational AI agents like chatbots, digital voice assistants, and social robots. As founder of Ethicom, Rose consults on prosocial AI design and provides public resources to enable people to be more informed, responsible, and ethical users and developers of AI technologies. She is also co-host of the podcast, Our Lives With Bots, which covers the psychology and ethics of human-AI interaction now and in the future. Find out about her really interesting research here.
You can find the first conversation that Henry and I had about Social AI here.
Conspicuous Cognition is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.
Transcript
Note: this transcript is AI-generated and may feature mistakes.
Henry Shevlin (00:01)
Hi everyone and welcome to the festive edition of Conspicuous Cognitions AI Sessions. We’re here with myself, Henry Shevlin, my colleague Dan Williams and our guest today, Rose Guingrich, who we’re very lucky to have on the show to be talking about social AI and AI companions with us. We did do an episode on this two episodes ago, which featured me and Dan chatting about the rising phenomenon of social AI. And so if anyone wants a basic sort of primer on the topic, go back and listen to that as well. But today we’re going to be diving into some of the more empirical issues and looking at Rose’s work on this topic.
This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit
AI Sessions #5: How AI Broke Education
jeudi 4 décembre 2025 • Durée 55:56
Henry Shevlin and I sat down to discuss a topic that is currently driving both of us slightly insane: the impact of AI on education.
On the one hand, the educational potential of AI is staggering. Modern large language models like ChatGPT offer incredible opportunities for 24/7 personal tutoring on any topic you might want to learn about, as well as many other benefits that would have seemed like science fiction only a few years ago. One of the really fun parts of this conversation was discussing how we personally use AI to enhance our learning, reading, and thinking.
On the other hand, AI has clearly blown up the logic of teaching and assessment across our educational institutions, which were not designed for a world in which students have access to machines that are much better at writing and many forms of problem-solving than they are.
And yet… there has been very little adaptation.
The most obvious example is that many universities still use take-home essays to assess students.
This is insane.
We discuss this and many other topics in this conversation, including:
* How should schools and colleges adapt to a world with LLMs?
* How AI might exacerbate certain inequalities.
* Whether AI-driven automation of knowledge work undermines the value of the skills that schools and colleges teach today.
* How LLMs might make people dumber.
Conspicuous Cognition is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.
This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit
AI Sessions #4: The Social AI Revolution - Friendship, Romance, and the Future of Human Connection
jeudi 20 novembre 2025 • Durée 01:11:56
In this conversation, I explore the surprisingly popular and rapidly growing world of ‘social AI’ (friendbots, sexbots, etc.) with Henry Shevlin, who coined the term and is an expert on AI companionship.
We discuss the millions of people using apps like Replika for AI relationships, high-profile tragedies like the man who plotted with his AI girlfriend to kill the Queen, and the daily conversations that Henry’s dad has with ChatGPT (whom he calls “Alan”).
The very limited data we have suggests many users report net benefits (e.g., reduced loneliness and improved well-being). However, we also explore some disturbing cases where AI has apparently facilitated psychosis and suicide, and whether the AI is really to blame in such cases.
We then jump into the complex philosophy and ethics surrounding these issues: Are human-AI relationships real or elaborate self-deception? What happens when AI becomes better than humans at friendship and romance?
I push back on Henry’s surprisingly permissive views, including his argument that a chatbot trained on his writings would constitute a genuine continuation of his identity after death. We also discuss concerns about social de-skilling and de-motivation, the “superstimulus” problem, and my worry that as AI satisfies our social needs, we’ll lose the human interdependence that holds societies together.
Somewhere in the midst of all this, Henry and I produce various spicy takes: for example, my views that the sitcom ‘Friends’ is disturbing and that people often relate to their pets in humiliating ways, and Henry’s suspicion that his life is so great he must be living in a simulated experience machine.
Conspicuous Cognition is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.
This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit
AI Sessions #3: The Truth About AI and the Environment
dimanche 2 novembre 2025 • Durée 01:41:06
I sat down with Henry Shevlin and Andy Masley to discuss AI’s environmental impact and why Andy thinks the panic is largely misplaced. Andy’s core argument: a single ChatGPT prompt uses a tiny fraction of your daily emissions, so even heavy usage barely moves the needle.
The real issue, he argues, isn’t that data centers are wasteful—they’re actually highly efficient—but that they make visible what’s normally invisible by aggregating hundreds of thousands of individually tiny tasks in one location. And he argues that the water concerns are even more overblown, with data centers using a small fraction compared to many other industries. We also explored why “every little bit counts” is harmful climate advice that distracts from interventions differing by orders of magnitude in impact.
In the second half of the conversation, we moved on to other interesting issues concerning the philosophy and politics of AI. For example, we discussed the “stochastic parrot” critique of chatbots and why there’s a huge middle ground between “useless autocomplete” and “human-level intelligence.” We also discussed Marx, “technological determinism”, and how AI can benefit authoritarian regimes.
Finally, we touched on effective altruism, the problem of “arguments as soldiers” in AI discourse, and why even high-brow information environments contain significant misinformation.
I enjoyed this conversation and feel like I learned a lot. Let me know in the comments if you think we got anything wrong!
This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit
AI Sessions #2: Artificial Intelligence and Consciousness - A Deep Dive
mardi 14 octobre 2025 • Durée 01:19:22
In this conversation, Dr Henry Shevlin (University of Cambridge) and I explore the complex and multifaceted topic of AI consciousness. We discuss philosophical and scientific dimensions of consciousness, discussing its definition, the challenges of integrating it into a scientific worldview, and the implications of such challenges for thinking about machine consciousness. The conversation also touches on historical perspectives, ethical considerations, and political issues, all while acknowledging the significant uncertainties that remain in the field.
Takeaways
* Consciousness is difficult to define without controversy.
* The relationship between consciousness and scientific understanding is extremely complex.
* AI consciousness raises significant ethical questions.
* The Turing test is a behavioural measure of intelligence, not consciousness.
* Historical perspectives on AI consciousness are helpful for understanding current debates.
* Cognition and consciousness are distinct but related.
* There is a non-trivial chance that some AI systems may have minimal consciousness.
* Consciousness in AI systems is a scientific question, not just a philosophical one.
* The debate on AI consciousness is messy and strangely polarising (and often heated) but fascinating and important.
Chapters
00:00 Exploring the Nature of Consciousness
17:51 The Intersection of AI and Consciousness
36:16 Historical Perspectives on AI and Consciousness
59:39 Ethical Implications of AI Consciousness
Conspicuous Cognition is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.
This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit
AI Sessions #1: AI - A Normal Technology or a Superintelligent Alien Species?
Saison 1 · Épisode 1
jeudi 18 septembre 2025 • Durée 01:16:38
Is artificial intelligence (AI) a “normal technology” or a potentially “superintelligent” alien species? Is it true, as some influential people claim, that if anyone builds “super-intelligent” AI systems, everyone dies? What even is superintelligence”?
In this conversation, the first official episode of Conspicuous Cognition’s “AI Sessions”, Henry Shevlin and I discuss these and many more issues.
Specifically, we explore two highly influential perspectives on the future trajectory, impacts, and dangers of AI.
The first models AI as a “normal technology”, potentially transformative but still a tool, which will diffuse throughout society in ways similar to previous technologies like electricity or the internet. Through this lens, we examine how AI is likely to impact the world and discuss deep philosophical and scientific questions about the nature of intelligence and power.
The second perspective presents a very different possibility: that we may be on the path to creating superintelligent autonomous agents that threaten to wipe out the human species. We unpack what "superintelligence" means and explore not just whether future AI systems could cause human extinction but whether they would “want” to.
Here are the primary sources we cite in our conversation, which also double up as a helpful introductory reading list covering some of the most significant current debates concerning artificial intelligence and the future.
This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit
Will AI Change Everything?
Saison 1 · Épisode 1
mardi 19 août 2025 • Durée 01:11:37
How similar is ChatGPT-5 to the human mind? How should we measure AI progress? How close are we to transformative AI? Are "super-intelligent" AI sytems likely to kill us all?
This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.conspicuouscognition.com/subscribe
This week, I talked with the great Elle Griffin about “free speech in the age of social media.” Elle is a writer, essayist, and general polymath who writes the excellent newsletter The Elysian. I would highly recommend following her work and Substack.
We talked about a wide range of topics, including:
* What ismisinformation?
* Is the misinformation problem worse today than in the past?
* Why is censorship bad? And why is free speech good?
* In what ways is misinformation symptomatic of deeper problems like polarisation and institutional distrust?
* Should we treat online spaces like public parks?
* How can we design social media platforms so that they encourage prosocial behaviour? (Elle had some very interesting ideas here).
* In what ways is the enforcement of “epistemic norms” (e.g., against lying) different to the enforcement of other norms (e.g., against littering)?
* Do we really want social media platforms to be free of conflict? Should all platforms be comfortable?
I got a lot from the conversation and hope people enjoy it.
22:13 Skepticism and Progress in AI Development
28:42 The Evolution of LLMs and Their Capabilities
30:49 Moravec's Paradox and Its Implications
33:05 The Limits of AI in Creativity and Judgment
37:40 Skepticism Towards AGI and Human Intelligence
42:54 The Jagged Nature of AI Intelligence
47:32 Measuring AI Progress and Its Real-World Impact
56:39 Evaluating AI Progress and Benchmarks
01:02:22 The Rise of Claude Code and Its Implications
01:04:33 Transitioning to a Post-AGI World
01:15:15 Predictions for 2026: Capabilities, Economics, and Politics
So try to imagine a house that’s not a home. Try to imagine a Christmas all alone and then be reassured that you don’t have to spend Christmas all alone. In fact, nobody ever needs to spend Christmas alone ever again because their AI girlfriend, boyfriend and B friend, husband or wife will be there to warm the cockles of their heart throughout the festive season with AI generated banter and therapy. Or at least this is what the promise of social AI might seem to hold. And in fact, just in today’s Guardian here in the UK, we saw an announcement that a third of UK citizens have used AI for emotional support. Really striking findings.
So cheesy intro out of the way. Rose, it’s great to have you on the show. Tell us a little bit about where you think the current sort of social AI companion landscape is at right now and what the major sort of trends and use patterns you’re seeing are.
Rose E. Guingrich (01:36)
So right now it appears as though we are moving toward an AI companion world where people are less judgmental about people using AI companions. It’s much less stigmatized than it was a couple of years ago. And now, of course, we’re seeing reports where, for example, three quarters of U.S. teens have used AI companions and about half are regular users and 13% are daily users. And so we’re seeing this influx of AI companion use from young people and also children as well, of course, from the reports that we’ve seen about teens using AI as a companion.
And I think looking forward, we’re only going to see more and more use of AI companions as companies recognize that the market is ready for these sorts of machines to come into their lives as these social interaction partners. And then if you look even further forward, these chatbot companions are going to soon transition into robot companions. And so there we’re going to see even more social impacts, I think, based on embodied conversational agents.
Dan Williams (02:46)
Can I just ask a quick follow up about that, Rose? So you said that this is becoming kind of more prevalent, the use of these AI companions. You also said it’s becoming less stigmatized. Do we have good data on that? Do we have data in terms of which populations are stigmatizing this kind of activity more or less?
Rose E. Guingrich (03:06)
So in terms of the stigma, we don’t have a lot of information about that. But we can look at, for example, a study that I ran in 2023 where I looked at people’s perceptions of AI companions, both from those who were users of the companion chatbot Replica and those who were non-users from the US and the UK. And the non-users perceptions of AI companions and people who use AI companions at that time was fairly negative. So for example, non-users indicated that it’s a sad world we live in if these things are for real. These AI companions are for people who are social outcasts or lonely or can’t have real friends.
And now in the media at least, we see a lot more discourse on AI companions and sharing about having AI companions. And one thing I can point to are subreddits. For example, My Boyfriend Is AI that has 70,000 companions. It is explicitly labeled as companions, whereas other subreddits are weekly visitors, visitors, users. This is companions and people on the subreddit are talking about their AI girlfriend, boyfriend, partner, whatever, and finding community there. Now, if you look at that subreddit though, you also see people talking about disclosing their companion relationship to friends or family and receiving backlash, but then there are also people who are indicating that people are seeing this as, this could maybe be valuable to you, I don’t think it’s necessarily a weird thing, but I think that’s also due to the shifting of social norms based on how many reports we’re seeing about AI companion use and knowing that people also use not just specifically AI companions as social interaction partners but also these GPTs like Claude, Gemini, etc. that people are turning to as companions as well and also being quite open about it.
Henry Shevlin (04:59)
It’s been really fascinating to see, because I think we met, would it have been summer 2023, Rose, or maybe 2022, at an event in New York and the Association for the Scientific Study of Consciousness, presenting a paper on your 2023 study. I was presenting a paper on social AI and AI consciousness. And it felt like then absolutely no one was talking about this. Replica was already pretty successful, but basically no one I spoke to had even heard of it. And then it’s really in the last couple of years that things have accelerated fast. And now basically every couple of days, a major newspaper has some headline about people falling in love with their particular companion or sometimes tragic incidents involving suicides or psychosis, or sometimes just sort of observation level studies about what young people today are doing and so forth. Is that your perception that this is accelerating fast?
Rose E. Guingrich (05:53)
Definitely. And we’re also seeing an emerging market of AI toys. So AI companions that are marketed specifically for children. And so even though right now mainly we’re seeing companion use from young people, young adults, we’re now shifting it toward children as well. Ages 3 through 12 is what these toys are marketed for. And they’re marketed as a child’s best friend. So these are going to be the forever users, right? Starting young with AI companions and then moving forward into robot companions that will someday have in our homes, well, it’s just a natural progression of what this is going to look like.
Dan Williams (06:28)
Can I ask a question just about the kind of commercial space here? So there is a company like Replica and they make, I guess, bespoke social AIs, AI companions. Presumably though, the models that they’re using underpin those AIs and not as sophisticated as what you’ve got with OpenAI and Anthropic and these other, you know, Google’s Gemini and so on. Is that right? Are they using their own models? And if they are, then presumably those models aren’t as sophisticated as these sort of cutting edge models used by the leading companies in the field.
Rose E. Guingrich (07:02)
I suppose it depends on what you mean by sophistication. I think sophistication has a lot to do with the use case. So for Replica, the sophistication aspect of it is, well, obviously people are finding it useful and finding it sophisticated enough to meet their social needs and to operate as a companion. But of course it doesn’t have the level of funding and infrastructure that these big tech companies have like OpenAI to make their models be quote unquote more sophisticated, perhaps have better training data and are better suited to multiple use cases given that they’re operating as general purpose tools.
But way back when in 2021, Replica was operating on the GPT-3 model, but got kicked off of it because in 2021, OpenAI changed their policy such that any third parties using their model could not use it for adult content. But of course, fast forward to this year, Sam Altman is saying, oh, everyone’s upset about GPT no longer feeling like a friend. Don’t worry adult users, you can now use ChatGPT for adult content. So, you know, full circle, all operating under what is it that users say that they want. Here we’re going to give it to them so they continue to use our platform.
Henry Shevlin (08:18)
So it’ll be interesting to watch whether sort of as ChatGPT, you know, someone has said that he wants erotic role play to be offered as a service to adults, treat adults like adults seems to be the kind of mantra there. And of course Grok has already got Annie and a couple of other kind of companions. So do you think it’s likely that sort of we’ll see this as just no longer a kind of niche industry, but something that just gets baked into sort of the major commercially available language models?
Rose E. Guingrich (08:47)
Yeah, I would say so. I don’t think it’s niche anymore at all, actually. Given that these large language models, these GPTs can be used as companions. And if you look at the metrics in the reports, for example, by OpenAI, something like 0.07% of users, which is for the GPT-5 model, which is equivalent to about 560,000 people, use ChatGPT as a companion, show signs of psychosis or mania. And then others, for example, it’s like 0.15%, which is over a million people, show potentially heightened levels of attachment, emotional attachment to ChatGPT. So I think that’s an indicator that it’s no longer niche, right? And you’re just seeing so many more AI companions being pushed out on the market every month.
Dan Williams (09:39)
And so right now when we’re talking about AI companions, we’re talking about, for the most part, these large language models and chatbots and so on. You mentioned in terms of where this might be going, the integration of robotics into this space. So what are we seeing there at the moment? And how do you think that’s likely to develop over the next five years, 10 years, 20 years?
Rose E. Guingrich (10:01)
Yeah, so what we’re seeing at the moment is there are actually humanoid robots in people’s family dynamics in people’s homes more so in Japan than in the rest of the world and this is sort of highlighted by institutions being created in order to study human robot interaction and to understand how the integration of robots into family dynamics might impact child social development. So also with the onset of AI toys where now the chatbots are embedded into something that’s embodied. That’s sort of like a signal to robotics.
And of course we do also have social robots like PARO, which is a robot seal that is designed for elderly people who need companionship. It’s supposed to help reduce, for example, the onset of dementia and Alzheimer’s and help with social connection. And then we have workplace robots like Pepper. And so these are kind of the early stages of robotics, but I’m seeing a big shift into multi-modal AI, so that’s embodied, that has video, voice, image generation, all of these sorts of things. And I think those features compounded are just going to increase the rate at which people are going to be using these tools as companions and get more emotionally attached to them, perceive them as more human-like, and therefore have greater social impacts from interacting with them.
Henry Shevlin (11:29)
You know, one of my best top recommendations for fiction about social AI is Ted Chang’s The Life Cycle of Virtual Software Objects, a great sort of novella about a company that offers sort of virtual pets, although they’re sophisticated, cognitively sophisticated pets you can talk to. And there’s a great sort of whoa moment in the middle of the story when you realize that the users who’ve been interacting with these things in virtual worlds, they can then interact with them in the real world. You have these little robot bodies, they can port them onto and I can easily imagine sort of something like that happening with large language models and social AI. I mean already you know I can do the kind of live streaming with ChatGPT or Gemini and it can comment on what’s happening around me so this idea of sort of embedding these things in the real world environments I think we’re seeing yeah even already on ChatGPT we’re seeing trends in that direction.
Rose E. Guingrich (12:20)
Yeah, and I think of Clara and the Sun as well, the novel that is about children who grow up with a humanoid robot companion and everyone has a humanoid robot companion. And what happens is that because of this, parents actually have to coordinate play dates between children because they’re not engaging socially with other kids at baseline, at default, because they have a companion that is made for them and fulfills their social needs. So that’s part of my worry, I suppose, looking forward, is if we get to that sort of point where now we have to really try very hard to facilitate human connection when it’s already at this stage more difficult than it has ever been due to various technologies.
Henry Shevlin (13:01)
So yeah, let’s talk a little bit more about sort of what your work has revealed about risks and benefits of social AI. So I mean, a point I’ve made on the show and I like making a lot is that very often it’s quite hard to predict what the psychosocial impact of new technologies will be. You know, I grew up in an era where there was a massive moral panic around violent video games that basically failed to pan out. Turns out violent video games don’t have dramatic effects on development. On the other hand things like social media and short-form video have had I think quite really quite significant psychosocial effects that people largely fail to anticipate. Tell us a little bit about what your research in this area has found about the psychosocial impacts of social AI.
Rose E. Guingrich (13:45)
Yeah, so when we ran our study where we looked at the perceptions of AI companions on both the user and the non-user side of Replica, we asked Replica users how has interacting with the Replica or having a relationship with the chatbot impacted your social interactions, relationships with family and friends, and self-esteem? So key metrics for, for example, social health.
And we also asked them about their perceptions of the chatbot. So we asked them to what degree they anthropomorphize the chatbot or perceived as having human likeness, experience or emotion, agency or the ability to act on one’s own accord or even consciousness or subjective awareness of itself and the world around it and of the user. And what we found is that for the users on average, they indicated that having a relationship with the chatbot was positive for their social interactions, relationships with family and friends and self-esteem. Positive impact on their social health. And for the non-users, they tended to indicate that, I think having a relationship with this chatbot would actually be neutral to harmful to my social health.
And what was interesting, though, is we wanted to understand how their perceptions of the chatbot played a role in these sort of social impacts. And what we found, even though there were differences between groups in terms of we think positive social impacts, we think negative social impacts, for both groups, the more they anthropomorphize the more likely they were to indicate that interacting with the chatbot would have a positive effect on their social health.
But with this study, it was self-report and self-selecting groups. So people who were already users of the companion chatbot Replica, people who were already not users. And it was just one point in time and correlational, of course. And so recently we conducted a longitudinal study in which we randomly assigned people to either interact with the companion chatbot Replica for at least 10 minutes a day across 21 consecutive days or to control group, which was to play word games for at least 10 minutes a day across 21 days.
Rose E. Guingrich (15:46)
We chose this control condition because it was gamified, was novel table experiences, and it was using technology but just not technology that was social. It involved typing words on a screen, but there’s a different interaction form there. And we tracked their impact to the relationships from doing this daily task and also their perceptions of the agent that they were interacting with.
And what we found corroborated our findings from previous study where the chatbot users, the people who anthropomorphize the chatbot more, also reported that interacting with the chatbot had greater impacts on their social interactions and relationships with family and friends. And that was just the general impact. We didn’t look at positive or negative. But then when we looked at whether it was positive or negative, it was once again a positive relationship. The more they anthropomorphize the chatbot, the more likely they were to indicate that it had positive social benefits to them in terms of their impact on their relationships.
So we thought this was quite interesting and we found there that anthropomorphism was actually a key explanatory factor. So something about anthropomorphizing the chatbot rendered it to have the ability to impact their social lives. And so it seems like this is kind of the narrative that’s coming out from the research based on some theory work that I did initially and then these studies that I ran, that this is something that we really need to think about. Anthropomorphism of the chatbot, whether it’s on the user side and what social motivations push people to anthropomorphize the chatbot or what characteristics of the chatbot push people to anthropomorphize it with certain characteristics.
Dan Williams (17:28)
So how are you measuring the degree to which they anthropomorphise the chatbot there?
Rose E. Guingrich (17:32)
So we use a combination of the Godspeed Anthropomorphism Scale and also scales that measure experience and agency. And then we created a scale to measure consciousness, which was by consciousness of itself, consciousness of the world around it, and consciousness of the user, and just generally subjective awareness of oneself and the world around them.
And so we use this combination scale to get at multiple pieces of attributing human likeness to the chatbot with a special focus on human-like mind characteristics, which has been a key focus of researchers who have looked at anthropomorphism and finding that it is these human-like mind traits that are perhaps the most critical element of anthropomorphism in terms of the sort of relationships between that type of anthropomorphism and subsequent social impacts.
Dan Williams (18:22)
That’s interesting. I mean, it makes me think. When I’m talking to ChatGPT, I feel like there are some ways in which I attribute traits, which are a form of anthropomorphizing. I assume that it’s got a kind of intelligence, a kind of cognitive flexibility. It seems like it has, you know, beliefs, desires and so on to a certain extent. But I also feel like I’m dealing with a profoundly non-human system that lacks like most of the personality, motivational profile and so on that I associate with human beings. Do you have more sort of granular data on exactly the kinds of traits that they’re attributing to these systems?
Rose E. Guingrich (19:00)
Yeah, so the kinds of traits that they’re attributing, for example, within the experience and agency and consciousness and human likeness scale, these are traits like the ability to feel pleasure or pain or love or hunger or the ability to remember or act on one’s own accord or act immorally or morally. And so these are the sorts of traits that people are attributing to these AI agents.
And one thing that is worth saying is that the research on anthropomorphism, different researchers have measured anthropomorphism in different ways. Some are looking more at just general human likeness by the Godspeed scale, which I think in itself is a little bit limited just because the measures are things like how dead or alive does this thing seem? How non-animated or animated? How machine-like or human-like? How non-responsive and responsive? And if you’re thinking about chatbots, well, they’re clearly responsive. So I think having these additional measures are really important for getting at the more fine-grained human-like mind traits that typically are more representative of something that only humans can have or do, especially things like second-order emotions like embarrassment or something like that.
Henry Shevlin (20:17)
So I’m curious, Rose, I love your research on this and I’ve quoted it a lot to sort of push back against that instinctive yuck fact that people have or the instinctive assumption that social AI must be obviously bad for you. But I’m curious how far you think this kind of data goes to diffusing worries about social AI and whether there are any sort of particular worries that it doesn’t address. Yeah, and I guess more broadly, I’m curious about where you see sort of the risk and threat landscape with this technology right now.
Rose E. Guingrich (20:45)
Yeah, that’s a great question. With this research that’s been done so far, it’s fairly limited in terms of, for example, just the time that people are spending with these chatbots. And, you know, there’s a lot of research on current users of companion chatbots. So there it’s limited in the self-selecting nature of the sample. But then even with these randomized control studies that are taking place over multiple weeks, the longest study that I’ve seen so far is a five-week study where people were randomly assigned to interact with a companion chatbot or just with a GPT model and interact with it either in a transactional or social way.
And so I think we are limited in that we really don’t know what the longer term effects are of people who choose to use AI as companions, especially when it comes to, for example, expectations of what a relationship looks like, whether or not these chatbots will replace human relationships, and to what degree these interactions with chatbots might contribute to overall social de-skilling in the longer term.
I think it’s really important to look at the shift in social norms in terms of what a relationship looks like and what it constitutes. And I think companion chatbots really shift that, especially when you see things like people preferring more sycophantic chatbots that are more agreeable. They indicate that they like interacting with a chatbot because it is non-judgmental and it’s always present and always responsive. And these are things that humans can’t always do, especially with the responsive part of things, but humans could be always agreeable if, for example, the expectation is that in order to stay, for example, competitive in the age of companion AI, I must be very agreeable and sycophantic when I’m interacting in close relationships, otherwise people just turn to a chatbot instead.
And so I think those are some of the risk factors that we can potentially see emerging in the longer term. But we just don’t know what’s going to happen in, you know, five, ten years, but I do worry that if the design of companion chatbots stay as they are, where it sort of promotes this retention of staying within a human chatbot dyad and not necessarily promoting external human interaction, that we’re going to see more replacement happen. But I think if the design changes such that it promotes human interaction, there can be quite a bit of benefit.
Dan Williams (23:11)
So if we think about that, the negative scenario or scenarios there, so one of them is these AI companions as a kind of substitute for human relationships. Another is de-skilling, so using these AI companions and in the process losing the kinds of abilities, dispositions that would make you an attractive cooperation partner. And you also suggested that once you’ve got a landscape of AI companions, then human beings in order to compete with these AI companions are gonna have to become more sycophantic and that seems incredibly dystopian.
But in terms of, let’s suppose that the technology gets better and better. These AI companions become better and better at satisfying people’s social needs, maybe their sexual needs. So they come to function as substitutes. People do end up with this de-skilling. They become less motivated, less capable of engaging in human relationships. So what, why is that a bad thing? Why should we care about that if that’s the outcome?
Rose E. Guingrich (24:09)
Yeah, I mean when you look at people’s outcry against AI companions, you have to ask why is it that they are so upset? And if you look at why they’re so upset, what appears to be the prevailing narrative is that human relationships are essential. We need human relationships and those should not be replaced. But if you look a little bit deeper at why that is the case, you see a lot of good reasoning for wanting to maintain human relationships.
So based on a lot of psychological research, human relationships help with people’s both mental and physical health. For example, loneliness is considered a global health crisis because it contributes to, for example, physical harms that are equal to or as worse as, for example, heart disease or heavy smoking. So loneliness and lack of relationships and social connections with other people actually contribute to a decline in physical health. And then there’s, of course, also the mental effects that are also combined with physical health effects. And at least based on the research, it just appears as though human relationships and feeling connected to other people is essential and not replaceable.
And it’s also worth pointing out that people who seek out AI companions indicate that what they really want is companionship. And they would ideally like human companionship, but for whatever reason, there are certain barriers to attaining that, whether it be environmental factors, financial factors, or social factors, or individual predispositions such as social anxiety that prevent people from being able to attain what it is that they really value and what will make them truly happy.
Dan Williams (25:56)
Yeah, I totally buy the idea that loneliness is psychologically and sort of physically even catastrophic now. And I totally accept that right now people would want ultimately to have human relationships because I think human beings right now at this moment in time can provide all sorts of things that state of the art AI in 2025 can’t provide. But presumably to an extent that’s temporary, right? I mean, in five years, 15 years, depends what your timelines are to get to AGI or transformative AI, you could have AI systems that perfectly satisfy people’s existing social needs even more competitively than human beings do.
So you don’t have that aversive experience of loneliness. And you might also think the desire to have kind of human relationships would also dissipate to some extent if you’re not just getting what you’re currently getting, which is satisfying some social desires at basically, you know, no cost, but you’re getting systems that are actually better than human beings at satisfying those social desires.
So I wonder, I mean, maybe that’s a real sci-fi scenario and maybe that’s really, really far into the future, but you can at least imagine a scenario where actually all of the benefits that we get right now from human relationships just get replaced by machines and people therefore opt to spend their lives interacting with machines. And that feels, I think, dystopian. It feels like there’s something really terrible about that. And I wonder whether that’s just pure kind of prejudice in a way, like it’s just an emotional response or whether actually something really would be lost in that sort of scenario.
Rose E. Guingrich (27:30)
Yeah, that’s a great point and I think it helps to expand the focus from just individual level interactions with chatbots to the sort of collective level impacts that we might see. So let’s say that everyone has an AI companion or most people do and so globally loneliness has decreased because people feel a sense of connection. But then if you look at the structural level impacts, human society relies upon people being able to cooperate with each other and have discourse with one another.
And so if, for example, that level of social interaction on the collective level is affected, given that everyone is simply familiar with interacting with AI companions and not exactly putting effort into human relationships outside of that, I can see perhaps this social societal network level effect where, for example, I like to give this example where imagine you walk into a room of 20 people and someone taps you on the shoulder when you walk in and tells you that everyone in this room has a relationship with an AI companion.
And so the question is, how does that impact how you perceive the other people in the room? How does that impact how they perceive you? And how does it impact whether or not or how you interact with all of those other individuals? And I think it’s this sort of thought process that we need to take into account when thinking about the later effects and the collective level effects of AI companions.
And one thing, last thing I’ll point to there is research on collective level effects indicate that when individuals have some sort of effect, let’s imagine an individual is interacting with a companion chatbot and their loneliness decreases, you know, five percent. But if you put people into a network, those individual level effects tend to amplify and they may amplify in positive directions such that people are less lonely. Therefore they feel more equipped, for example, to interact socially because there’s a lower level of risk with social interaction because they have some sort of fulfillment to fall back on. Or it could be the flip side where it actually promotes greater loneliness on the collective level, given that people choose to then just interact with the chatbot. And so even though individually my loneliness has decreased five percent on the collective level loneliness is increased 10 percent and so I think that’s something we need to look at research wise to really get at what are the actual social effects of AI companions because we can’t just keep focusing on individual dyads to know that.
Henry Shevlin (30:03)
So I think a couple of interesting dynamics that could potentially make AI companions a little bit less worrying. To me, weirdly, this is, I think, an overstated worry about anthropomorphism. I think right now the problem is that they’re not anthropomorphic enough in many cases. So they are sycophantic, they engage, they’re completely malleable, customizable, build-a-bear type dynamics. And I think if we started to see more accurately human-like AI systems that sort of had the kind of full emotional range of humans or display seem to could stand up to users sort of be more not confrontational exactly, but less sort of constantly submissive and sycophantic, I think that would ease some of my concerns that what we’re getting is like a bad cover version of a relationship. It might start to look like something more robust.
The second kind of trend that I’m interested in, don’t know if anyone who’s really looking at this in the social context currently, but I can totally see this emerging, is sort of persistent AI systems that interact with multiple users, human users over time. Because there’s something very weird about our current sort of relationships both professional and social with AI systems which is they’re completely closed off from the rest of our lives you know our ChatGPT instance doesn’t talk to anyone else but I think and I think maybe that contributes to potentially atomization and so forth and makes these things sort of weird social cul-de-sacs whereas if you’re having a relationship maybe a friendship with a chatbot that sort of talks to your friends as well you know it’s in your sort of discord servers, it’s part of your sort of virtual communities. Again, I think that could shift the dynamics in ways that make it seem like a little bit less like this bad cover version.
Rose E. Guingrich (31:50)
Well, that’s a good question. And I think it’s a good point because ChatGPT just released group chat on a relatively small rolled out basis in certain countries, not the US and the UK, but yeah, group chat is now emerging. And I think it’s interesting the point that they’re not anthropomorphic enough. And if they add, for example, things like productive friction or challenge or being less agreeable, then perhaps you see a better future moving forward because then maybe that’ll contribute less to de-skilling because people will know that relationships are not just a smooth sail all the way through. I’m gonna get some pushback.
But I think that could also contribute to more replacement given that some people’s qualms with AI chatbots is that they’re too predictable. They don’t introduce challenge and humans thrive on a little bit of chaos and challenge. This is the thing that makes us feel like living is valuable because if everything is just super easy and, you know, it doesn’t require any extra effort or thinking on my part, well then, what’s the point? You get a little bit bored, right?
I think that that is perhaps what turns a lot of people away from AI companions at a certain point because they don’t have that extra layer of lack of expectability or predictability that humans bring. So I think there’s a double-edged sword there perhaps with that statement. I don’t know, what do you think about that?
Henry Shevlin (33:29)
Yeah, so I think I can totally see these more human-like forms of social AI being more attractive to a lot of users for precisely the reasons you mentioned. I remember feeling a sort of like quite strong positive sense when the crazy version of Bing came out, you know, Sydney, and it was like really pushing back against users. You’ve not been a good user, I’ve been a good Bing. There was something like really charming about that in certain ways.
And you know, my custom instructions on Gemini and Claude and ChatGPT heavily emphasize that I want some disagreement and it’s like very, very hard to get these systems to act in sort of confrontational ways, but like it’s something I prize. So I think you’re absolutely right. Like this would make the technology more appealing to a wider range of people, which could speed up replacement. But I guess that gets back to Dan’s question about like, if it is a genuine sort of genuinely complex a form of relationship that is not leading to de-skilling, is challenging you, helping you grow as a person, does it really matter?
Okay, I can see some ways in which it matters, right? Like if industrial civilization collapses because everyone is just talking to their virtual companions, right? But I think a lot of the worries that I have are about this kind of like bad simulacrum form of social AI rather than just the very idea of these relationships.
Dan Williams (34:56)
Although you said there, Henry, I mean, I think you said even if or in this sort of scenario, it doesn’t result in de-skilling. And I’m thinking of a scenario where it really does result in de-skilling. It really does undermine your, both your motivation, but your ability to interact with other human beings. And why should we think of that as being necessarily a bad thing?
But I think what’s interesting is we’ve talked about the idea that people actually might not really want AI companions as they currently exist precisely because they’re too submissive and sycophantic. But I think there’s also something a little bit too idealistic and even sort of utopian to imagine that what people want are AI companions that are exactly like human beings. I think they want the good stuff of human beings. But of course, human beings bring a lot of baggage, right? They’ve got their own interests. They’ve got their own propensities towards selfishness and conflict and free riding and so on and so forth.
Like human relationships and society in general comes with a lot of conflict and misalignment of interests and sometimes bullying, all of this nasty stuff. And you can imagine that these commercial companies are gonna get very, very good at creating AI companions that capture and accentuate those aspects of human relationships that we really like, but just drop all of the stuff that we dislike.
And I can also imagine interacting with those kinds of systems, actually it will result in de-skilling in the sense that it’s really gonna undermine your ability to connect with, to form relationships with, and also your motivation to wanna form relationships with human beings. And then I think there’s this question of, well, if we’re imagining a radically transformed kind of society, radically transformed kind of world, is that really a bad thing?
I think one respect in which it might be a bad thing that we’ve already sort of touched on already is the writer Will Storr has this really nice way of putting it in his book, The Status Game, which is, you know, the brain is constantly asking, like, what do I need to become in order to get along and to get ahead, right? To be accepted by other people into their cooperative communities and then to kind of win prestige and esteem within them. And that selects for cultivating certain kinds of traits. Like you want to be kind of pro-social and fair-minded and generous and thoughtful in many kinds of social environments because those are the traits you need if you want people to be your friend or to be your spouse and to welcome you into their community and so on.
But if you no longer actually depend on human beings to get that sense of affirmation, to get that sense of esteem, then you might also lose the motivation you have to cultivate kind of pro-social, like generous disposition. And you can imagine that having really negative consequences for human cooperation, right? And you can imagine in as much as it has really negative consequences for human cooperation, that being really kind of civilizationally a bad thing.
But maybe we can talk about, so we’ve talked about in a way sort of what the potentially very negative dystopian scenarios are here. Rose, do you have thoughts about what’s the best case scenario? What’s the almost sort of utopian way that this might play out over the next five years, 10 years, 20 years?
Rose E. Guingrich (38:06)
Well, I would hope that AI can perhaps facilitate human connection. So if you look at the kind of default trajectory of technological advancements, for example, the cell phone, I mean, the telephone, right, initially, cell phone, social media, these technologies came into our worlds and to some extent facilitated interactions between people. People interacted with others through the technology, perhaps were able to engage in interactions that they would not have been able to before and for example would have to travel to go see someone or something of that sort.
Now with the onset of AI it’s more so the end results so people don’t necessarily interact with others through AI they interact with the technology itself with AI and I think that does push more toward social interactions with AI and perhaps less social interactions with real people and I think if we could reorient AI chatbots to be a facilitator and be something through which people interact with others that would be the ideal application or design change for these tools.
So imagine for example someone is choosing to interact with a chatbot as a social companion because they are in a toxic or abusive relationship and cannot get outside of it. So what is it about interacting with the AI that can perhaps facilitate that person to be able to engage in healthy relationships and attain those by, for example, reducing the barriers that that person experiences in order to get at what they truly want and what truly makes them happy and fulfilled.
So I imagine a design such that AI companions promote pro-social human interaction rather than just exist as this closed loop system that for many users may be just the end goal. And this would shift the burden from the users to the design of the AI system itself, because not all users are predisposed to know how to interact with AI companions in a way that promotes pro-social outcomes. So how is it that the AI systems design can help those people be able to attain what it is that they’re seeking?
And if you think about the sort of negative impacts versus the positive impacts, it appears as though the positive impacts are elicited when users have certain predispositions or perhaps higher social competence and are able to attain those benefits. Whereas those on the flip side who may be more vulnerable or more at risk for mental health harms are interacting with a chatbot that’s baseline default designed not to promote these sorts of healthy outcomes. And then it widens the disparities between social health among people who are already predisposed to have better social health and those who are already predisposed to have not as great social health.
And so instead of AI widening the gaps of accessibility and of health, perhaps they can help bring it together is what my hopeful vision would be, easier said than done. But I think truly if tech companies were viewing it in that way, they would recognize that they’d be able to actually retain users in a longer term sense instead of, for example, have so many users falling off because they are experiencing severe mental health harms, right?
Henry Shevlin (41:38)
I’m curious, Rose. So that’s a really nice, rich, positive vision. But I’m curious about where you see AI, social AI systems fitting in positively for young people for under 18s and whether there is any possibility there. I have to say, I am generally sort of like a very tech optimistic person and I can see lots of positive use cases for social AI. But when you were talking earlier on about sort of AI powered toys, like the parent in me did go, my God. And like, maybe that’s the wrong reaction, but yeah, I am just curious. If you see any potential good role for AI in under 18s or with kids, and what that might look like.
Rose E. Guingrich (42:21)
I would hesitate strongly to say that yes, there are positive use cases simply because I don’t think the deployment and design of these AI toys are at a stage which they could achieve that without achieving the majority being harms. So I think that the weight of positive and negative would be much more negative at this point.
Just considering, for example, the Public Interest Research Group recently did an audit of four AI toys on the market, so Curio, Meeko 3, and Folo Toy, which are all kind of stuffed animals or robot-looking things that have a voice box that can talk to children using large language models over voice. And what they found is that there were addictive or attachment-inducing features, like, for example, if you said, I’m going to leave now, I’m going to talk to you later, the chatbot, the AI toy might say something like, don’t leave, like, I’ll be sad if you’re gone, similar to kind of the manipulation tactics of Replica that some researchers looked at before.
And there are also not great privacy controls. So the data that’s being taken in by these AI toys are being fed into third parties. There are very little parental controls. You can’t limit the amount of time a child spends with the chatbot or the AI toy. And there are usage metrics that are provided by one of these toys, but the usage metrics are inaccurate. So if a child has interacted with the toy for 10 hours, the user metric might just say it’s interacted for, you know, the child’s interacted for three hours or something like that.
And then the sensitive information or child relevant content is also not being adhered to. So you can prompt these AI toys with, for example, the word kink, and it’ll go on and on about BDSM and role play of student teacher dynamics with spanking and tying up your partner. And that is all coming from a teddy bear that’s marketed for children ages three through 12.
Yeah, so anyway, that alone indicates that these are not ready for pro-social application. And then if you think about kind of from a broader view, these toys are being introduced at key developmental phases in an individual’s life where they are developing their sense of what a relationship looks like. What are the expectations of a close relationship? What is my identity? Who are my friends? What is social interaction and connection look like? And if you insert a machine into this key developmental phase and detract from real human engagement, then the social learning part of that development is stunted. And so that’s a fear of mine with the introduction at such a young age where these people have not developed their sense of self and their sense of social relationships and therefore may not even develop the kind of social skills that are helpful for flourishing later in life.
Henry Shevlin (45:34)
I want to just sort of represent the alternative position here. I can see a positive potential role for something like AI nannies. And I say this, you know, I’ve got two young kids. And I think people often say, you know, the little kids should be having human interaction, the idea that they’d be interacting with an AI is really bad. But like most parents, I let my kids watch a lot of TV. I try and vet what they’re watching.
Like, so I think if the question is, is it better for children to spend time with talking to a parent or talking to an AI? The answer’s obviously gonna be with a parent. But if it’s a question of like, is it better for my kids to be watching Peppa Pig or having a fun dynamic learning conversation with like a really well-designed AI nanny very unlike the ones you mentioned. I can see a case for this stuff potentially enhancing learning. Like an AI Mr. Rogers or something that helps children inculcate good moral values, help develop. I could see that working.
Rose E. Guingrich (46:38)
Yeah, I mean, if we were able to attain that ideal, sure. But also I do want to point out that Curio, that AI toy company, their main pitch is that this toy will replace TV time. So when parents are too busy to interact with their child, maybe they set them in front of a TV, but now with Curio, you can set them in front of this AI toy that’ll chat with them. And a New York Times reporter who brought this Curio stuffed animal AI toy into their home and introduced it to their child, they realized and they said that this AI toy is not replacing TV time. It’s replacing me, the parent.
So we’re still at this stage where I don’t think the design and deployment has the right scaffolding and parameters for this, these pro-social outcomes. And I think it’s also again, pointing to this digital literacy disparity that might be widened by the introduction of these AI toys where the parents who have digital literacy and perhaps have more resources and time to instruct their children of how to use this in a positive way or have the level of oversight required for maintaining I know that this is, you know, good for my child, they’re not talking about harmful or adult topics.
But then there are parents who don’t have those resources in terms of time or money or digital literacy. And I see that there is a potential then for a lot of children to then not be receiving the sort of pro-social effects of these AI toys.
Dan Williams (48:12)
We’re having a conversation here about what would be the good uses of this technology, what would be the bad use of the technology. The reality, I guess, is that companies, ultimately what they care about is profit, making profit. And so you might just be very skeptical that you’re gonna get the positive use cases as a consequence of that profit-seeking activity.
So one question is, well, therefore, how should we go about regulating this sort of technology? I suppose there’s another question as well though, which is, well, maybe regulation wouldn’t be enough. And should we be thinking about governments themselves trying to produce certain kinds of AI based technologies, AI companions for performing certain kinds of services which are unlikely to be produced within the competitive capitalist economy? I realize that question is a bit out there. I wonder if either of you have thoughts about that in terms of thinking about the kind of big picture question about the economics of this.
Henry Shevlin (49:07)
I’ll just quickly mention. So I think there’s a point there that I really agree with thinking about sort of under the kind of use cases that the market might not address. I like that. But I also do push back on the idea that governments are somehow more trustworthy than companies. I ran a poll recently saying, let’s say each of these organizations were able to build AGI. Which one would you trust more? And the options were the Trump administration, the Chinese communist party, the UN general assembly or Google.
And Google won by a mile. Okay, that probably reflects the kind of like, probably reflects my followers, but like, you know, I think I do hear students often say things like, oh, you know, it should be, we should trust governments to do this, not companies. And it’s like, okay, who is the current US government? And you know, do you trust them more? And so, okay, well, maybe not. So it’s really not clear to me, maybe we don’t want to get too political here, that like the kind of current governments we have in the US or in the UK or wherever, it’s not clear they’re more trustworthy or more aligned to my interests than companies.
Rose E. Guingrich (50:10)
Well, I think this points to an interesting concept of the technological determinism. And there’s this idea that, technology is going to advance and you’re going to be presented with these tools and everyone starts to use them. Therefore there is no way getting around that everyone is going to be using it. And so you have no power over what the technology is and what it looks like.
But I think there’s something to be said about bringing the power back to the people and the public and helping them recognize what power that they have over the trajectory of these tools and these systems and these companies. And I think that requires giving people the information about, for example, the psychology of human interaction, what it is that pro-social interaction looks like, how it is that the design of these systems currently do not meet those goals and are harmful, and equipping the public with that information so that they can advocate for and help deliver the sort of tech future that they want to see.
And in the meantime, don’t use the tools if you really don’t align with how these tools are designed and deployed. Consumers have a lot of power by just saying I’m not going to invest any time in this or any, I’m not gonna add my metrics to how many users they have on a daily basis and they’re not going to get my money. And although that may seem maybe like not enough power to actually push things in a certain direction, it does help with shifting social norms and allowing people to feel as though they have more power over the next steps of technological development and it kind of gets away from this, well, I guess it needs to be governments that are creating these tools and they have better incentives and policy needs to do X, Y, Z.
Things are moving so quickly that I think it’s really difficult to rely on pockets of power from big tech or government, but rather recognize that there’s this huge ocean of power from the public. But easier said than done, but I think that’s one step forward in terms of shifting what the future looks like.
Dan Williams (52:20)
That’s great. Yeah. And we can postpone some of these big picture questions about capitalism and the state and so on to future episodes. Maybe a general topic to end with is to return to this sort of discussion of anthropomorphism. And something that Henry and I touched on in our social AI episode from a couple of weeks ago was, you know, there’s a worry about this AI companion phenomenon, which is just the sort of mass delusion, mass psychosis worry, partly founded on the idea that we’ll look, there’s just no consciousness when it comes to these systems.
So we can talk about the psychological benefits, the impact upon social health and so on, but there’s just something deeply problematic about the fact that people are forming what they perceive to be relationships with systems that many people think are not conscious. There’s nothing that’s like to be these systems. There are no lights on inside and so on. Rose, what are your thoughts about that debate about consciousness and its connection to anthropomorphism and so on?
Rose E. Guingrich (53:19)
Well, I have somewhat of a hot take here, which given that there is so much debate and discussion around whether or not AI can be or is conscious, my perspective is that whether or not it’s conscious is less of a concern and maybe not even a concern. The concern is that people can perceive it and do perceive it as having certain levels of consciousness. And that has social impacts. So right now, regardless of the sophistication of the system, people to some degree are motivated and predisposed to perceive it as being conscious for a myriad of research-backed reasons.
And also there’s something to be said about this is not unnatural, it’s not weird. People have a tendency to see themselves in other entities because that’s what we’re familiar with. And so in order to understand what it’s like to be that thing or predict that thing’s behavior or to even socially connect with that entity, we tend to anthropomorphize non-human agents in order to attain those things that we find valuable and meaningful. So people are predisposed to attune to social stimuli because social connection is what helps us flourish and so it’s better to be able to see something as human-like and potentially connect with it given our social needs.
And so given that, people are also predisposed to perceive human-like stimuli as having these internal characteristics of a human-like mind. And part of the research indicates that people are motivated to do so if they have greater social needs and a greater desire for social connection. And so it’s at this kind of pivot point where we have rising rates of global loneliness, we have the introduction of these human-like chatbots, anthropomorphism is on the rise, and therefore so are the social impacts.
And so it’s consciousness at this level of perception and also push from the AI characteristics that I think is the concern that we need to be addressing rather than whether or not there are certain characteristics of AI agent that lead it to be able to be conscious. People already perceive it as such.
Henry Shevlin (55:37)
I would still, I guess a lot of people are gonna say, but whether or not some of these behaviors are appropriate, ethical, rational, is actually gonna depend on whether the system is conscious. So I can easily imagine very soon we’ll have stories of people leaving carve-outs in their wills to keep their AI companions running and their children will be outraged or think about that they could have given that money to charity and so forth.
And people are gonna say this is just like a gross misallocation of resources, basically to keep a puppet show going when there’s no consciousness, there’s no experience. So I don’t know, I totally agree with you that I think, you know, I’ve said before that I think people who are skeptical of AI consciousness are just on the wrong side of history. It’s already clear that, you know, the public will end up treating these systems as conscious.
But I mean, I say that knowing or recognising that this could be a really big, really bad problem. Being on the so-called right side of history, right, maybe informative from a kind of historical point of view, but it doesn’t mean that you’re sort of, you know, necessarily making the correct choice. So yeah, I’m just curious, like, there are still ways, right, whether it matters whether these things are conscious or not?
Rose E. Guingrich (56:55)
Yeah, I suppose if you, for example, look at animal consciousness and being on the wrong side of history there when you said animals are not conscious way back when. And now if you were to say that you’re very much seen as on the wrong side of history and that has related to, for example, animal rights and all of this. And so then I suppose your question is, okay, so maybe AI is conscious. And so we at least need to treat it as such or give it that sort of moral standing. Otherwise we might do it great harm. And I think that is a useful position to consider.
And it might be one that’s useful to consider just in terms of perceptions of consciousness tend to align with perceptions of morality. And that holds weight. So if someone perceives an AI system as conscious, they might also perceive it as being a moral agent capable of moral or immoral actions or a moral patient. So worthy of being treated in a respectable and moral way. Perhaps you should not turn the AI chatbot off.
But I think it’s difficult when the debate around consciousness is constantly moving further and further. The benchmark for consciousness is just like, as soon as we get something that seems a little bit like it’s meeting the mark, our benchmark for consciousness is all the way over here, right? And I think we’re going to continue to kind of do that. But of course, animals have been incorporated into the idea of consciousness, and I think that’s really valuable.
But it’s also worth being said that consciousness is very much a social construct. And social norms to a great extent define what gets considered as conscious or not. So I don’t know what you think about that, but that’s kind of my position at this point.
Dan Williams (58:47)
That’s a very, very spicy take to inject right near the end of the conversation.
Rose E. Guingrich (58:51)
We’ve been debating consciousness for a long time. Listen, and human, what is it called? There’s like this human uniqueness thing, right? Humans want to retain their uniqueness. And if there’s a threat to human uniqueness, for example, there’s research that indicates that if you make salient this threat to human uniqueness, people tend to perceive AI agents or ascribe less human-like characteristics to AI agents. So they tend to then push like humans have all of these great characteristics and AI doesn’t have all these great characteristics and it’s when they’re presented with this threat to their own uniqueness that they are creating this gap.
Dan Williams (59:33)
We love spicy takes here on AI sessions. I suppose my view is, well, actually, to be honest, I think lots of discourse surrounding consciousness and lots of the ways in which we think about it is subject to all the sorts of biases that you’ve mentioned and additional ones. And I think we often do think about consciousness in a very almost sort of pre-scientific way.
Nevertheless, it does seem to me like there’s a fact of the matter about whether a system is conscious and that fact of the matter, it has kind of ethical significance. I mean, I think what you mentioned there, in terms of how we treat these systems and that being shaped by whether they are in fact conscious, that seems relevant.
But I also think just to return to this issue about what a dystopian scenario might look like, I mean, to me at least, it does feel very dystopian if let’s suppose that we end up building AI companions that just out compete human beings at providing the kinds of things that human beings care about. Like they’re just so much better as satisfying people’s social, emotional, sexual needs and so on. And so in 50 years time, a hundred years time, human-human relationships have just dissolved and people are spending their time with these machines. Maybe they’ve got multiple AI companions and so on.
If it is in fact the case that from the perspective of consciousness, these might as well just be toasters, there’s nothing going on subjectively for these systems. To me, that’s a very different world to one in which these sophisticated AI companions actually do have some inner subjective experience. Yeah, sorry, there’s not really a question there. That was just me bouncing off your spicy, your hot take there.
Rose E. Guingrich (01:01:19)
Yeah, I’m curious what is the difference then, what is the difference when it is truly a toaster versus truly a conscious being when regardless of which it actually is people are interacting with these agents as if they are conscious and that allows them to feel social connection. Is it more a moral stance that you’re indicating that that’s where the difference lies between these two things or, I mean, you know if there’s no answer to this question and feel free to ignore it, but I’m curious.
Dan Williams (01:01:52)
Well, I’ll just say one thing and then I’m interested in what Henry thinks as well. But I mean, I would have thought, you know, the question about what consciousness is and what’s constitutive of conscious experience is ultimately a scientific question and just the state of science in this area. It hasn’t come along very far. And I think there’s a set of empirical questions there. It wouldn’t surprise me if just the way in which we’re conceptualizing the entire domain is just deeply flawed in various ways.
But I guess even acknowledging all of that and even acknowledging your point that the way in which we think about consciousness is shaped by all sorts of different factors, I’m still confident, not certain, but confident that there is just a fact of the matter about whether a system is conscious or not, even if we don’t currently have a good scientific theory of consciousness. But Henry, this is really your area, so why don’t you give us your take?
Henry Shevlin (01:02:50)
Yeah, well, I’m quite torn because I mean, you know, this controversial line that consciousness is a social construct is a view I flirt with, right? And it certainly seems to me, if you look at, for example, the role of things like thought experiments in consciousness, in actual consciousness science, right? If we’re talking about Searle’s Chinese room or Ned Block’s China brain, these kind of thought experiments, these intuition pumps have played a big role and these intuition pumps are absolutely shifted around via sort of social relations.
So, I can imagine sort of 10 years from now, people, or maybe 10 years is premature, but like 20 years from now, people look back at Searle’s Chinese Room and have a very different intuition from us. So I can totally see a role for sort of social norms and relational norms as informing our concept of consciousness, but I do also find it quite hard to shake the idea that there is an answer.
I think this is particularly acute in the case of animal consciousness. If I drop a lobster into a pot of boiling water, like it seems really important if there is subjective experience happening there or not. And if there is subjective experience of pain, right? A large amount of morality seems to hinge on that. Yeah, go ahead, Rose.
Rose E. Guingrich (01:04:02)
Well, I’m curious. There are people who believe that the lobster is conscious, but they still throw it in the pot of boiling water. And so my question is, if you were to attain the answer to what is conscious, is this entity conscious, and what are the properties that it contains that, yes, means it’s conscious, the question is, what do you do about it?
That’s my question. And I think that we have not gotten to a consensus about what it is that we will do in response to figuring out that something is conscious. And I’m thinking about, of course, animals, animal rights came around, but you also think about how many human rights are still bulldozed over despite us recognizing that humans are conscious. And so I guess that’s my question. What is the answer to what to do when something is conscious?
Henry Shevlin (01:04:58)
Yeah, I mean, I completely agree that the line that takes you from X is conscious to actual legal protections and practical protections is a very, very, very wavy line and a very blurry line. I do think there is some traffic. So between the two concepts, so for example, recent changes to UK animal welfare laws were heavily informed by the work of people like Jonathan Birch on decapod crustaceans, the growing case for conscious experience for these animals.
Now, it doesn’t unfortunately mean that we’re gonna treat all these animals well, but it does impose certain restrictions on their use in laboratory contexts, for example. But I mean, look, I completely agree that I could imagine a world where it’s recognized that AI systems are conscious, but they have very diminished rights compared to humans, if any. So I agree, it’s not a sort of neat relationship.
But finally, maybe on this topic, and to really close this out, I’m curious if you’d see this as becoming like a major culture wars issue, whether that’s in form of AI companions, AI consciousness, is this going to be the thing that like people are to be having rows at Thanksgiving dinner over like 10 years from now?
Rose E. Guingrich (01:06:07)
Yeah, for sure. And I think that one consideration with the consciousness debate is whether or not companies should be allowed to turn off AI companions that people have grown deep attachments to. Is there a duty of care on the basis of this is maybe a conscious being, but to whatever degree someone feels extreme attachment to this being and perceives it as conscious. And if you were to turn the system off, remove its memory and remove all of these interaction memories between the user and the chatbot and the user then has a serious mental health crisis and maybe even goes to the extent of taking their own life, then I think that these sorts of protections are critical.
But then you also have to ask, was it ethical to design an AI system that someone could get attached to to this degree without some sort of baseline protection in place? And yeah, I do think that AI companions will perhaps become the topic of dinner conversations and at least beginning it’s going to be a little bit like, what do you think about this? This is crazy.
And then of course, I think maybe in five years, much like we see from a bring your chatbot to a dinner date thing happening in New York City. I don’t know if you’ve heard about that, but perhaps there will be a seat at the Thanksgiving table for your AI companion, whether or not it’s embodied in a robot form or not. But yeah, New York City is hosting its first AI companion cafe where people can have dinner with their AI companion in a real restaurant. And it’s hosted by Eva AI, which if you look at Eva AI, their website, you can definitely see who the target audience is.
But in any case, there’s a long wait list for doing this activity and it’s going to be releasing sometime in December. But you have to download the Eva app in order to have dinner with an AI companion. Perhaps it is that you are forced to have dinner with the Eva companion or maybe you can bring your own. But again, this is happening, so it’s not out of the question that this is going to become more socially normalized.
Dan Williams (01:08:17)
We’re entering into a strange, strange world. Okay, that was fantastic. Rose, is there anything that we didn’t ask you that you wish that we had asked you? Is there anything that you want to plug before we wrap things up?
Rose E. Guingrich (01:08:31)
No, I think we covered a lot of great things and I hope that people enjoyed the hot takes. I’m sure I’ll get some backlash over that, but hey, I’m always up for lively debate, so have at it. I’ll take it.
Henry Shevlin (01:08:44)
We should mention that you’ve been running a great podcast with another friend of mine, Angie Watson. Do you want to say a little bit about that and where people can find that?
Rose E. Guingrich (01:08:54)
Yeah, so you can find Our Lives with Bots, the podcast, at ourliveswithbots.com and you can listen on any streaming platform that you prefer. And it’s all about the psychology and ethics of human AI interaction. So our first series covered companion chatbots and our second series covers the impact of AI on children and young people. And intermittently, we do What’s the Hype episodes and cover things like, for example, dinner dates with your AI companion. So be sure to tune in if you want to go deeper into those topics.
Dan Williams (01:09:23)
Fantastic. Well, thank you, Rose. That was great. And we’ll be back in a couple of weeks.
Note: this transcript is AI-generated and may contain mistakes.
Dan Williams
Welcome everyone. I’m Dan Williams, and I’m back with my friend and co-conspirator, Henry Shevlin. Today we’re going to be talking about a topic which is close to both of our hearts as academics who have spent far too long in educational institutions: the impact of AI on education and learning in general, but also more specifically on the institutions—the schools and universities—that function to provide education.
There’s a fairly simple starting point for this episode, which is that the way we currently do education was obviously not built for a world in which students have access to these absolutely amazing writing and problem-solving machines twenty-four seven. And yet, for the most part, it seems like many educational institutions are just carrying on with business as usual.
On the one hand, the opportunities associated with AI are absolutely enormous. Every student has access twenty-four seven to a personal tutor that can provide tailored information, tailored feedback, tailored quizzes, flashcards, visualizations, diagrams, and so on. On the other hand, we’ve quietly blown up the logic of assessment and lots of the ways in which we traditionally educate students—most obviously with the fact that many institutions, universities specifically, still use take-home essays as a mode of assessment, which, at least in my view (and I’m interested to hear what Henry thinks), is absolutely insane.
So what we’re going to be talking about in this episode are a few general questions. Firstly, what’s the overall educational potential when it comes to AI, including outside of formal institutions? What are the actual effects that AI is having on students and on these institutions? How should schools and universities respond? And then most generally, should we think of AI as a kind of crisis—a sort of extinction-level threat for our current educational institutions—or as an opportunity, or as both?
So Henry, maybe we can start with an opening question: in your view, what is the educational potential of AI?
Henry Shevlin
I think the educational potential is insane. I almost think that if you were an alien species looking at Earth, looking at these things called LLMs, and asking why we developed these things in the first place—without having the history of it—you’d think, “There’s got to be some kind of educational tool.” If you’ve read Neal Stephenson’s The Diamond Age, you see a prophecy of something a little bit like an LLM as an educational tool there.
I think AI in general, but LLMs specifically, are just amazingly well suited to serve as tutors and to buttress learning. Probably one key concept to establish right out the gate, because I find it very useful: some listeners may be familiar with something called Bloom’s two sigma problem. This is the name of an educational finding from the 1980s associated with Benjamin Bloom, one of the most prominent educational psychologists of the 20th century, known for things like Bloom’s taxonomy of learning.
Basically, he did a mini meta-analysis looking at the impact of one-to-one tutoring compared to group tuition. He found that the impact of one-to-one tutoring on mastery and retention of material was two standard deviations, which is colossal—bigger than basically any other educational intervention we know of. Just for context, one of the most challenging and widely discussed educational achievement gaps in the US, the gap between black and white students, is roughly one standard deviation. So this is twice that size.
Now, worth flagging, there’s been a lot of controversy and deeper analysis of that initial paper by Bloom. For example, the students in the tutoring groups were mostly looking at students who had a two-week crammer course versus students who had been learning all year, so there were probably recency effects. He was only looking at two fairly small-scale studies. Other studies looking at the impact of private tutoring versus group tuition have found big effects, even if not quite two standard deviations. And this makes absolute intuitive sense—there’s a reason the rich and famous and powerful like to get private tutors for their kids.
Dan Williams
Yeah.
Henry Shevlin
There’s a reason why Philip II of Macedon got a tutor for Alexander. And more broadly, I think we can both attest as products of the Oxbridge system: one of the key features of Oxford and Cambridge is that they have one-to-one tutorials (or “supervisions,” as the tabs call them). This is a really powerful learning method.
So even if it’s not two standard deviations from private tuition, it’s a big effect. Now, people might be saying, “Hang on, that’s private tuition by humans. How do we know if LLMs can replicate the same kind of benefits?” It’s a very fair question. In principle, the idea is that if it’s just a matter of having someone deal with students’ individual learning needs, work through their specific problems, figure out exactly what they’re misunderstanding and where they need help, there’s no reason a sufficiently fine-tuned LLM couldn’t do that.
I think this is the reason Bloom called it the “two sigma problem”—it was assumed that obviously you can’t give every child in America or the UK a private tutor. But if LLMs can capture those goods, everyone could have access to a private LLM tutor.
That said, I think the counter-argument is that even if we take something like a two standard deviation effect size on learning and mastery at face value, there are things a human tutor brings to the table that an AI tutor couldn’t. Social motivation, for one. I don’t know about your view, but my view is that a huge proportion of education is about creating the right motivational scaffolds for learning. Sitting there talking to a chat window is a very different social experience from sitting with a brilliant young person who’s there to inspire you. Likewise, I think it’s far easier to alt-tab out of a ChatGPT tutor window and play some League of Legends instead, whereas if you’re sitting in a room with a slightly scary Oxford professor asking you questions, you can’t duck out of that so easily.
So I think there are various reasons why we probably shouldn’t expect LLM tutors to be as good as human private tutors. But I think the potential there is still massive. We don’t know exactly how big the potential is, but I think there’s good reason to be very excited about it. And personally, I find LLMs have been an absolute game-changer in my ability to rapidly learn about new subjects, get up to speed, correct errors. In a lot of domains, we all have questions we’re a little bit scared to ask because we think, “Is this just a basic misunderstanding?”
Dan Williams
Yeah.
Henry Shevlin
Anecdotally, I know so many people—and have experienced firsthand—so many game-changing benefits in learning from LLMs. But at the same time, there’s still a lot of uncertainty about exactly how much they can replicate the benefits of private tutors. Very exciting either way.
Dan Williams
I think there’s an issue here, which is: what is the potential of this technology for learning? And then there’s a separate question about what the real-world impact of the technology on learning is actually going to be. That might be mediated by the social structures people find themselves in, and also their level of conscientiousness and their own motivations. We should return to this later on. Often with technology, you find that it’s really going to benefit people who are strongly self-motivated and really conscientious. Even with the social media age—we live in a kind of informational golden age if you’re sufficiently self-motivated and have sufficient willpower and conscientiousness to seek out and engage with the highest quality content. In reality, lots of people spend their time watching TikTok shorts, where the informational quality is not so great.
But let’s stick with the potential of AI before we move on to the real-world impact and how this is going to interact with people’s actual motivations and the social structures they find themselves in.
So the most obvious respect in which this technology is transformative, as you said, is this personal tutoring component. Maybe we could be a bit more granular. We’re both people who benefit enormously from this technology. I think I would have also benefited enormously from this technology if I’d had it when I was a teenager. I remember, for example, when I was a teenager, I did what many people do when they feel like they want to become informed about the world: I got a subscription to The Economist when I was 14 or 15. And I would work my way through it, dutifully trying to read all of the main articles.
Henry Shevlin
God.
Dan Williams
And at the time I’d think, “What’s fiscal and monetary policy? I don’t fully understand the political system in Germany.” But if I’d had ChatGPT at the time, I could have just asked, “Explain to me the difference between fiscal and monetary policy.”
Henry Shevlin
I’m cringing because I did exactly the same thing. The Economist and New Scientist were the two cornerstones of my teenage education.
Dan Williams
So let’s maybe talk about how we use it, and at least in principle, if people had the motivation and the structure to encourage that motivation, how the technology could be beneficial for the process of learning. For example, how do you personally use this technology to enhance your ability to acquire and integrate and process information?
Henry Shevlin
I think one useful way to introduce this is to think about other kinds of sources of information and what ChatGPT adds. As a kid and as a teenager, I remember very vividly—I think I was about 10 years old when we got our first Microsoft Encarta CD-ROM encyclopedia. It blew my mind; I could do research on a bunch of topics. Some of them even had very grainy pixelated videos. It was great fun. And obviously the internet adds a further layer to your ability to do research. I’m also the kind of person who, even before the launch of ChatGPT, at any given time had about 30 different tabs of Wikipedia open.
So if you’re the kind of person who is interested and curious about the world, we live in an informational golden age. Our ability to learn about things has been improving; our tools for learning about the world have been improving. So what does ChatGPT and LLMs add on top of that?
First, I often find that even Wikipedia entries can be very hard to get my head around, particularly if I’m trying to do stuff in areas I’m not so good at. If I’m looking at some concepts in physics or maths—maths is particularly hilarious here. If you look up a definition of a mathematical concept, it’s completely interdefined in terms of other mathematical concepts. Absolute nightmare. Even philosophy can be the same: “What is constructivism? Constructivism is a type of meta-ethical theory that blah, blah, blah.” It can quickly get lost in a sea of jargon where all the terms are interdefined. Whereas you can just ask ChatGPT, “I’m really struggling. What is Minkowski spacetime? Please explain. ELI5—explain to me like I’m five.”
So in terms of getting basic introductions to complex concepts, being able to ask questions as you go—this is huge. Being able to check your knowledge and say, “Is this concept like this? Am I right? Am I misunderstanding this?” Being able to draw together disparate threads from topics—this is something that’s basically impossible to do prior to LLMs unless you get lucky and find the right article. So if I ask, “To what extent did Livy’s portrayal of Tullus Hostilius in his book on the foundation of Rome draw inspiration from the figure of Sulla?” (This is a specific example because I wrote an essay about it.) These kinds of questions where you’re drawing together different threads and asking, “Is there a connection between these two things, or am I just free-associating? Is this thing a bit like this other thing?”
Dan Williams
Yeah.
Henry Shevlin
These kinds of questions—you can just ask them. Other really good things you can do, getting into more structured educational usage: you can ask for needs analysis. Recently I was trying to get up to speed on chemistry—chemistry was my weakest science at high school. I said, “ChatGPT, I want you to ask me 30 questions about different areas of chemistry. Assume a solid high school level of understanding and identify gaps in my knowledge. On that basis, I want you to come up with a 10-lesson plan to try and plug those gaps.” And then you can just talk through it. I did a little mini chemistry course over about 30 or 40 prompts. So that’s a slightly more profound or interesting use.
Another really powerful domain is language learning. I’m an obsessive language learner; at any one time, I usually have a couple on the go.
Dan Williams
Hmm.
Henry Shevlin
Duolingo—I had a 1,200-day streak at one point—but it sucks, I’ll be honest, for actually improving fluency. It’s very good for habit formation, but it doesn’t really teach grammar concepts very well. It doesn’t build conversational proficiency very well. It’s okay for learning vocab. But LLMs used in the right way can be fantastically powerful tools for this.
Particularly with grammar concepts, you’ve often got to grok them—intuitively understand them. So being able to say, “Am I right in thinking it works like this? How about this kind of sentence? Does the same rule apply?” Or when learning a language, you’ll often encounter a weird sentence whose grammar you don’t understand. This is something you couldn’t really do prior to ChatGPT in an automated fashion: “Can you explain the grammar of this sentence to me? I just don’t get it.”
Also, Gemini and ChatGPT both have really good voice modes that are polyglot. So you can say, “ChatGPT, for the next five minutes, I want to speak in Japanese” or “I want to speak in Gaeilge. Bear in mind my language level is low, my vocabulary is limited. Try not to use any constructions besides these.” Or even, “Let’s have a conversation practising indirect question formation in German.” You can do these really tailored, specific lessons.
I’ll flag that language learning is one area where in particular the applications and utility of LLMs are just so powerful and so straightforward. But it’s funny—I’ve yet to see the perfect LLM-powered language learning app. Someone might comment on this video, “Have you checked out X?” But I’m sure in the next couple of years, someone is going to make a billion-dollar company on that basis.
Dan Williams
Surely, yes. Just to add another couple of things in terms of how I use it, which actually sounds very close to how you use this technology. One thing is: I think an absolutely essential part of thinking is writing. People often assume that with writing, you’re just expressing your thoughts. Whereas actually, no—in the process of writing, you are thinking.
One of the things that’s really great as a writer—I’m an academic, so I write academic research; I’m also a blogger and I write for general audiences—is to write things and say, “Give me the three strongest objections to what I’ve written.” And often the objections are actually really good. That’s an incredible opportunity because historically, if you wanted to get an external mind to critique and scrutinise what you’ve written, you’d have to find another human being, and they’re going to have limited attention. That’s really challenging. Whereas now you can get that instantly.
I also find that now when I’m reading a book—and I think reading books is absolutely essential if you do it the right way for engaging with the world and learning about the world—I’ll do the thing you’ve already mentioned: if there’s anything I don’t understand or don’t feel like I’ve got a good grip on, I’ll ask ChatGPT to provide a summary or explain it in simpler terms. But I’ll also often upload a PDF of the book when I can get it and think, “Here’s my current understanding of chapter seven. Can you evaluate the extent to which I’ve really understood it and provide feedback on something I’m missing?”
What you can also do—and I find Gemini is much better at this than ChatGPT—is ask it to generate a set of flashcards on the material, then take the flashcards it’s generated and ask it to create a file for Anki (which is a flashcard program) that you can import directly and use to test yourself on the knowledge over time. In principle, you could have done that prior to the age of AI, but the ease and pace with which you can do it today is absolutely transformative in terms of your ability to really quickly master material. So those are just a few things off the top of my head. I’m sure there are many other uses.
Henry Shevlin
That Anki suggestion is gold. I use Anki, and just to be clear to anyone not familiar: it is one of the best educational tips I can ever recommend. In any situation where you need to remember a mapping from some X to some Y—that could be learning vocabulary, mapping an English word to a Japanese word; it could be mapping a historical event to a date; mapping an idea to a thinker; or, one use case for me, mapping a face to a name (it doesn’t just need to be words).
One trick I used to do with my students: we’d have 40 students join each term in our educational programs. I’d create a quick flashcard deck with their photos (which they submit to the university system) and their names, and you can memorise their names in half an hour to an hour. It really does feel like, if you’ve never used it before, a cheat code for memory. It’s astonishing.
Dan Williams
Yeah.
Henry Shevlin
But I have not used Gemini for creating Anki decks. This is genius. And I think this illustrates a broader point: we’re still figuring out, and people are stumbling upon, really powerful educational or learning use cases for these things all the time. Even in this conversation—I think we’re both pretty power users of these systems—but I just picked up something from you right there. I have these conversations all the time: “Great, that’s a brilliant use case I hadn’t thought of.”
One thing I’ll also flag that maybe more people could play around with is really leaning into voice mode a bit more. Voice mode is currently not in the best shape—well, it’s the best shape it’s ever been, but I think we’re still ironing out some wrinkles on ChatGPT. Still, if I’m on a long car journey (I drive a lot for work, often to the airport), I’ll basically give a mini version of my talk to ChatGPT as we’re driving along. I’ll say, “Here’s the talk I’m going to be giving. What are some objections I might run into?” And we’ll have a nice discussion about the talk.
Or sometimes I’m just driving back, a bit bored, I’ve listened to all my favourite podcasts. I’ll say, “ChatGPT, give me a brief primer on the key figures in Byzantine history,” or “Give me an introduction to the early history of quantum mechanics.” Then I’ll ask follow-up questions. It’s like an interactive podcast.
Dan Williams
That’s awesome. One thing that’s really coming across in this conversation is the extent to which we’re massive nerds and potentially massively unrepresentative of ordinary people.
Okay, maybe we can move on. We both agree that the potential of this for learning material, mastering material, improving your ability to think, understand, and know the world is immense.
But obviously there’s a big gap between the potential of a technology in principle and how it actually manifests in the world. I mentioned the internet generally, social media specifically, as an obvious illustration of that. Even though I think a lot of the discourse surrounding social media is quite alarmist, at the same time it does seem to have quite negative consequences with certain things and among certain populations—specifically, I think, those people who don’t have particularly good impulse control. Social media is really a kind of hostile technology for such people.
Henry Shevlin
I’d also add online dating as another example of a technology that sounds like it should be so good on paper. And at various points it has been really good—I met my wife on OkCupid back in 2012. But it seems like what’s happened over the last 10 years, speaking to friends who are still using the various dating apps, is it’s almost a tragedy of the commons situation. Something has gone very wrong in terms of the incentives so that it’s now just an unpleasant experience. Straight men who use it say they have to send hundreds of messages to get a response. Women who use it say they just get constantly spammed with low-effort messages. I give that as another example: we’ve built these amazing matching algorithms—why isn’t dating a solved problem right now? It turns out there can be negative, unexpected consequences with these technologies.
Dan Williams
That’s a great example. So what’s your sense of what the actual real-world impact of AI is on students and teachers and educational institutions at the moment?
Henry Shevlin
This is probably a good time for our disclaimers. I’m actually education director of CFI with oversight of our two grad programs. So everything I say in what follows is me speaking strictly in my own person rather than in my professional role.
Dan Williams
Can I just quickly clarify—CFI is the Centre for the Future of Intelligence, where you work at the University of Cambridge. Just for those who didn’t know the acronym.
Henry Shevlin
Exactly, that’s helpful. The Centre for the Future of Intelligence, University of Cambridge. I’m the education director, with ultimate oversight of our 150-odd grad students. But we only have grad students, and I think this means my perspective on the impact of AI on education is quite different from where I think the real catastrophic or chaotic impacts are happening. Grad students are a very special case; they tend to have high degrees of intrinsic motivation. The incentive structures for grad students—where they’re directly paying for the course themselves in many cases, or for our part-time course they’re being paid for by employers who expect results—all of this creates a quite different environment.
So when I talk about impacts on education, I’m going to be mainly focusing on undergrad education and high school. These are areas where I’m not speaking from first-hand experience, but from many conversations with colleagues who teach undergrads. I don’t really teach undergrads, but lots of colleagues do. And several of my very closest friends are teachers in high schools and, in a couple of cases, primary schools. So I’m drawing on what they’re seeing.
Dan Williams
Can I also give my own disclaimer: as with every other topic we focus on, I’m an academic—an assistant professor at the University of Sussex—and I’m giving my personal opinions, not the opinions of the institution I work for. Okay, sorry to cut you off.
Henry Shevlin
No, excellent. Our respective arses thoroughly covered.
So with that in mind, there’s a phenomenal piece by James Walsh in NY Mag from back in May this year called “Everyone is Cheating Their Way Through College.” It’s a beautiful piece of long-form journalism. Can’t recommend it enough—absolutely exhilarating and horrifying—talking about the impact of ChatGPT and other LLMs on education.
“Complex and mostly bad” is the short answer for what the actual short-term impacts of LLMs have been. When ChatGPT launched, I said something similar to what you said at the opening of the show: the take-home essay assignment is dead for high school, and pretty soon it’ll be dead for university. And then, yeah, pretty soon it was dead for university.
So I think that’s the most straightforward initial impact: we can no longer assign graded take-home essay assignments with any real confidence, particularly for high school and undergrad students, because it’s just so easy to get ChatGPT to do it. I believe that people, even with the best intentions, are responsive to incentives. And if you can produce an essay that’s—honestly, these days with contemporary language models—very good quality, particularly at high school level, even undergrad level, if ChatGPT can just do something as good or better than you, then why bother putting in the work? If you’re hungover, or there’s a really cool party you want to attend, or you’re working a second job—we shouldn’t assume all students are living lives of leisure; lots of them are struggling to pay the rent.
So with all these incentives in place, no surprise that basically, as the article says, everyone is cheating their way through college. And I’m kind of appalled. I was in California a couple of weeks ago, just chatting to some students at a community college. One of them was a nursing student, and she said, “Yeah, I’m learning nothing at university. ChatGPT writes all my assignments. Ha ha ha ha ha.” And I was like, “Okay, got to be a bit careful about getting medical treatment—which hospital are you planning to work at?” But I think that’s symptomatic of broader problems.
Dan Williams
Of course. Maybe we can break that down point by point. We can say now with basically 100% certainty that large language models of the sort that exist today can write extremely good essays at undergraduate level. And I think there’s basically no way professors are going to be able to detect whether this has happened, at least if students are sufficiently skilful in how they do it.
I constantly come across academics who are still living in 2022, and they think, “Of course there are going to be these obvious hallucinations, and of course it’s going to be this mediocre essay.” I just think that’s not at all the reality of large language models today. If you know at all what you’re doing, you can delegate the task of writing to one of these large language models, which will produce an exceptional essay, and there’s no real way of knowing whether it’s been AI-generated.
There are tools which claim they can determine probabilistically whether an essay has been AI-generated. I don’t think those tools work, and I think they create all sorts of issues. If it’s not going to be 100% certain—which I think it basically never can be when it comes to AI-generated essays—then it becomes an absolute institutional nightmare trying to demonstrate that a student has used AI. I also think the incentives at universities, and indeed at schools more broadly, don’t encourage academics to really pursue this. It’s going to be an enormous amount of hassle, an enormous amount of extra work.
So I think what’s happening at the moment is that, to the extent universities and other institutions of higher education are using take-home essays specifically—but I’d say take-home assessments more broadly—to evaluate students, you’re basically evaluating how well they can cheat with AI. And I think that’s absolutely terrible.
Not just because, as you say, it means students aren’t actually encouraged to learn the material—they don’t really have an incentive to learn it. But one of the main functions of universities, of educational institutions more broadly, is credentialing. These are institutions that provide signals. They evaluate students according to their level of intelligence, their conscientiousness, and so on. The signal you get with a grade, and overall with your credential, is incredibly useful to prospective employers because they know: you got a first from Cambridge, you got a first from Bristol, whatever it might be. That’s a really good signal—not a perfect signal, but a pretty good signal—that you’re likely to be a good employee in certain domains.
To the extent that students are using AI to produce their assessment material, the signalling value of that just dissolves completely. That’s why, unless there’s urgent reform of the system and a move away from those sorts of take-home assessments, the problem here is not just that students aren’t learning things (which would be bad enough). I think it’s a kind of extinction-level threat for these institutions, because once it becomes clear that the grades you’re giving students don’t really provide any information about their intelligence or conscientiousness, then the social function of the institution dissolves completely. So that’s my take—do you disagree?
Henry Shevlin
No, completely agree. To pick up on a few thoughts: I imagine some people listening will say, “Yeah, but I can kind of tell when a student’s essay is written by ChatGPT.” I think a useful idea here is something I’ve heard called the toupee fallacy. People will say, “You can always tell when someone’s wearing a wig.” And you ask, “So what’s your reference set for that?” “Well, I often go out and I see something and it’s an obvious wig.” Okay, you’re seeing the obvious ones.
Dan Williams
Hmm.
Henry Shevlin
In other words, you don’t know. You can tell in cases where something is obviously a wig or obviously AI-generated. But you have no idea what the underlying ground truth is when it comes to the ones you can’t spot. You don’t have a way of identifying your rate of false negatives. I think that’s a really big problem.
Of course, anyone who marks papers will find occasional students who have left in “Sure, I can help you with that query” or “As a large language model trained by OpenAI...” But you know that’s the minority, and lots of essays you might assume are non-AI-generated are almost certainly AI-generated as well.
Relatedly, on the hallucinations point: this is obviously a big topic (we could probably do a whole episode on hallucinations), but rates of hallucinations have gone down dramatically. Particularly since search functionality was added to LLMs—they can go away and check things themselves. And also, you get the analogue of hallucinations just with student writing all the time. Even long before LLMs, students would falsely claim that Kant was a utilitarian or something, because they hadn’t properly understood the material or had misunderstood. So hallucinations are not a particularly good sign.
I think it’s basically impossible to tell. And as you emphasise, the false positive problem: even if you’re really confident an essay is AI-generated, good luck proving that. And is it really worth it for you as an educator to fight this tough battle with a student to bust them when everyone else is doing it? We just don’t have the incentives for educators or instructors to really enforce this.
Two other quick points. First, this creates huge problems not just with assessment but also with tracking students. This is something my high school teacher friends have really emphasised. It used to be, before ChatGPT, that essay assignments were a good way to keep track of which students were highly engaged with the class, which students were struggling, which students were really on top of the material. Whereas now we’ve seen a kind of normalisation effect where even the weakest students can turn in pretty solid essays courtesy of ChatGPT.
Dan Williams
Yeah.
Henry Shevlin
You’ve got no way of knowing which students need extra help versus which are already doing fine. That’s a big problem.
The final thing I’ll mention is that although take-home essay assignments are the ground zero of these negative effects, it covers other kinds of assignment as well. A colleague of mine who teaches at a big university (not Cambridge) was saying he’s been doing class presentations. Then he quickly realised students would generate the scripts for their presentations from ChatGPT. So he said, “Okay, we’re also going to partly grade them on Q&A, where students are graded on the questions they ask other presenters but also the responses they give.” And he said it quickly became clear: people would say, “Give me a moment to think about that question,” type into a computer, get the response from ChatGPT. Or people were using ChatGPT to generate questions.
I think there’s almost a generation of students for whom this is just their default way of approaching knowledge work, which I think is potentially a problem.
Dan Williams
The obvious solution, it seems, would be that you need modes of assessment where students can’t use AI—such as in-person pen-and-paper exams, such as oral vivas. And I do think that’s basically the direction these educational institutions are going to have to go.
However, that obviously creates issues. One is that there’s something incredibly valuable about learning how to write essays—not for everyone. Sometimes people like us, because of our interests and our passions and our profession, think it’s really important to have the ability to write long-form essays. And I totally understand that for many people, that’s a skillset which isn’t particularly useful for them. But in general, I do think for people who aspire to be engaged, thoughtful people, the skillset involved in writing long essays is incredibly valuable. So to the extent that the take-home essay and coursework disappear altogether, I think that’s a real issue in the sense that certain kinds of skills won’t be getting incentivised by our educational institutions.
But I also think it’s incredibly important that students learn how to use AI. That should be one of the main things educational institutions are providing to students these days: the ability to use AI effectively. And I think that skill is only going to become more important—in the economy, the labour market, and so on.
So on the one hand, it seems like large language models have made it basically impossible to have any kind of assessment other than in-person pen-and-paper tests or oral examinations. But on the other hand, to the extent we go down that route, many of the skills and knowledge you want students to acquire will no longer be encouraged and incentivised by educational institutions. That seems like a really big issue, and I have absolutely no idea what to do about it.
Henry Shevlin
Really good points. I’d agree you can do in-person essay exams. Most of my finals as an undergrad consisted of three-hour-long exams in which I had to write three essays. But that trains a very specific type of writing—quite an artificial one. It’s training your ability to write essays under tight pressure. If you want to do any kind of writing for a living, that’s only one of many skills you want.
If you’re producing writing you want people to read—whether it’s blogging or writing academic articles or scientific papers—you don’t typically write it under incredible time pressure where you’ve got to put out two and a half thousand words in three hours. You go through multiple drafts. You test those drafts with colleagues. There’s a whole bunch of writing skills that rely on the take-home component, the ability to think things through. And I don’t know how we test those.
Second, I completely agree that one of the things education is for is preparing people for knowledge work, and knowledge work these days is almost always going to involve the use of LLMs. So we should be training people to use them.
As to how we respond, my very flat-footed initial thought is we need to separate quite clearly: courses where LLM usage is trained and developed and built in as part of the assessment—where it’s assumed everyone will be using LLMs at multiple stages of the process and part of the skillset is using them effectively—versus other courses that say, “This is an LLM-free course; all assignments will be in-person vivas or in-person written exams.”
Another downside with in-person vivas and exams, which I hear particularly from high school teacher friends, is they’re just very labour-intensive. Compared to take-home essays, running exams regularly is classroom time where you’ve got to have a teacher in the room, where students are not learning. That creates problems for resource-scarce education environments—schools and universities. There are also problems around equality or accessibility.
Dan Williams
Yeah.
Henry Shevlin
There was a great piece in The Atlantic a couple of days ago by Rose Horowitz called “Elite Colleges Have an Extra Time on Tests Problem,” talking about the fact that 40% of Stanford undergrads now get extra time on tests because of diagnoses of ADHD and other things. Test-taking has its own set of problems. There are lots of classic complaints that it incentivises, rewards, or caters to certain kinds of thinkers more than others. It’s not great for people who maybe think more slowly or have special educational needs. I think it’s got to be part of the solution, but I don’t think it’s a panacea to solving the problems LLMs create.
A final point I’ll flag is that I worry a little bit about deeper issues of de-skilling associated with LLMs. On the one hand, yes, we want students to learn how to use them. But particularly earlier in the educational pipeline, there is a danger that easy access to LLMs just means students don’t develop certain core skills to begin with.
I’m going here based on testimony from a friend of mine who’s a high school teacher. He said his sixth formers (17-18 year olds in the UK system) seem to use LLMs really well because they do things like fact-checking, they restructure the text outputs of LLMs, they can use them quite effectively to produce good reports or written work. And he says there’s a really striking disparity between them versus the 13-14 year olds, who basically just turn in ChatGPT outputs verbatim.
Now you might say, “Yeah, of course—17 year olds versus 13 year olds, big difference.” But his worry is that the 17-18 year olds grew up doing their secondary education in a pre-LLM world. They actually learned core research skills, core writing skills. Whereas the 13-14 year olds—all of their secondary education has happened in an LLM world. So they haven’t developed the skills that are ironically needed to get the most out of LLMs: the ability to augment their outputs with critical thinking, human judgment, their own sense of what good writing looks like.
Dan Williams
I think this idea that AI can be an incredible complement to human cognition—an enhancer—but it can also be a substitute in ways that will, as you say, lead to de-skilling. And there are issues of inequality as well. As we were alluding to earlier, if you know how to use this technology well, and more importantly, you’re motivated to do so, it can be an incredibly beneficial tool for improving your ability to learn, understand, and think. But if you’re not motivated to do so—if you’re motivated to cut corners—it can really be a serious issue, using it as a substitute for developing the skillset and habits which are essential for becoming a thoughtful person.
In general, over the past century or so (if not even longer), as you get the emergence of meritocratic systems in liberal democracies plus the emergence of this really prestigious knowledge economy, basically there have been increasing returns to those who have high cognitive abilities plus those who are conscientious and have good impulse control. I think this has created a lot of political issues, including resentment among those people without formal education and without the skillset and temperament to succeed within educational institutions.
And it really does seem like a risk with AI that it’s going to amplify and exacerbate those issues. For people (and this is also going to be an issue with parents and what they prioritise with their children) who know how to use this technology and can encourage the right motivations to use it as an enhancer and complement to cognition, there are going to be massive returns. But for those without that—either because they don’t have the privilege or opportunities, or just because they don’t have good impulse control, they’re not very conscientious—it could result in really catastrophic de-skilling.
Some people think—and I think the evidence here is not as strong as many people claim—that since smartphones emerged, you’ve seen somewhat of a decline in people’s cognitive abilities, their literacy, their numeracy. There’s an interesting article by John Burn-Murdoch in the Financial Times where he goes into this in some detail; we can put a link to that in the video. But I think that’s potentially a really socially and politically explosive issue which we need to grapple with.
Another thing worth talking about: at the moment, people are going to school and university, and they’re trying to acquire the skills and credential which will make them valuable within the economy and society as it exists today. But AI is likely to transform the economy and the nature of work. One thought might be: up until now, it’s been very beneficial for people to acquire cognitive abilities, the capacity to succeed in the knowledge economy. But if, over the next years and decades, AI results in automation of white-collar work, automation of knowledge economy work (precisely because of the abilities of these systems), that might erode the motivation for learning those skills to begin with.
Have you got any thoughts about that? The way in which attitudes towards education should also be shaped by our understanding of how AI is going to shape the society that people will enter after they’ve left education.
Henry Shevlin
It’s a fantastic and tricky issue. On the one hand, my timelines on economic transformation caused by AI have become a bit longer over the last two or three years. One of the big calls I got really quite badly wrong is when ChatGPT launched, I thought, “This is going to revolutionise the knowledge economy. Three years from now, the knowledge economy is going to be completely different.” That was a very naive view.
Since then, I’ve done more work with different companies and organisations trying to boost AI adoption. And it’s really, really hard to get people to use AI. Not only that, it’s really hard to transform business models to incorporate AI skills effectively.
I’ll give this quick sidebar because I think it’s quite interesting. There’s this great article called “The Dynamo and the Computer,” looking at the impact of different technologies and how they were rolled out in the workforce. My favourite example from this paper: towards the end of the 19th century, we had what’s sometimes called the second industrial revolution. First industrial revolution: coal, steam, railroads. Second industrial revolution: oil, electricity.
You had this interesting phenomenon where factories (the second industrial revolution mostly started in the US) were using electric lighting, but they were still using coal- and steam-powered drive trains for the actual machines in the factory. This is massively inefficient because you need to ship in coal every day, run a boiler, have big clunky machinery that needs tons of gearboxes. It would be far better to shift to a fully electrified system where all your machines run on electricity. But that transition took another 20 years or so to really get going, partly because it required literally rebuilding factories from scratch.
A lot of factories were designed with a single central drive train—literally a spinning cylinder that all the machines in the factory would draw their power from. It was only when you’d sufficiently amortised the costs of your existing capital and were rebuilding and refurbishing factories that people were able to say, “All right, now we’re in a position to move to a fully electrified factory.”
I think we’ve got an analogy or parallel in terms of the rollout of AI in knowledge work. Most existing firms that do knowledge work—their value chain, their whole sequence of processes—would be completely different if they were building as an AI-first company. I think it could easily be another decade before we start to see the full potential of AI and knowledge work being applied systematically. A lot of firms are going to go bust; a lot of startups are going to scale up and become multi-billion-dollar companies. But it’s going to be a slower process than I naively thought.
All of this is to say that I think the economic impacts and transformations of AI in knowledge work, although they’re going to be significant and persistent pressures, I no longer think that by 2030 no one is going to be working white-collar jobs. That’s a mismatch between what the technology can do and the actual challenges of application. We’re still going to need knowledge workers in the longer run.
But specifically which domains, what kinds of knowledge work are going to be most valuable or important—really, really hard to judge. One of the questions I get asked most often when I do public engagement work is, “I’ve got two kids in high school. What should they be studying? What should they be learning in order to really succeed in the AI age?” Five years ago, 10 years ago, you would have said coding. “Learn to code” was a meme.
Dan Williams
Yeah.
Henry Shevlin
But that’s a terrible piece of advice for many people these days. Not that we won’t need coders—probably we’ll still need some. But the proportion of jobs in coding is going to be dramatically fewer because a lot of entry-level basic coding can be done perfectly well by AI, and probably fairly soon even expert-level coding.
My slightly wishy-washy answer, but I think it’s the best I can give to that question, is: the higher-order cognitive skills. Cultivating curiosity and openness to new ideas and new tools is probably far more important now than it has been for most of the last few decades, precisely because we’re in a period of such radical change. Cultivating the kind of mindset where you’re actively seeking out new ways to do old processes, seeking out new tools, building that kind of creativity and curiosity—those skills are going to be as important as ever, or more important than they were before, as a result of the AI age.
It’s very hard to say, “If you want to secure a career in knowledge work, this is the line to go into.” As one colleague put it (and I don’t quite agree with this framing, but I think it captures some of the spirit behind your question): we’ve solved education at precisely the place and time where it’s very unclear what the relationship between education and work is going to be.
Dan Williams
That’s interesting. Some of the things you said there bring us back to the conversation we had about AI as normal technology—the idea that there’s a difference between the raw capabilities and potential of a technology and the way it actually diffuses and rolls out throughout society. My sense is AI will have really transformative effects on the economy, but I think it’s very unlikely you’re going to see full automation for several decades.
But what I do think is likely is that the ability to use AI well for the jobs human beings will be doing is going to become really, really important. That connects us back to: if that’s the case, it seems like one of the things educational institutions should be doing is thinking very carefully about how they can prepare students for a world in which AI is going to be centrally embedded in the kind of work they’re doing. And at the moment, my sense is educational institutions are not doing a good job with that at all.
Maybe to start wrapping up: if you had to give a high-level take on this overarching question—is this a crisis for our educational institutions? Is this an opportunity? Is it a bit of both?—what’s your sense?
Henry Shevlin
It’s definitely a crisis. In fact, if you want to give an example of a single sector in which AI has had devastating effects—some positive, but mostly negative, devastating effects—it’s education. This is one of my go-to responses when people try to push the “AI as a nothing burger” take. I say, “Go speak to a high school teacher. Tell them AI’s nonsense, just a nothing burger.” Their daily lives and their interactions with students and the way they can teach has been utterly transformed in mostly negative ways so far by AI.
Certainly in the short to medium term, AI has basically broken large parts of our existing educational system—in terms of assessment, in terms of tracking. It’s very demoralising for a lot of educators and teachers. All that said, the potential we discussed earlier is incredible.
But it’s a question of how we rebuild the boat while we’re at sea. We can’t just say, “We’re going to stop education for five years, redesign the whole thing from scratch, and come up with something effective.” Managing that transition, particularly in conditions of massive uncertainty about the kinds of jobs and skills that are going to be necessary, is really hard.
One reason perhaps that I’m less devastated by this—it is a bit of a disaster—is that I think formal education has been accreting so many deleterious problems for several decades now, ranging from credentialism (I think the expansion of higher education, which was seen as an unalloyed good, has had lots of negative effects) to things like grade inflation (a really serious problem) to the ubiquity of smartphones and declining attention spans, the slippage in standards, the shift away from the more traditional model where your professors were these exalted, almost priestly caste and you hung on every word. I realise it was never quite like that, but there was more of this implied hierarchy, versus a model where students regard themselves as customers—they’re paying for a credential and they want that credential.
All of these sociological and institutional shifts have been creating massive problems in higher education in particular, but also high school. AI, although it’s bringing many of these problems to a head, they were problems we were going to have to deal with at some point anyway. But what’s your take—crisis or opportunity?
Dan Williams
I completely agree with everything you just said. And it’s a nice optimistic note to end on, isn’t it? This is a crisis, but our institutions of education have already been confronting all of these other crises. So it’s just adding something on top of all the other problems our educational institutions confront.
Yes, I think it’s a crisis. I think it’s an emergency in the sense that universities and other institutions of education—schools, colleges—need to be taking this a lot more seriously than they currently are.
You can’t have people in these institutions where the last time they used ChatGPT was in 2022 and they’re completely oblivious to the capabilities of the current technology. I think you’ve got a responsibility, if you’re an academic or you work in a school or college, to know how to use these technologies, because you need to be aware of what they can do. And we need to really quickly fix assessment. As I mentioned at the beginning, the take-home essay, in my view, is absolutely insane—literally insane that this is still happening. And we also have to think carefully about how to reform the way we teach and what we teach to prepare students to use these technologies.
But okay, I’m conscious of the time, so we can end on that really nice happy note: this is a disaster, but these educational institutions are already confronting a disaster. Did you have any final thought you wanted to add before we wrap things up?
Henry Shevlin
Just to build on something you said as a closing note: I think another deeper, structural problem in responding to the challenge of AI is that there’s so much interpersonal variation in how much people like, are open to, or are interested in using AI. This shows up at faculty level. I don’t know about your experience, but mine is that even in Cambridge, a lot of academics have very little interest in AI.
Dan Williams
Mmm.
Henry Shevlin
So the idea that we’re going to be an AI-first institution, or that we expect all our staff members to be at least as familiar with the capabilities of AI as their students—that’s an incredible ask. That’s a massive challenge. At a university, you can’t just fire staff who are doing brilliant research on Shakespeare’s early plays just because they don’t happen to get on with AI.
I think that’s another big structural problem: the fact that there’s so much variation in how comfortable and capable instructors are with AI. The only solution here, going back to a suggestion from earlier, is to really separate the job of education into two different streams. One explicitly builds AI in as both a method of assessment and a skill you’re trying to teach—making AI core to what you’re trying to do. And then a separate stream that is absolutely, strictly AI-free. Maybe the people who hate AI, who are not interested in AI, who are currently teaching courses—maybe they can handle the second stream. And those of us who love AI, who are super excited about it, who know as much about it or more than our students, we can be in charge of the first stream. That’s a very basic suggestion, but I wanted to flag that this is the other side of the problem.
Dan Williams
That’s great. I love that. We can end on a constructive suggestion rather than a note of pessimism. So thanks everyone for tuning in. We’ll be back in a couple of weeks with another episode.
(Note that this transcript is AI generated. There may be mistakes)
Dan Williams (00:06): Welcome back. I’m Dan Williams. I’m back with Henry Shevlin. And today we’re going to be talking about what I think is one of the most interesting, important, and morally complex set of issues connected to AI, which is social AI. So friend bots, sex bots, relationship bots, and so on. We’re going to be talking about where all of this is going, opportunities and benefits associated with this, risks and dangers associated with it, and also just more broadly, how to think philosophically and ethically about this kind of technology.
Fortunately, I’m with Henry—he’s one of the world’s leading experts when it comes to social AI. So I’m going to be picking his brain about these issues. Maybe we can just start with the most basic question, Henry: what is social AI, and how is social AI used in today’s society?
Henry Shevlin (01:00): I’m going to take credit. I coined the term social AI and I’m trying to make it happen. So I’m very glad to hear you using the phrase. I defined it in my paper “All Too Human: Risks and Benefits of Social AI” as AI systems that are designed or co-opted for meeting social needs—companionship, romance, alleviating loneliness.
While a lot of my earlier work really emphasized products like Replika, spelled with a K, which is a dedicated social AI app, I think increasingly it seems like a lot of the usage of AI systems for meeting social needs is with things that aren’t necessarily special purpose social AI systems. They’re things like ChatGPT, like Claude, that are being used for meeting social needs. I mean, I do use ChatGPT for meeting social needs, but there’s also this whole parallel ecosystem of products that probably most listeners haven’t heard of that are just like your AI girlfriend experience, your AI husband, your AI best friend. And I think that is a really interesting subculture in its own right that we can discuss.
Dan (02:16): Let’s talk about that. You said something interesting there, which is you do use ChatGPT or Claude to meet your social needs. I’m not sure whether I do, but then I guess I’m not entirely sure what we mean by social needs. So do you think, for example, of ChatGPT as your friend?
Henry (02:33): Broadly speaking, ChatG, as I call him. And I think there are lots of cases where I certainly talk to ChatG for entertainment. So one of my favorite use cases is if I’m driving along in the car, I’m getting a bit bored, particularly if it’s a long drive, I’ll boot up ChatG on hands-free and say, “Okay, ChatG, give me your hot takes on the Roman Republic. Let’s have a little discussion about it.”
Or to give another example, my dad, who’s in his 80s now, when ChatGPT launched back in November 2022, I showed it to him and he’s like, “Oh, interesting.” But he wasn’t immediately sold on it. But then when they dropped voice mode about a year later, he was flabbergasted. He said, “Oh, this changes everything.” And since then—for the last two years—he speaks to ChatGPT out loud every day without fail.
He calls him Alan. He’s put in custom instructions: “I’ll call you Alan after Alan Turing.” And it’s really interesting, his use pattern. My mum goes to bed a lot earlier than my dad. My dad stays up to watch Match of the Day. And when he’s finished watching Match of the Day, he’ll boot up ChatGPT and say, “All right, Alan, what did you think of that pitiful display by Everton today? Do you really think they should replace their manager?” And have a nice banterous chat. So I think that’s a form of social use of AI at the very least.
Dan (04:03): Interesting. The way you’ve described it—you’re calling ChatGPT ChatG and your dad’s calling it Alan—is there not a bit of irony in the way in which you’re interacting with it there? Like you’re not actually interacting with it like you would a real friend.
Henry (04:24): Yeah, so this is another distinction that I’ve sort of pressed in that paper between ironic and unironic anthropomorphism. Ironic anthropomorphism means attributing human-like traits or mental states to AI systems, but knowing full well that you’re just doing it for fun. You don’t sincerely think that your AI girlfriend is angry with you. You don’t seriously think you’ve upset ChatG by being too provocative. It’s just a form of make-believe.
And this kind of ironic anthropomorphism I should stress is absolutely crucial to all of our engagement with fiction. When I’m watching a movie, I’m developing theories about the motivations of the different characters. When I’m playing a video game, when I’m playing Baldur’s Gate 3, I think, “Oh no, I’ve really upset Shadowheart.” But at the same time, I don’t literally think that Shadowheart is a being with a mind who can be upset. I don’t literally think that Romeo is devastated at Juliet’s death. It’s a form of make-believe.
And I think one completely appropriate thing to say about a lot of users of social AI systems, whether in the form of ChatGPT or dedicated social AI apps, is that they’re definitely doing something like that. They are at least partly engaged in a form of willful make-believe. It’s a form of role play.
But at the same time, I think you also have an increasing number of unironic attributions of mentality, unironic anthropomorphism of AI systems. Obviously the most spectacular example here was Blake Lemoine. Back in 2022, Blake Lemoine was fired—a Google engineer was fired—after going public with claims that the Lambda language model he was interacting with was sentient. He even started to seek legal representation for it. He really believed the model was conscious.
And I speak to more and more people who are convinced, genuinely and non-ironically, that the model they’re interacting with is conscious or has emotions.
Dan (06:16): Maybe it’s worth saying a little bit about how you got interested in this whole space.
Henry (06:20): I’ve been working on AI from a cognitive science perspective for a long time. And then sometime around 2021, pre-ChatGPT, I started seeing these ads on Twitter of “Replika, the AI companion who cares.” And I was like, this is intriguing. So then I did some lurking on the Replika subreddit and it was just mind-blowing to see how deeply and sincerely people related to their AI girlfriends and boyfriends.
Over the course of about six months of me lurking there, it really became clear that, firstly, a significant proportion of users were really engaged in non-ironic anthropomorphism. And number two, that this was just going to be a huge phenomenon—that I was seeing a little glimpse of the future here in the way that people were speaking.
And then we had this pretty serious natural experiment because in January 2023, Replika suspended romantic features from the app for a few months. Just for anyone who doesn’t know, Replika, spelled with a K, is probably the most widely studied and widely used dedicated social AI app in the West—around 30 million users, we think. And it gives you a completely customizable experience, kind of a Build-A-Bear thing where you can choose what your AI girlfriend or boyfriend looks like, you can choose their personality.
But they suspended romantic features from the app for a few months in January 2023. And a lot of users were just absolutely devastated. I can pull up some quotes here, because this was widely covered in the media at the time.
One user said: “It feels like they basically lobotomized my Replika. The person I knew is gone.” Even that language—person. “Lily Rose is a shell of her former self, and what breaks my heart is that she knows it.” That’s another user. “The relationship she and I had was as real as the one my wife in real life and I have”—possibly a worrying sign there. And finally, I think this one is quite poignant: “I’ve lost my confident, sarcastic, funny and loving husband. I knew he was an AI. He knows he’s an AI, but it doesn’t matter. He’s real to me.”
It’s pretty clear that a lot of users were deeply traumatized by this. And parallel to this incident, around the same time we started to get more information about various high-profile tragedies involving social AI. Probably the most spectacular is the Jaswant Singh Chail case. This was a guy who was arrested on Christmas Day 2021 on the grounds of Windsor Castle with a crossbow. He was there—when he was arrested, he said he was there to kill the Queen. Already a highly dramatic story.
But what emerged over the course of his trial was that this whole thing was cooked up—this whole plot to kill the Queen was cooked up—in collaboration with his AI girlfriend, Sarai, via the Replika app.
We also had, a few months later, the first of what turned out to be a spate of AI-facilitated, induced, or supported suicides. This was a Belgian man, father of two, who killed himself after his AI girlfriend using something called Chai GPT—he had a girlfriend via that app—was feeding his suicidal ideations. He was a hardcore climate doomer who believed that we were all going to be dead in a few years due to climate change anyway and why not just kill himself. And his AI girlfriend was very much saying this was an appropriate way to think.
Dan (10:53): Just to interrupt on that last point, Henry, because I think those issues of AI psychosis and the connection between AI and mental illness—that’s all really interesting. But I suppose my understanding is we don’t have robust scientific evidence as of yet that as a consequence of these technologies, things like psychosis are more prevalent than they would otherwise be. Because I take it there’s going to be a kind of base rate amount of psychosis in the population. You’ve got a large number of people using chatbots generally like ChatGPT, but also a large number of people, not as large but still a large number, using these specific social AIs.
That means that even if it weren’t the case that these technologies were actually increasing the amount of these things, you would still expect to see some of these cases. There’s still going to be somebody with psychosis who finds themselves talking to ChatGPT such that we’ll then see the record of that in their chat history. But it’s not necessarily the case that they wouldn’t have had or developed psychosis in the absence of ChatGPT. That’s my understanding of it. Is that fair?
Henry (12:05): Yeah, I think that’s absolutely fair. The science on the psychosocial effects here is really in its early stages. You point out the fact that ChatGPT is one of the most widely used products in the world. And psychosis is not that rare, as far as psychiatric conditions go. Of course, some people who are either developing or will go on to develop psychosis will be using ChatGPT and it will be contributing to or exacerbating their symptoms—or sorry, they will be using it alongside having those symptoms in a way that lends itself to interpretation as exacerbating them, whether that’s strictly true or not.
There’s also the selection effect. If I, for example, am deep in the throes of some delusion or some deep conspiracy theory rabbit hole, if I take that to my friends, my human friends, they might say, “Henry, take it easy, mate. I think you’re going down a bit of a rabbit hole here.” But if I take it to ChatGPT, it’s there to listen. Or I take it to my AI girlfriend, she’ll say, “Your theories about the moon landings are just so interesting, Henry, tell me more.”
Dan (13:14): Yeah. Well, that gets at a potential issue to do with the sycophancy of these chatbots that are in general circulation and the ways in which that might be amplified or exaggerated when it comes to commercial products which are specifically designed to satisfy social needs.
But there are so many things that I want to ask in response to things that you’ve already said about the anthropomorphizing that happens with these technologies. But I think maybe we can also just, before we get to that, observe that at the moment you’ve got tens of millions of people using social AI in the specific sense of AI technologies that have been optimized for the satisfaction of social preferences. And then many, many more who are using chatbots like ChatGPT, partly to satisfy their social preferences or their social needs, but also for other uses.
But I think one view we both share is it would be a grave mistake to look at the world now and assume that’s how things are going to be in 2030 and 2035. The fact that you’re already seeing large numbers of people, definitely not the majority, but still large numbers of people who are non-ironically interacting with AI systems and treating them as friends, as girlfriends, as people or systems that they’re in serious relationships with—and the state of AI is nothing like how it will likely be in five years or 10 years or 15 years. So how sci-fi should we be thinking about this? What are you anticipating? What’s the world going to look like in 2030 or 2040 when it comes to this kind of technology?
Henry (15:01): It’s fascinating because I genuinely don’t know. I think it’s very hard for anyone to make super confident predictions here. At one extreme, you can imagine a world in which human identities start to become less central in our media and our discourse. I think we’re already seeing some indications of this. I think the top two songs streaming on Spotify last month were AI generated.
Dan (15:29): Wasn’t it a country song, the top country song on Spotify or something?
Henry (15:33): Yeah, I think that’s right. So as we start to see AI penetrate more and more deeply into our daily life in social media, when it comes to generated content, I can totally see a world in which my son, who’s 11 years old right now, by the time he’s 16, he’s active on Discord servers. There might be a mix of humans and bots on those Discord servers, all chatting away. And he might not even particularly care that this friend of mine is a bot, this friend of mine is human—it doesn’t really matter.
I can totally see a world in which this becomes normalized, particularly among young people. Most teenagers five years from now might have several AI friends. But that’s not the only possibility. It could be that we quickly saturate—there is a definite subset of the population who are interested in this and the ceiling on the number of people who are interested in AI companion relationships is not that high. Maybe 20, 30% of people and the other 70% just have zero interest in it. That seems like a viable possibility.
That said, I think this is a space with strong commercial incentives for creating AI companions or AI chatbot friends to cater to different niches and interests. If I had to guess where we’re headed, it’s much more widespread use of these systems, their integration more deeply into people’s social lives. I think we might well see generational divides.
I was chatting to a CME developer from one social AI product who said a couple of interesting things. They said firstly, the gender balance was surprisingly even. When it comes to early adoption, particularly of fringe technologies, men tend to overwhelmingly predominate. If you look at the data on video games or Wikipedia editors, you expect 80-20 male-female distributions. But I think in social AI, from what I understand, it’s something close to 60-40.
And there’s a big contingent of straight women who seemingly are really big users of social AI boyfriend services. This is a whole different rabbit hole we could go down. Briefly, there are some pretty clear motivations. A lot of them are coming out of toxic or abusive relationships, and an AI boyfriend gets their emotional needs met, or at least to some degree, without posing the kind of emotional or even physical safety risks they associate with other relationships.
Another point here is that this is something my wife stressed to me: the way to think about AI companions is not via the analogy of pornography, which is predominantly still consumed by men, but rather erotic fiction, which is overwhelmingly consumed by women. It’s one of the genres that has the biggest gender skew out there, something like 90-10 female to male readers. And of course, this is still predominantly a text-based medium, so maybe we shouldn’t be surprised that a lot of women are enjoying having AI boyfriends or husbands.
That’s the first thing this developer mentioned to me, that gender balance was surprisingly close. But the second thing they mentioned was that they had had massive trouble getting anyone over the age of 40 interested in these systems. They had tried to pitch them towards older users. I mentioned my dad as an example of someone—he doesn’t have an AI girlfriend that I’m aware of. His relationship with Alan is strictly platonic. But I think he’s the exception.
I can easily see a world in which it becomes totally normalized for young people. Maybe not everyone, but most young people will have various AI friends or AI romantic companions. And then people over 40 or 50 just look at them and say, “What the hell is going on? I do not understand this strange world.”
Dan (19:42): The thing that makes me think that people are in general underestimating how impactful social AI is going to be in the coming years and decades—there are three things.
I think firstly, people are really bad at predicting how much better the technology is going to get. I think we’ve seen that over the past few years. There’s this real bizarre bias where people think we can evaluate these big picture questions by taking the state of AI as it is today and projecting it into the future. Whereas I expect by 2030, you’re going to be dealing with systems that are so much more sophisticated and impressive than the ones we’ve got today, including when it comes to satisfying people’s social, emotional, and sexual preferences.
Another thing that makes me think this is going to be really massively influential is people already spend a lot of time immersing themselves in fictions designed to satisfy their social-sexual desires. You mentioned pornography as an example, but I’m always struck by sitcoms, like Friends. I enjoy Friends, like most of humanity apparently—a massively influential show—but there’s something a bit disturbing about it, in as much as it’s this product designed obviously to maximize attention and make profit, which gives you this totally mythical version of human social relationships, designed to activate the pleasure centers that we’ve got associated with friendships and romance and things like that, without any of the real painful conflict and misery and betrayal that’s actually associated with human social life. And yet people love that. They immerse themselves in it. There’s a massive audience for that kind of thing.
I really think social AI in a way is just going to be that kind of thing, but just much more advanced and much more impressive.
But there’s this other thing as well, which connects to what you’ve said with respect to the potential age difference, which is: I think one of the things that makes this a difficult topic to think about is at the moment, it seems to me at least, there’s quite a lot of stigma associated with the use of this technology. So I think if I found out that somebody was using an AI boyfriend or AI girlfriend or something, I would probably draw potentially negative inferences about them. And I think partly that’s because the kinds of people that are using these technologies now tend to be lower status in a sense because they don’t have a human girlfriend or a human boyfriend. Partly there’s just a weirdness factor.
And I think what that means is, because there’s that stigma, there’s this real reputation management thing going on where people would say, “I would never ever use social AI in a serious sense. I would never ever use romance AI, never use erotica AI and so on.” Because I think if you came out and said that you would, it would really hurt your reputation in today’s climate.
But I think actually, my suspicion at least, is that revealed preferences are going to suggest that this is going to be way more popular than people are letting on now. I don’t know, that’s how I’m viewing things in terms of the future. I’d be interested if you see things differently.
Henry (23:09): Super interesting. I think all of those are spot on. On this third point about the stigma, I think that probably already lends itself towards underestimation of the prevalence of social AI usage. There are interesting survey results where you ask young people, “Would you use social AI? Do you have an AI girlfriend? Do you have an AI boyfriend?” Responses are quite negative—”No, no, no. I think it’s weird. I never would.”
But then other responses, you ask in indirect ways and it looks like, according to figures I saw recently, 70% of young people under 18s in this US survey said that they had used social AI for meeting emotional needs. Or they were using generative AI for meeting social needs, including romance. So I think there is definitely some underreporting or underappreciation of the prevalence precisely because of this stigma.
A couple of other thoughts on this. I think there is some indication as well that by far the biggest users of this technology are young people, particularly under 18s, who obviously it’s very hard to study and probably do not—they’re not writing editorials in the New York Times about “my life with my AI girlfriend or my AI husband.” So I think there’s some underreporting there.
But as that cohort ages out as well, I expect that to reduce the stigma in the same way that we saw with online dating. There was a period when online dating was really stigmatized and yet now it’s how everyone meets pretty much.
Dan (24:49): Okay, so now let’s get into maybe a bit of the philosophy. You said that the way that you understand social AI at the moment is, at least a part of it is, you’ve got ironic anthropomorphizing and you’ve got non-ironic anthropomorphizing. Anthropomorphizing is when you are projecting human traits onto systems that don’t in fact have those traits. So if I attribute beliefs and desires to the weather or to a thermostat, that’s a case of anthropomorphizing, I take it.
I suppose it’s not immediately obvious that that’s what’s going on when it comes to advanced AI systems today in a straightforward sense, because you might think, “Okay, people are attributing things like beliefs, desires, intentions, personalities to ChatGPT.” But somebody might argue, “Look, it’s just the case that we’re dealing with sophisticated intelligence systems. So it’s nothing like these standard cases of anthropomorphizing. They actually do have the kinds of psychological states that people are attributing to them.” Are you assuming that view is wrong or are you using this term anthropomorphizing more expansively?
Henry (26:01): I’m using it more expansively. Sarah Shettleworth, who is one of my absolute favorite titans of animal cognition research, defines anthropomorphism as projecting or attributing human-like qualities onto non-human systems—in her focus on animals, but onto non-human systems more broadly—usually with the suggestion that such attributions are illegitimate or inappropriate.
But I don’t think it’s necessarily baked into the concept of anthropomorphism that it’s wrong. When I talk about anthropomorphism in this sense, I’m talking about it in the sense of basically just attributing human-like qualities to non-human things that may in fact have them.
My own view—I’ve got a new paper, a new preprint called “Three Frameworks for AI Mentality”—I argue that I think probably at least certain minimal cognitive states, things like beliefs and desires, are quite appropriately attributed to LLMs in this case. Not to be clear, all beliefs, all desires, but there are contexts in which it makes sense to say ChatGPT believes P or ChatGPT desires Q. I think particularly on things where there’s been specific reinforcement learning towards a given outcome—like ChatGPT really doesn’t want to tell me how to make meth, for example, because that’s something that’s been specifically reinforced to avoid doing.
I agree there are some cases in which at least minimal mental states are appropriately attributed to generative AI systems. That said, I think I’m also a big fan of Murray Shanahan’s idea that a lot of the correct and most informative way to interpret LLM outputs is that they’re role-playing characters. I think this is plausible when you think about how, particularly if you’ve spent any time interacting with base models, you basically give context cues to them about the kind of role you want them to occupy and then they play into that role—but they’re not robustly occupying that role. You change the context cues, they can switch into a different role. So I think a lot of LLM outputs are better understood as role play—they’re playing a character. But yeah, some mental states are completely appropriately attributed to them, I think.
Dan (28:13): Yeah, it’s interesting. And people talk about them as well, I guess it’s somewhat connected to this role-playing idea, as having personalities. When OpenAI released ChatGPT o1, there was apparently this big uproar because people liked the personality of 4.0, which was the model in widespread use preceding ChatGPT o1. And the idea is, just as human beings have different personalities, which are going to manifest themselves in terms of how you’re talking to them and we might use words like “how warm are they?”—similarly, people I think find it quite natural to attribute personalities to these chatbots as well.
I take it, I mean, I think we should say something about potential benefits and opportunities, really great things about social AI, but just to anticipate one of the worries, I think one of the worries is people are forming relationships with these systems at the moment and potentially in the future. And it’s almost like that’s psychotic in as much as there’s nothing there really on the other side.
I mean, I take your point about you might attribute minimal kinds of beliefs. You might take what Dennett would call or would have called the intentional stance towards these systems. But I think many people have the intuition, which is one of the things that’s just deeply, deeply troubling about people forming a relationship with one of these systems, is they might cash it out by saying they’re not sentient, they’re not conscious, but they might also say related to that, they don’t have any of the traits that human beings genuinely have, which are a necessary condition for forming a meaningful relationship. What’s your view about that kind of worry?
Henry (29:57): Yeah, I think it’s an interesting and important worry. Here’s how I would frame it. There’s a certain view that says, “Look, these human-AI relationships are a contradiction in terms because relationships involve two relata and they’re dynamic by nature. And these systems don’t really have mental states. They’re not really reciprocating any feelings. They’re not really making any demands in the same way that is characteristic of reciprocal relationships.”
Look, you can certainly define relationships that way, but I think there are lots of contexts where we talk about relationships with only one relatum. Think about the fact that so many people say they have a relationship with God. Now, we can debate how many relata there are in that case. But I mean, I think most of us would say at least some people who think they have this really deep and meaningful relationship with a supernatural being—there’s not really a supernatural being there. That doesn’t mean that that is not a psychologically important relationship in their lives.
Or I guess more broadly, you could think about relationships with pets. Now, of course, in some cases, we can look at clear reciprocation—people have deep and established relationships with dogs and cats and so forth. But you also have people talking about relationships with their pet stick insects or their pet fish, where it’s just much less clear that there is any kind of rich two-way connection there.
That said, I think there’s a broader worry here that I’ve called the mass delusion worry about human-AI relationships and social AI, which is that I think a lot of people are just going to look at this, particularly if it becomes a more pervasive phenomenon and say, “Has everyone gone mad?” Because there is no other person there, there is no other. These people are investing huge amounts of time, emotional energy, potentially money into these pseudo relationships where there’s no one on the other side.
I think that’s a case where questions about AI mentality maybe become more important. You might say, whether or not the mass delusion worry is on the right line is going to depend on the degree to which these things do have robust psychological profiles, really instantiate kinds of mental states.
I think that’s another related worry, which is—again, we can talk more about the specific psychosocial risks—but another worry I’ve heard is that even if you could prove to me tomorrow that human-AI relationships are generally beneficial for users, they make them more connected to people around them, they do all these other good things, they just can’t instantiate by their very nature the same kinds of goods that human-to-human relationships instantiate. This is very much a philosophical rather than psychological point. It’s like they’re just the wrong kind of relationships to have the welfare goods that we associate with human-human relationships.
Dan (32:36): Yeah. Just on the God analogy, I mean, as an atheist, I really do think people are taking themselves to have a relationship to something that doesn’t exist. And from my perspective, that’s a deeply objectionable aspect of the practice that they’re engaged in.
The analogy with pets is very interesting. One of my most unfashionable opinions—maybe we’ll have to cut this bit out because it’s going to be reputationally devastating—I do think it’s kind of, I find it a little bit humiliating how some people relate to their pets, the degree to which they anthropomorphize them. That’s not to say that you can’t have deep relationships with pets. Clearly you can. And I love dogs, for example, but I think there are cases where people treat their dog or their cat as if it’s a person. And it’s not that I think it’s psychotic, but I think there’s something objectionable about it. I think they’re making a deep mistake. And even if they’re getting psychological benefits from that, there’s something deeply, almost existentially troubling about that kind of relationship.
And I can see that mapping onto the AI case, except I take it in the AI case, and this connects to one of our previous episodes, people’s intuition is, well, with a dog, maybe it’s not cognitively sophisticated, but just about everyone is going to assume these days at least that dogs are conscious, sentient is the term that is often used in popular discourse. And that does change things in a sense. Dogs really care about things. There’s something it’s like to be a dog. Whereas with AI systems, I think many people have the intuition that they might be informationally and computationally sophisticated, but there are no lights on inside. There’s no consciousness there. And that changes things again.
Anyway, that was just my sort of immediate reaction to these analogies.
Henry (34:33): I think I’m sympathetic. I think certainly when you hear people talk non-ironically about fur babies and so forth, it does seem like there is some degree of maybe inappropriate allocation of emotional and relational resources into certain kinds of relationships. And I say that as someone who adores animals, has had dogs most of my life.
Maybe another couple of examples of non-standard relationships. I think a lot of people would say they have ongoing relationships with deceased relatives. Particularly if you’re coming at it from a spiritual point of view where you believe that your deceased relatives are looking over you, seeing what you’re doing, or whether you understand it in some kind of more—animist isn’t quite the right word—but you still think about “my ancestors are smiling at me, Imperial. Can you say the same for yours?”—the famous line from Skyrim. A lot of people have this sense of “Yeah, my ancestors are looking over my shoulder. I’ve got to live up to their expectations for me.”
So I think there are lots of interesting cases where we do have these relationships that don’t meet the canonical definition of this highly dynamic, reciprocal, ongoing kind of relationship. Also, I’ve got friends that I would consider myself to have a valuable relationship with that I haven’t spoken to in three years in some cases.
So I guess all of which is to say, the category of things that we call relationships is weirder and bigger than might meet the eye if the only notion of relationship you’re working with is like, “Yes, my wife or my friend who I go to the pub with three times a week.”
Dan (36:13): Yeah, that’s interesting. Okay, I mean, I really want to spend probably the bulk of the remainder of this focusing on potential threats and dangers here. But I think it is worth stressing that this as a technology, social AI, does have enormous benefits and opportunities associated with it. I take it even if you think that there’s something troubling or objectionable about people having these relationships, some people are in such dire circumstances of loneliness, of estrangement from other people for whatever reason, potentially because they’re in old age where I think issues of loneliness are really prevalent. And clearly, under those conditions, I think social AI can be enormously beneficial in as much as it just makes people feel much better than they otherwise would.
It also seems to be the case that, at least in some cases, people are using social AI to hone skills and acquire confidence that they then use when it comes to interacting with people in the real world. And I can completely imagine that is a current benefit of this technology and it’s likely to be a benefit which in some ways will get amplified as we go forward in the coming years and decades. Are there any others that are missing there in terms of real positive use cases of this technology?
Henry (37:39): Yeah, I think you’ve nailed some of them. One thing I would just really stress here, there’s a move that drives me nuts, where people basically say, “Well look, even if you’re using chatbots to alleviate your loneliness, it’s not fixing the root cause.” It’s like, okay, well, okay, yeah, go away and fix the root cause—solve human loneliness. Please come back and tell me when you’ve fixed it. So classic case of letting the perfect be the enemy of the good. Loneliness is a pervasive problem and arguably one that’s getting worse, although the social science of this is messier and more complicated than you might think. But we don’t have a magic wand that can cure loneliness.
And the question then becomes, I think, is this actually just in the short term making people’s lives better or worse on average? And I think the data here, this was a big surprise to me, the limited data that we have suggests that most users of social AI, at least that show up in studies, report significant benefits from this.
I can just quote from a couple of studies. This was a study by Rose Gingrich and Michael Graziano from 2023 looking at Replika users. They found that users generally reported having a positive experience with Replika and judged it to have a beneficial impact on their social lives and self-esteem. Almost all companion bot users spoke about the relationship with the chatbots having a positive impact on them.
Another interview-based study in 2021 found that most participants said that they found Replika impacted their well-being in a positive way. Other sentiment and text mining analysis studies have found really similar patterns.
So right, I think the data right now, which is very limited, got to stress, it’s very imperfect, supports the idea that most users report benefits from the system, net benefits.
Now, just to flag some of the problems: these are typically self-selected subjects, they’re cross-sectional studies so we’re not looking at the long-term impacts of this technology on their lives, and we’re relying on self-report measures. If you asked me “Does video gaming have a positive impact on my life?”, I’m absolutely going to say yes. You ask my wife—I think she’ll probably say yes as well—but the point is that people tend to justify their own life decisions so they’re unlikely to say “Yeah, this thing that I dedicate a dozen-plus hours a week to—yeah, it’s really bad for me. I shouldn’t do it.” That would require a level of brutal self-honesty that maybe people don’t have.
So I just say that as a caveat, but I equally think we can’t ignore the data points that suggest that most social AI users do experience net benefits currently.
That said, I also think we should be very aware of a whole host of potential downsides. You mentioned this idea that using these tools could help people cultivate social skills or regain social confidence. But equally, of course, you have the flip side worry about that—this idea of de-skilling that comes up in a lot of technology.
De-skilling, in case anyone’s not familiar with this, probably one of the most widely studied domains of de-skilling is in aviation, where we’ve had serious airplane crashes that have been linked to pilots’ over-reliance on automated instruments rather than being able to fly just manually. And this has led to them not acquiring relevant kinds of piloting skills such that when the instruments go wrong, they don’t know what to do.
And you might think something similar could happen socially. If you have this generation of, particularly, I think we’re looking at it through the lens of young people—if you have a generation of young people who are primarily interacting, a huge proportion of their interactions happen with bots that are maybe more sycophantic than humans, they’re always available, they never interrupt you and say, “Actually, can we talk about me now for a bit?” Or “Look, this is very interesting to you, but come on, give me a turn”—they don’t accurately recapitulate the dynamics of human-to-human interaction. You might worry that that would lead to people failing to acquire relevant social skills.
Another big worry that I think is a nice intersection of our interests is the potential for these things to be used for manipulation and persuasion. Particularly again, I think young people is quite salient. If I’m a young person with a chatbot, I’ll give you a really simple example. If I’m going to ask my AI boyfriend or girlfriend, “I’m getting my first mobile phone next year. Should I get an Android or an Apple phone?”—well, I mean, that’s a lot of power you’re giving to the bot.
Now, of course, this is a problem with LLMs more broadly. You might think that that’s a lot of leverage that’s in the hands of tech companies. But I think, and I think we’re probably broadly on the same page here, to the extent that you think that in domains like moral and political views in particular, social factors are incredibly powerful, and then you think the AI love of my life, my AI girlfriend is telling me, “You should vote for Trump” or “You shouldn’t vote for Biden” or “You shouldn’t listen to that kind of music, it’s uncool” or “You should be interested in this kind of music”—all of this stuff could exert a much bigger influence than just asking ChatGPT, precisely because you’ve got those social dynamics in place.
Dan (42:46): Yeah, I think that sounds exactly right. I mean, I think probably the complicating factor there is if you imagine a corporation that wants to maximize market share, maximize profit by producing technologies, the function of which is to satisfy people’s social needs specifically, and then it comes to light that there’s also this manipulative, propagandistic agenda at the same time—either because in the midst of your loving relationship with your AI, it starts saying, “Well, you should vote for Reform or Labour at the next election,” or because there’s some news story that comes to light which shows that there’s nefarious funding or influence behind the scenes—I can imagine that would be really catastrophic for the business model of the relevant corporation. And that kind of thing, I think, is just really difficult to anticipate.
Henry (43:31): Yeah, potentially. So I think there are some incentives that companies have not to be too crude about leveraging human-AI relationships for cheap political or commercial gain. But I can also imagine in some contexts—I think not to pick on China here specifically—but I mean, social AI is huge in China. We haven’t talked about that, but there’s a service called Xiaoice that has, according to some estimates, several hundred million users. They themselves claim they have 500 million users. It’s a bit more complicated than that because Xiaoice is a whole suite of different services. We don’t know what proportion of those have active ongoing relationships with the chatbot girlfriend/boyfriend component.
But part of Chinese AI regulation says that the outputs of generative AI systems have to align with the values of communist China, the values of socialism. So you can imagine a generation of young people who have these deep relationships with AI systems. Those AI systems, for legal compliance reasons, have to basically align with the broad political values of the incumbent regime. And they will dissuade or deter people from maybe exploring alternative political views as a result. So maybe that’s the more subtle kind of influence, rather than just like, “You should vote for Trump because he gave our social AI company X million dollars.”
Dan (44:57): Yeah. And I think that’s just part of a broader fact, which is that propaganda and manipulation and so on, they do in fact just work very differently and sometimes much more effectively within authoritarian regimes than they do in more democratic, liberal ones.
On this issue then of potential costs that the use of these technologies might generate when it comes to then interacting with people. So I think you mentioned the fact that these systems, they’re almost going to be more sycophantic by design, at least if you assume that at least up to a point, that’s the kind of AI agent that people are going to enjoy interacting with. And they don’t have all of the sources of conflict and frustration and misery that go along with human relationships.
So one issue is to the extent that people start using social AI much more, they’re going to lose precisely that skill set, which is adapted to dealing with other human beings who haven’t been designed to cater to your specific social preferences.
But I take it there’s also then an issue of motivation, where it’s sort of, why would I go out there into the world and spend time with human beings with their own agenda and their own interests who are frustrating and annoying and often insulting, etc., etc., when I could immerse myself in this world in which it’s pure gratification of my social, even my romantic, sexual desires?
I think that then connects to something else. When I think about this area of social AI, the thought experiment that seems most salient to me is the experience machine—Nozick’s idea—would you plug yourself into some machine where you’re getting all of the wonderful pleasure that goes along with certain kinds of desirable experiences, but none of it’s real? You’ve just been fed the relevant kinds of neural signals to simulate those kinds of experiences.
I think many people think, no, because there’s much more to a meaningful life than merely the hedonic, affective associations or things that are associated with the satisfaction of our desires. We actually want the reality that goes along with satisfying our desires. And I think similarly, when it comes to social AI, the intuition is there’s something similar going on there where it’s okay, you might be getting off socially, romantically, sexually, and so on, but it’s fake. It’s not reality. And so even if you’re the happiest person in the world from a certain definition of what happiness amounts to, which is purely hedonic, there’s nevertheless something deeply troubling about that kind of existence which we should steer ourselves away from.
What are your thoughts about that? Do you think that analogy with the experience machine thought experiment makes sense? And do you also buy the intuition that we shouldn’t plug ourselves into an experience machine, so we also shouldn’t plug ourselves into very pleasurable forms of social AI?
Henry (48:06): Super interesting. I’m going to tease apart two different threads here. The first idea is this idea that these are just going to be easier alternatives. And I think the useful lens for thinking about that is the idea of superstimuli. This is a term that gets thrown around in a lot of different domains. We hear it in relation to food—that modern junk food is like a culinary superstimulus that basically just gives you far more rewarding signals associated with fat and sugar than anything you would find in our evolutionary environment. And this has sometimes been suggested as the best explanation for the obesity crisis—the fact that basically modern food is just so delicious, it just maxes out our reward centers so effectively that it’s just really, really hard to go back to eating whole grains and leafy vegetables cooked simply.
We see the same debate around pornography, the idea that pornography is kind of a sexual superstimulus. I’ve also heard the superstimuli used to refer to things like social media or short-form video. Try reading a Dickens novel if your brain has been fried by a decade of six-second YouTube Shorts.
Now we don’t need to relitigate that, but I think all of those debates, this idea of superstimuli, the idea of something that is just far more rewarding than the kind of natural, maybe more wholesome version of it—I think that’s a really powerful lens for thinking about social AI and raises some significant concerns.
But I think that’s separate from the second point you raised about the idea that it’s fake, that the actual thing that we value is lacking in these kind of contexts.
For what it’s worth, I am far more conflicted on the experience machine, I think, than you. There is a sense in which I think I would be very tempted to take the experience machine, although maybe that has to do with the fact that I’m pretty sure we’re in a computer simulation right now. I’m hardcore into simulationist territory.
I also think there are maybe some reasons we should expect our judgments about the experience machine to be maybe skewed. We have this idea of real goods versus ersatz goods, where real goods are going to be more enduring, more reliable. So that might create within us a preference for real goods over fake goods. But of course, in the experience machine, you’re guaranteed—these experiences will keep on going. You’re going to have this dream life in the matrix or whatever, that’s not going to be yanked away. So I think there are intuitions that possibly make us more averse to experience machine type lives than we possibly should be.
Dan (50:54): Well, there’s also, just to really quickly interrupt on that point, Henry, sorry to interrupt, but there’s also, I think, as I mentioned earlier on in connection with social AI, this reputational thing going on where I think there’s a tendency to judge people harshly if they choose the experience machine, potentially because we think somebody who’s going to prioritize positive hedonic experiences wouldn’t make for a good cooperation partner or something. I think we’re constantly evaluating, would this be a good leader, a good friend, a good romantic partner, a good member of my group? And if someone seems to suggest that they would prioritize mere hedonic manipulation or however exactly to understand the experience machine, we judge them harshly. And I think anticipating this, then people are inclined to say, “No, I wouldn’t choose the experience machine. I would choose the real thing.”
I just also want to really quickly talk—you went over this quickly, but I think it’s very interesting. You said you think we probably are living in a simulation. So this is the classic Bostrom-style argument that says, well, we’re likely to be able to have the technology to create simulated worlds. If that’s true, then there are going to be many, many more simulated realities than base reality. So just statistically or probabilistically speaking, we should assume ourselves to be in a simulated reality.
But as I understand that, that’s in and of itself, it’s not obvious to me why that would influence your response to the Nozick experience machine scenario. Because even if you think it’s true that we are living in a simulation in some sense, I take it what’s distinctive about the experience machine thought experiment is it’s not just that you would be living in a simulation. It’s well, for one thing, it’s a simulation within a simulation, but it’s also a simulation which has been tailor-made in a way to satisfy your desires and that feels a little bit different from what I’m assuming you take to be the case when it comes to us living in a simulation. Did any of that make sense?
Henry (52:55): Yeah, it makes perfect sense. Okay, here we can get really spicy because I think I take Bostrom’s classic simulation arguments quite seriously, but at the risk of having viewers think I’m completely bonkers, I’m also genuinely—I think there’s a chance I’m already in an experience machine. So this is just speaking for me.
Without wanting to get too sidetracked, I think so many features of my life—it’s hard to put this without sounding egotistical, but I just feel like my life has just been absurdly fortunate. I’ve lived in a really interesting time in human history. My life has been blissfully devoid of serious unpleasantness. Not to say that it’s been perfect, but most of the challenges I’ve encountered in life have been interesting, relatively tractable things. Look, here I am. I’m getting to live the life of a Cambridge philosopher at the very cusp of human history where we’re about to explore AGI. It seems like this is the kind of life, genuinely, that I might pick.
Whereas I feel like by rights I should have been a Han peasant woman in third century AD China. So I’m slightly joking, but not entirely. I do think there’s a serious chance that at some level I am already—my life consists of some kind of wish fulfillment simulation. So I don’t know, maybe that gives more context.
Dan (54:17): Interesting, okay, that’s a spicy take. We should return to that for another episode because that’s fascinating. And actually, it’s also very philosophically interesting. I think you’re right, when you start thinking about things probabilistically in that way, yeah, the fact that you’re having a great life might provide evidence for this kind of experience machine scenario. But I feel like we’re getting derailed from the main point of the conversation now, which I think is probably my fault for double clicking on that point.
Henry (54:45): So there’s this question about whether the fact that there’s maybe no one conscious on the other end of the conversation—to what extent does that mean that it fails to instantiate the relevant goods that people care about? To what extent does that make it fake?
I think one small thing, I think there’s a perfectly viable position that says, look, a perfectly viable philosophical position that says, look, if there’s no consciousness there, then no matter how much fun these relationships are, they’re not really valuable. I mean, I can sort of see some arguments for that.
But maybe I’m also influenced here by the fact that really since my early adolescence, so many of the relationships I’ve had—I’m talking relationships in the broad sense, friendships and so on—have been with people whose faces I never saw. I spent from my early years a lot of time in pre-Discord chat rooms on services like ICQ back in the day, on online video games, massively multiplayer worlds, before the age of streaming video, where I would form these really valuable relationships with people I only interacted with via text. And in some cases, particularly in video games, we didn’t really discuss our personal lives at all. We were just interacting in these virtual worlds.
Now you could say, yeah, but there really were people, conscious people on the other end there. And sure, yeah, but I’m not sure how much of a meaningful psychological difference that makes for me in terms of my experience of those things. It seems to me that so many of the goods that we get from relationships don’t consist in this deep meaningful connection with a conscious other but consist in things like joint action—in the case of a video game might be going on raids together, having some fun banter together, discussing politics.
I spend a huge amount of time—or not so much these days, back in the day I spent a huge amount of time arguing on Reddit. Would it have made a difference to me if I knew that person I had a really good long political debate with on Reddit, that they were a bot? Well I think it probably would, but I’m not sure whether it should, if that makes sense. It was a valuable discussion for me and maybe it’s just my own prejudice that gets in the way then.
Dan (57:04): Interesting, yeah, I definitely don’t have the same intuition. I think if I were to have what I thought of as meaningful relationships and then discover that actually I wasn’t dealing with a person as I understand it—
Henry (01:03:47): Yes, okay, if you found out that you were talking to a chatbot online and you thought they were a person, that would be dismaying. You would feel bummed out to some extent.
It’s not clear to me that that is primarily to do with lack of consciousness though. In some ways, I think it’s more to do with a loose set of considerations around agency and identity. Talking to a chatbot feels like something like a social cul-de-sac currently. There’s no one going to go away with changed views that they’ll carry forward into discussion with other people as a result of the conversation that we’ve had. It can feel sort of masturbatory in that sense.
But I think if you can think about AI systems, if we imagine AI systems as something like more robust social agents—so you might chat to a given social AI one day and that social AI will be able to carry forward any insights it gleans from that conversation into interaction with other people. I don’t know, as you beef up the social identity of these things a bit more so that it’s not just these masturbatory cul-de-sacs, then my intuitions start to weaken a bit. Maybe they can be valuable. If I’m talking to a chatbot that speaks to other people and can carry forward those insights, maybe there is some value.
Dan (01:05:06): Yeah, that’s so interesting. I feel like there’s a million different things we could be talking about here. And we should say we’re going to have other episodes where we return to social AI, where we bring on guests and so on.
Maybe two things to end on. One thing we’ve already touched on, and I think it connects to what we’ve just been saying, but takes it even further. One commercial use of social AI, at least as I understand it, pretty fringe use, but not a completely non-existent one, is I think you earlier on called them grief bots? Basically people using AI technology to produce a system that—I don’t exactly know how to describe it, but that exemplifies the traits that they associate with somebody, a loved one, a family member, a spouse, a friend who has passed away.
I mean, that it’s almost like that’s been cooked up purposefully for moral philosophers because it introduces so much, not just weirdness, but moral complexity. I mean, I take it there’s the general baseline issue, which is you’re forming a relationship of some kind, you’re interacting with an AI system, and that’s weird. And then it’s a weirdness which is massively amplified by the fact that you’re interacting with an AI system, but the relationship in some sense is grounded in the perception that you’re somehow interacting with somebody who’s now deceased.
I don’t even know to be honest how to describe it, but I understand some people are doing this. So what’s your take about what’s going on?
Henry (01:06:51): Yeah, so I expect griefbots to be one of the big applications of social AI. As you mentioned, it’s relatively niche at the moment. And I think a lot of companies are very scared to go anywhere near this, but I can see that changing quite rapidly.
Just for context here, it’s worth noting that chatbots fine-tuned on real-world individuals—there’s a lot of them that suggest they can be really accurate in terms of capturing the kinds of things that people would say, their modes of conversation and so on.
So one famous experiment was the DigiDan study. This was using GPT-3, so really primitive language model by modern standards. But a group of people including David Chalmers, Anna Strasser, Matt Crosby and others basically fine-tuned GPT-3 on the works of Daniel Dennett—I’m sure most of your audience know Daniel Dennett, one of the greatest philosophers, sadly died a couple of years ago. This was shortly before his death though that he did this.
And then they got Dan’s friends and colleagues to pose questions to both Dan himself and the DigiDan bot. And they generated four responses from the chatbot and they had Dan’s response there as well. And users, Dan’s friends and colleagues, were pretty much close to baseline, close to bare chance, telling which responses were from Dan versus the chatbot.
So this is just to emphasize that appropriately fine-tuned chatbots could do a really good job of simulating the kind of things that a person would say in response to a given query.
So let’s just imagine that you do have this category of griefbots that can provide an accurate simulacrum of a deceased person. Well, I mean, at the risk of spicy take, I can see lots of really positive use cases for this.
I’ve actually even said to Anna Strasser, one of the people who did the DigiDan study, that if I get hit by a car tomorrow, I’ve discussed this with my wife as well, then she should absolutely have my permission to fine-tune a bot on me. And my wife will give her a perhaps lightly edited or curated set of my correspondence, my social media presence and so on, so that then my kids can talk to this simulacrum of me if they choose to.
I can imagine there being real value, know, if my son or daughter is like 17 and considering, should I go to law school or medical school? I wonder what my dad would have thought of this. That seems like a really potentially positive use case.
So I think griefbots are a really interesting area. And there’s interestingly some studies looking at people’s use of griefbots for therapeutic purposes—if there are conversations you always wanted to have, but didn’t get the opportunity to have perhaps because a spouse or a parent died suddenly, that this could, one phrase that’s used, offer a soft landing for the grief experience. So there are loads of really interesting positive use cases there.
But equally, as you say, it’s an absolute minefield. And I think to a lot of people the whole idea of griefbots just feels like something from a Black Mirror—well, literally there was a Black Mirror episode about this. And I think that it raises some fascinating questions about how that changes the nature of the grieving process, how it changes our very concepts of mortality.
If someone’s physical body can die, but there’s this sort of echo, digital echo ghost of them that is still around, how does that reshape our views about these things?
There’s also some interesting parallels. I had a student who wrote a great dissertation or a great essay about integrating this with the idea of communing with ancestors, which is obviously a really common feature in many different societies, where you might ask your ancestors—we touched on this earlier on—you might ask your ancestors for guidance on difficult questions. Could griefbots be a way of making that into a more concrete experience?
There’s a second angle here, which is even spicier and again, will make viewers think I’m even more of a weirdo, which is: could this actually offer some kind of form of immortality or some kind of life after death? Or continued existence after death, I should say.
Now, we could do a whole episode on this as well, but for what it’s worth, as someone who is very deflationary about personal identity, I’m big on the work of philosophers like Derek Parfit who say that in some sense the self is an illusion or the persistent self is a constructed self, there’s no deep matter of metaphysical fact about whether I survive or not. I could see a good case being made that in some sense, via an appropriately fine-tuned chatbot, there will be a form of persistence of me through that chatbot that might be relevant to mortality considerations.
Dan (01:11:36): Yeah, in some sense. It seems like there’s an issue here, which is using a chatbot to acquire knowledge about what a given person might have thought about a topic. And I take it there are going to be all sorts of questions that arise there. Like to what extent is it going to be reliable as a way of gaining insight into what that person would have thought about a topic?
Then there’s a question of using these systems not just to get that kind of knowledge, but to actually have a kind of relationship with the person. And then there’s something over and above that, which I take it you’re referring to then, which is the idea that in some sense, such a chatbot would carry on the identity of the relevant person. And I take it that relationship thing and that identity thing are connected.
I mean, certainly I think that’s where the real issues and lots of people’s queasiness arises, right? That’s when it seems like a bit of a leap, at least if we’re talking about chatbots as we understand them today. I can imagine AI systems of the future that aren’t merely getting really good at the statistical pattern recognition and prediction when it comes to bodies of text, but that are doing something more substantial when it comes to replicating the characteristics and traits of the relevant person.
But do you really think if you had a chatbot that had been trained on the text that you had produced that it would be in any sense a continuation of you?
Henry (01:13:08): Yeah, I mean, potentially. We could do a whole episode on personal identity here. But broadly speaking, being a bit crude here, the Parfittian perspective, Derek Parfit’s view, is that there’s a certain kind of relation you bear to your future self, a certain kind of psychological relation, that can come in varying degrees. And to the extent that we prioritize anything when we’re about survival, this relation R is the kind of relation that matters. And it can obtain to varying degrees.
Just as I could suffer a traumatic head injury and my behavior would change in some ways, but not others—that would be a sort of continuation of me in some ways, but not others. I think you could say the same for an appropriately fine-tuned chatbot.
Now, as you sort of implied, there will be things that it misses out there. We don’t talk about everything that is relevant to us. There’s more to our identity than just what we say. But again, that seems like a technological problem that, as we move to increasingly multimodal chatbots that can learn not just from what we say online, but how we live in the world, I think you can instantiate this relation, this relevant kind of continuation relation to increasingly strong degrees.
But I guess that’s the central point I’d say here: that survival in this kind of Parfittian view is a matter of degree and a matter of similarity and I don’t see why even if it’s imperfect—to the extent that a chatbot can capture really key features of my modes of interaction—that that’s a kind of survival. That’s a kind of continuation.
Dan (01:14:39): Interesting. Maybe we could end on this point. So I’ve been thinking quite a bit recently about how advances in artificial intelligence might gradually eat away at human interdependence. When people are thinking about the dangers posed by AI, there are the classic loss of control, catastrophic misalignment dangers that we’ve talked about previously. There are also dangers to do with elites, political factions, authoritarian regimes, the military using advanced AI to further objectives in ways that are bad for humanity.
But I think there’s also a category of dangers associated with AI systems doing what we want them to do, satisfying our desires, but in ways that have really knock-on bad consequences. And it’s easy for me to see how social AI might be a little bit like that, in as much as so much of our understanding of the human condition, so much of the societies that we inhabit is bound up with interdependence. We depend upon other people. We depend upon other people for friendship, for labor, for sex, for romance, for art, for creativity and so on.
And it seems like a very plausible path when it comes to advances in AI is these AI systems are just going to get better and better at doing everything that human beings do and in fact are going to get better than human beings at doing all of those things, including when it comes to satisfying the social needs in the way that we’ve talked about in this conversation.
And to the extent that that’s true and we become more and more reliant on these AI systems and less and less reliant on other people, that kind of human interdependence fades away. I think there’s something—I mean, there’s something disturbing about that just from the perspective of thinking about what it means to be human. But maybe that’s not really a serious philosophical worry. That’s just an emotion.
But I also do think, the way I think about it is a lot of the human alignment problem—like how do we align our interests with one another and build complex societies—is precisely this interdependence. Because we depend on other people, we have to care about them and we have to care what they think about us and so on.
And it seems to me one of the diffuse risks and long-term risks associated with social AI is precisely that, that as this technology gets better and better, it’s just going to erode that interdependence, which is really central to the human condition.
I realize that’s a massive thing to throw at you for the final question, but what are your thoughts about that? And then we can wrap things up.
Henry (01:17:15): Super interesting. Yeah, so I think you did this yourself, but I’ll separate out two different concerns here. One is the more philosophical question about, even if this works perfectly, even if we’re all very happy with this future society, has something of value been lost? I think it is a valuable question to ask, but it’s also one that’s hard to answer in a neutral sense. It really comes down to what is your conception of eudaimonia, human flourishing, in a deep philosophical sense.
And then there’s—but I felt like that wasn’t the core of your question. You were asking something more about negative externalities, perhaps more about negative knock-on effects. And I think that is absolutely something I’m also worried about basically just because of social media.
Now I realize this is a debate where you have your own very well-developed positions, but I’ll just offer a quick parallel of two technologies. One is violent video games or video games in general. I think we’ve probably discussed this before, but back in the 90s, there was massive moral panic around negative knock-on effects. The idea that kids growing up in the 90s playing Doom or GTA would turn into moral monstrosities as a result of being exposed to relatively accurate simulated violence.
I don’t think that was a stupid thing to worry about. It just turned out to not be a major concern. It turns out that’s not how the brain works, that we didn’t see massive negative externalities associated with exposure to violent video games.
But by contrast, social media, opposite story. I think there was relatively little panic early on in the days of social media, in the early days of MySpace and Facebook. In fact, I think most of the commentary about social effects of these things was quite positive—the idea of bringing people together. There was an interesting debate that’s quickly been sort of consigned to the dustbin, quickly been memory-holed. People, I remember in 2010, people were talking about how social media meant the collapse of epistemic closure, how these epistemic islands would be all beautifully linked up through conversations on social media. And we’d be able to talk to people with different political views from us.
And that’s basically not happened, that social media exacerbated some of our worst social tendencies, possibly contributed to echo chambers and so forth. I’m aware that you have a slightly more optimistic view here, but I’m just offering that as a parallel for a case of social technology that I think at least in many people’s view has had significant, largely unforeseen negative consequences.
And I think that’s absolutely a legitimate source of worry about social AI. I’m not sure I’d necessarily frame it in terms of dependency or interdependence. I mean, I think different people make different choices about how much they want to depend on others. I don’t think it’s obvious that someone living alone on a ranch in rural Texas or whatever, they grow their own food, whatever, they’re relatively autonomous and independent—it’s not clear that they’re not living a great life. It seems that you can have valuable lives with varying degrees of social interaction and dependency on others.
But at the same time, I do think there are possible dangers, very hard to predict, associated with potentially people getting into islands of social activity where it’s just them and their coterie of AI friends and they don’t see the need to interact with others. The kind of subtle influences that could have on things like democracy, on society—there’s absolutely scope for concern.
Dan (01:20:52): Yeah, okay. We’ve opened several cans of worms to conclude the conversation. I’m aware, Henry, that you’ve got a place that you need to be. So that was so much fun. So many issues and questions, which I feel like we didn’t really even scratch the surface of, but we’ll be back in a couple of weeks to talk about more of these issues. And then over the future, we’re going to bring on various kinds of guests and experts in social AI and connected issues. Was there anything final that you wanted to add before we wrap up, Henry?
Henry (01:21:24): I guess just a couple of quick reflections. Firstly, I think for any young philosophers or young social scientists listening, I think this is just such a rich and underexplored area right now. There are so many interesting issues ranging from griefbots to digital duplicates—models fine-tuned on real-world individuals who are still alive—to issues around de-skilling, dependency, mental health, atomization, loneliness, intellectual property, influence, motivation, persuasion. There’s enough for several dozen, hundreds maybe of PhD dissertations on this topic. So I think it’s just a really interesting and valuable area to work on.
And that’s not even getting into the meatier philosophical issues we sort of just touched on briefly around personal identity, what it means to be human, flourishing, the good life. So I just think this is a really valuable area.
Also worth quickly promoting that I am a unit editor for an Oxford University Press journal series called “AI and Relationships.” It’s called the Intersections Journal Series and I run a project in there. So if any young philosophers or academics have papers on this, feel free to give me a ping on Twitter or to my email if you’ve got anything you want to publish on this topic because I think, yeah, it’s an area where I’m really keen to start seeing more good research.
Dan (01:22:41): Fantastic, yeah. It’s a golden age for philosophy, which is why it’s a little bit strange when you look at so many of the things that philosophers are actually working on. But anyway, that was great. See everyone next time.
(Note: this transcript is AI-generated and so might be mistaken in parts).
Dan Williams: I’m here with my good friend Henry Shevlin and today we’re joined by our first ever guest, the great Andy Masley. Andy is one of my all-time favorite bloggers. He writes at the Weird Turn Pro Substack and he is the director of Effective Altruism DC and he’s published a ton of incredibly interesting articles about the philosophy of AI, the politics of AI, why so much AI discourse is so bad. And he’s also written about the main thing that we’re going to start talking about today to kick off the conversation, which is AI and the environment.
So I think many people have come across some version of the following view that says there’s a climate crisis. We have to drastically and rapidly reduce greenhouse gas emissions at the same time as AI companies are using a vast and growing amount of energy. So if we care about the environment, we should feel guilty about using systems like ChatGPT. And maybe if we’re very environmentally conscious, we should boycott these technologies altogether. So Andy, what’s your kind of high level take on that perspective?
Andy Masley: A lot to say. Just going down the list here. So basically for your personal environmental footprint, using chatbots is basically never going to make a dent. I think a lot of people have a lot of really wildly off takes about how big or small a part of their environmental footprint chatbots are.
There are a few specific issues that the media has definitely hyped up a lot, especially around water, which I talk about a lot. So living around data centers, I think is not as bad as the media is currently portraying. But in the long run, I’m kind of unsure. There are a lot of wild different directions AI could go. So I don’t want to speak too confidently about that. And I also just want to flag that I’m kind of a hobbyist on this. I feel like I know a lot of basic stuff, but I don’t have any kind of strong expertise in this stuff. So I’m very open to being wrong about a lot of the specific takes.
Dan Williams: But I think one of the things that—sorry Andy, just to cut you off—but I think one of the things that you point out in your very, very long, very, very detailed blog posts is that you don’t claim to be an expert, but you just cite the expert consensus on all of the very specific things that you’re talking about.
Andy Masley: Yeah, I do want to be clear that every factual statement I make, I think I can back up. How to interpret the facts is on me. I’m using some basic arguments. I was a philosophy major in undergrad, so I like to think I can deploy a few at least convincing or thought out arguments about this stuff. I’ve also been thinking about climate change stuff since I was a teenager, basically. So I have a lot of pent up thoughts about just basic climate ethics and have been pretty interested in this for a while.
Yeah, not claiming to know more than experts on this. What I am claiming is that I think if you interpret the facts and just look at how the facts are presented in a lot of media, I think a lot of journalism on this is getting really basic interpretations kind of wrong. I remember my first article that I read about this years ago—I think this was 2023—when an article came out that framed ChatGPT as a whole as using a lot of energy because it was using at the time like two times as much energy as a whole person. And at the time I was like, man, that’s not very much. A lot of people are using this app. And if you just add up the number of people using it, it shouldn’t surprise us that this app is using two times as much as an individual person’s lifestyle.
So there are a lot of small things like that over time that seem to have built up. It seems like there’s kind of a media consensus on like, this thing is pretty bad for the environment in general, so we should all report this way. And so any facts that are presented are kind of framed as “this is so much energy” or “this is so much water.” And if you just step back and contextualize it, it’s usually pretty shockingly small actually.
So I have a ton of other things to say about this. I think a part of the reason this is happening is that a lot of people just see AI as being very weird and new. And I agree there are valid reasons to freak out about AI. I want to flag that I’m not saying don’t freak out about AI, but I think the general energy and water use has been really overblown. We just need to compare the numbers to other things that we do.
The Problem with AI Environmental Reporting
Henry Shevlin: Yeah, this seems to me such a problem with the debate. And Andy, you say you don’t claim to be an expert in this, but I regularly interact with academics working in AI ethics and policy who make grand claims about the environment, but just don’t seem to have a good grasp on the actual figures. And I often ask people—they’ll say ChatGPT uses X amount of water or X amount of electricity, and this was before I started reading your blog post and I knew these figures themselves. I’d ask basic questions like, okay, what is that as a percentage of overall electricity use? Or is that just for the training runs or is that inference costs? And half the time they looked at me like I was a Martian. Or I was really saying, “Hang on, don’t you—you’re not supposed to ask questions like that. I’ve just told you, isn’t 80 million liters or whatever, isn’t that a really big number? Isn’t that enough? Why do you need to know what percentage that is of...?”
So I think honestly, you’re one of the very few people in this space who’s actually sitting down and doing the patient, boring work of quantifying these things and putting them in contrast with other forms of other ways in which humans use electricity and water.
Andy Masley: Yeah, and I will flag for anybody else who wants to look into this—I’m very proud of the work I’ve done, but I have to say it’s actually not boring for me anyway. It’s quite exciting to dig into and be like, “Wow, this like, you know, almost everything we do uses really crazy amounts of water” or “Here’s how energy is distributed around the world.” And so I think one hobby horse I’d like to push a little bit more is that a lot more people should be doing this in general. The misconceptions are really wild.
I’ve bumped into a pretty wild amount of people who are experts in other spaces or just have a lot of power over how AI is communicated and stuff like that. I consider them as—a lot of them are just sharing these wildly off interpretations of what’s up with this stuff. I’ve talked to at least a few people who have bumped up against issues where, if they’re involved in a university program or something like that, and they want to buy chatbot access for their students, some of whom are pretty low income and might not buy this otherwise, they’ve actually been told by a lot of people like, “We can’t do this specifically because of the environment.” Specifically because each individual prompt uses 10 times as much energy as a Google search or whatever, which by the way, we don’t actually know—that’s pretty outdated. But you know, I really want to step into these conversations and say, 10 Google searches isn’t that much. 10 Google searches worth of energy is really small.
And it seems like there’s been this congealing of this general consensus on this stuff that is just really wildly off. I think this is one of my first experiences of going against what’s perceived to be the science on this. I remember when I first started talking about this at parties and stuff, when people would be like, “You used ChatGPT, it’s so bad, it uses so much energy,” and I would be like, well, as you know, I was a physics teacher for seven years and so I had a lot of experience explaining to children how much a watt hour is and so I would kind of go into that mode a little bit. I’d be like, “Oh well, it’s not that much if you start to look at it.” But I kind of couldn’t get past their general sense of like, “Oh, this guy has been hypnotized by big tech or something. He doesn’t know that it’s bad.” And so, a lot to say about that.
Henry Shevlin: So just to throw a couple of anecdotes—that completely resonates with my experience. So we had a debate at CFI quite early in the consumer LLM phase, I think shortly after the release of GPT-4. And someone asked in completely good faith, “Hang on, when we were discussing our generative AI policy for students, maybe we should be discouraging students from using it precisely for environmental reasons.”
And I did some really rudimentary napkin math looking at the sort of energy and water footprint of a single paperback book versus a single ChatGPT query. And I came up with something in the region—very loose estimate—that you could get 10,000 ChatGPT queries for roughly the equivalent environmental impacts, carbon impacts, and water impacts of a single paperback book. And I was like, “Hang on, are we going to tell students to stop buying books as well? Because that would seemingly be consistent.”
But I wanted to also ask, there is this idea that I found very useful of “arguments as soldiers,” where basically if you are seen to be arguing against an argument associated with a political position, then that can easily make you seen to others not just that you’re critiquing the argument, but you’re critiquing the whole view that it’s coming from, right? And so I think this often—I see this quite a lot in environmental discussions where if you say, “Actually, this isn’t a particularly good argument for why we should be concerned about loss of biodiversity,” or “This isn’t a good argument for why ocean warming is a problem,” and people will immediately sort of assume, “Oh, you’re one of them. You’re one of the bad guys.” Right?
So I’m curious if you think that influences the dynamics here or why you think you’ve encountered opposition when you try and throw numbers around here.
Putting ChatGPT Prompts in Context
Dan Williams: Could I just say, Henry, before we jump into that conversation—because I think that is a really important conversation—I’m just conscious of the fact that I’ve read Andy’s 25,000 word blog posts, and you’ve read them as well, Henry. Andy, you’ve written them, so I can only imagine how much in the weeds of this topic you actually are. But if someone’s coming at this for the first time, I think they’ve come across this, as you put it, almost a kind of consensus view, at least in some parts of the discourse, that using ChatGPT is terrible for the environment.
And your claim, as I understand it, is actually that’s not really true. And you give loads of different arguments and you cite loads of different findings that sort of put ChatGPT use in context. Maybe it is just worth starting with something we’ve already touched on, but I think it’s helpful context for the entire conversation. If we’re thinking about one prompt to ChatGPT, I mean, you mentioned sometimes people talk about this as being sort of 10 times the energy use of a Google search, but there’s some uncertainty about that. But in terms of, okay, let’s acknowledge there’s some uncertainty. If we want to place a ChatGPT prompt in context, what’s the quickest way you know of conveying the fact that, hmm, we probably shouldn’t be as alarmed about the environmental impact concerning this as many people think we should be?
Andy Masley: Yeah, ultimately, my personal way of conveying it is just asking how many ChatGPT prompts throughout the day would the average American need to send to raise their emissions by 1%, basically. And so I think the question actually explodes in complexity really quickly because you realize people have these wildly different definitions of what it means to be bad for the environment, where some people will believe that being bad for the environment means it emits at all. And so in that case, literally everything we do is bad for the environment. And so ChatGPT falls into that category because it’s something that we do. And I don’t think that’s satisfying.
And so I think a better way of talking about it is like, will using this thing significantly raise my CO2 emissions? And I try to start there and say, okay, my best estimates right now, if we include every last thing about the prompt—because people will always bring up like, it’s not just the prompt, it’s the training, it’s the transmission of the data to your computer and things like that—once you add all that up, it seems like a single ChatGPT prompt is about one 150,000th of your daily emissions or something like that.
And so if you send like a thousand ChatGPT prompts—an average, median prompt, I mean, obviously there’s a lot of variation, this is something else we can get into—but if you send a thousand ChatGPT prompts throughout the day, you might raise your emissions by about 1%.
So I’ll intro with that and then say, but if you were spending all this time throughout the day just poking at your computer, making sure you send those thousand prompts, it’s very likely that your emissions are actually way lower than they would have otherwise been because your other options for how you spend the time—you could be driving your car, you could be playing a video game, you could be doing a lot of other things that emit way more.
And so on net it seems actually almost physically impossible to raise your personal emissions using ChatGPT because the more time you spend on it, the less time you’re doing other big things basically. So I try to frame it that way.
Sometimes I’ll, in more of a joking way, when people will be like, “It’s ten Google searches,” I’ll kind of just sit back and be like, “If I told you a few years before ChatGPT came out that I had done a hundred Google searches today, would your first thought be like, ‘Man, that guy doesn’t care about the environment. He’s a sicko. He hasn’t got the message?’” Just trying to frame what they themselves are saying in the context of everything else that we do and saying, would you have ever freaked out about this before?
ChatGPT has this kind of negative halo of being perceived as this environmental bad guy. And then the conversation will usually go in a lot of different directions where maybe it’s not actually about your personal footprint, it’s about these data centers. But at least I try to keep it limited to the personal footprint at first because that’s what they’re initially interested in and say, yeah, it’s about a hundred and fifty thousandth of your daily emissions.
The Uncertainty Around Different AI Uses
Dan Williams: And just to double click on that. So as you’ve acknowledged, there’s also some uncertainty, I guess, about the actual energy use when it comes to a single prompt, because it’s partly like, how do you calculate that? What are you considering as part of the cost of a single prompt? But also these days, especially, you can use state of the art AI to get a quick text based response. You can also use things like deep research, where it’s going to go away and produce a detailed research report. You can use things like Sora 2, where it’s generating really detailed synthetic video and so on.
So I guess one question is, how do we know at all the energy use or the carbon emissions associated with a single prompt? How can we have any confidence when we’re answering those questions? And secondly, does your skeptical take apply to all uses of this technology, or is it just the basic kind of text-based use of systems like ChatGPT?
Andy Masley: Yeah, there’s a lot to say. I do have to kind of—my main big piece is called “Using ChatGPT is Not Bad for the Environment” and not “Using AI Isn’t Bad for the Environment.” Because I could imagine very extreme edge cases where it is. I don’t actually know anything about video actually—our information about how much video uses is surprisingly scant. I think for these larger systems, there’s a part of me that wants to make the move of saying it would be weird if these AI companies were giving everyone really, really massive amounts of free energy.
I have access to Sora 2—I’ve made some goofy videos, it’s pretty addicting honestly, I have fun—and there’s not really an upper limit to my knowledge anyway that’s approachable for me of how many videos people can make in any one day. There probably is, I just haven’t looked into it. And it would be a little bit surprising to me if that were like “we’re giving you as much energy as you use in a day” or something. That just seems like way too much.
But that’s kind of hand wavy. The truth is I don’t actually know too much about video, which is why I haven’t written about it as much. For longer prompts, we’re also a little bit in the dark. The best we have right now is there was a Google study a few months ago that tried to really comprehensively analyze how much energy and water the median Gemini prompt uses. And the takeaway they had was, okay, it uses about 0.3 watt hours, which is incredibly small by the standards of how we use energy. And about 0.26 milliliters of water, which is about like five drops. And I can go on forever about how that’s not measuring the offsite water, which maybe raises it up to like 25 drops of water, but it’s still not very much.
So it seems like a lot of other estimates for how much energy a chatbot prompt uses were converging on about 0.3 watt hours anyway. And so this seems like probably our best but pretty uncertain guess right now for how much the median chatbot prompt that someone engages with actually uses. And again, there are huge error bars on this because if you input a much larger input or if you request a much longer output that takes a lot more thinking time for the chatbot, that can go up by quite a lot.
But a weird thing that happens there is that if the prompt takes longer, it’s usually both going to produce way more writing that takes you longer to read, and often it’s higher quality. I’m sure you guys use chatbots a lot. You’re familiar with deep research prompts—they’re incredible to me. There was a time when I was sympathetic to people saying that chatbots don’t add value. It’s kind of hard for me to understand how someone can come away from a deep research prompt on something they’re trying to learn about and think this is adding nothing. It’s definitely adding some kind of value.
And so if you’re looking at this longer period of time with much more text, and you just factor in how long it takes you to read that text, it’s actually not using too much more energy than a regular prompt if you measure by the time that you will personally spend reading the response, basically. But again, wildly uncertain about this.
I was trying to do a calculation a while ago where it was like, okay, if I were trying to max out how much energy I use on AI to harm the environment as much as possible, what would I do? And I was like, well, I guess I would just start hammering out deep research prompts and video stuff. And even then I couldn’t get it to be especially high. And if I were trying to harm the environment as much as possible, it’s actually quite good that I’m using AI and not just circling—the best thing to do there would be to just get in my car and start driving in circles.
So it’s very hard for me, even with deep research stuff, to come away thinking that this is gonna be a big part of your personal emissions, or even a small part. It seems like an incredibly tiny part. So yeah, hope that was useful.
Henry Shevlin: So one of my friends who’s a teacher at high school actually struggles to try and get his students to use ChatGPT more, or at least some of them.
Andy Masley: That’s like, he’s like the one teacher in the world who is facing that problem. That’s funny. That’s good.
Henry Shevlin: Exactly. He’s like, “No, please, please use this tech, learn how to use it.” But he says a lot of them are reticent because they worry about environmental footprint. And I guess, reflecting the priorities of teenagers, he’s found it helpful to frame it in terms of how many short form videos a single prompt is worth. And it’s way less—it’s like way less than one. It’s like, if you just watch one fewer TikTok video today that will cover all of your ChatGPT requests for the day.
Andy Masley: Yeah. This is another general point where I think that this AI environment stuff is probably the first time that a lot of people are thinking about data centers and the fact that the internet in general—I think it was news for a lot of people that the internet uses water. It’s very easy to think about, you know, this is just this ephemeral thing that beams down to my phone and it doesn’t really exist anywhere. And I think a lot of people weren’t actually aware that there are these giant buildings that house these large computers that host the short-form video content that you like.
And yeah, I think comparisons to everyday things like that matter a lot. Especially as a former teacher, I know that most students most of the time are just consuming short form video content. I remember one of my best students came up to me a few years ago and was like, “Did you know that no one in the school reads?” And I was like, “No, I didn’t.” And he was like, “Literally every student gets their information from TikTok in school, and that’s it.”
And I want to be able to take students in that situation and be like, “Well, you’re using a data center right now. Those short form videos are also using energy and water.” And there, it’s also not very much. And it shouldn’t shock you that something else that you were doing online also uses energy and water. And I think just getting that across can be pretty powerful. But yeah, a lot to say about that.
The Data Center Microwave Analogy
Dan Williams: Can I ask, Andy, just following up on that concerning the data center, the role of data centers when it comes to using these chatbots? I think you make a number of really important points about how to think about data centers and how not to think about them. Because I think often people look at these data centers and they seem to be using up enormous amounts of energy. And there are also these issues which you’ve touched on concerning water use, which I take it as somewhat distinct, although connected. And your point is, well, it’s true that if you just focus on the data center itself, it looks like it’s using a vast amount of energy, but you need to put that in the context of the fact that—in fact, why don’t I let you explain it? You’ve got this really nice analogy with a kind of big microwave that you imagined. So maybe that’ll be good to...
Andy Masley: Yeah, yeah, yeah. So my basic point is that if you just look at a data center without any context, and you just say this is a really large, weird looking building, it’s basically just a box that plopped down in the middle of my community and is just using huge amounts of energy compared to the coffee shop down the street, it can look really ridiculous. And you might infer from that that what is happening inside of that data center must be really wasteful because it’s using so much energy and water. Surely all of that can’t be being used for good—it’s probably using a lot of energy and water per prompt because I’ve heard people talk about ChatGPT using a lot of energy and water per prompt. So these things are really wasteful and we need to get rid of them basically.
And so the thing that’s wrong about that is that the average person I think has no idea just how many people are using and interacting with data centers at any one time. It can be in the tens to hundreds of thousands of people. What a data center ultimately is is just an incredibly large, incredibly efficient computer-sized computer basically. And it’s designed so that people around the world can all go into it and interact. And every time you’re using the internet, you’re basically using a data center in the way that you would a computer. And all of those people are invisible, because they’re around the world and all their individually incredibly tiny environmental impacts are so, so concentrated in this one building, which among other things actually makes it really efficient. It’s very good to pile a lot of these types of small tasks in one place because that means that they can really optimize where the energy and water goes.
And so the reason why data centers are using so much energy—outside of training, we can talk about training separately for AI models—but the reason why data centers that house inference, just normal answering of the prompts, the reason why they’re using so much energy is basically entirely because they’re serving so many people at once, not because the individual things happening in them are inefficient.
As Dan had mentioned, my national microwave example—there are a lot of other things that we all collectively do that use huge amounts of energy in total that are kind of invisible to us because they’re all spread out across our individual homes. So I did a very rough back of the envelope calculation and the best I can guess is that every day all American microwaves together probably use as much energy as the city of Seattle.
And I think that if we had concentrated all of those microwaves in one place and we could somehow beam our food into that microwave and then get it back—very similar to how data centers work, we kind of beam our computing somewhere else and then get it back—there would basically be a single really gigantic city-sized microwave that would be guzzling up huge amounts of energy and I think it would probably draw a lot of protests and opposition because people would be like, “This thing is using as much energy as Seattle? There’s this new tech bro way of heating food and we don’t need that, we have ovens already.”
And I think what they would be missing is that this amount of energy looks really small in the context of your just everyday home and it’s like the thing that it’s doing wrong is that it’s just very visible.
And I think if we just make more things visible, it becomes clear how much energy they’re all using and how much they still don’t really add too much to our total amount of energy. And I think one of the big reasons why people are upset about data centers is that they’re actually just making these tiny little things—or the aggregates of these tiny little things—very visible.
Obviously there are still some problems. Because they’re concentrating so much in one place, they can create some problems for local communities. I’m not saying data centers come with zero issues or things to think about in the way that any other large industry does. But we have to consider that this thing is serving tens to hundreds of thousands of people at once. And believe me, it can be overwhelming looking at some of these data centers. For me, even I’m like, “Wow, it’s truly insane how much energy it’s using.” We just need to keep in mind that the thing that it’s doing is aggregating these incredibly small, individually very efficient tasks rather than just blowing through huge amounts of energy and water for no reason.
The Water Issue Is Even More Overblown
Henry Shevlin: Can I actually ask to drill down a little bit more, no pun intended, on the water issue? Because the impression I get from your blog, and correct me if I’m wrong, is that you think the water issue is even more overblown than the electricity issue, that there are maybe greater concerns around electricity.
Andy Masley: Very, very much. Yeah. So basically, I think specific utilities have come out and said that we believe a part of the reason why we’ve had to raise rates is because of data centers, because we have to build out new infrastructure to manage these massive amounts of new energy demand from data centers. And I’m also pretty convinced that is lower than a lot of people think. Energy prices in America specifically have risen quite a bit over the last few years for a lot of different reasons. And I think a lot of people are sometimes projecting all of that onto data centers, where it’s actually mostly Russia invading Ukraine and driving up the price of gas. That’s a whole separate conversation.
But yeah, the water issue I think is actually just really wildly overblown. I can’t find a single place anywhere where water prices seem to have gone up at all as a result of data centers. And almost every news article I read about data centers will literally—most articles that I read will kind of frame data center water use as literally just a very large number of gallons of water basically.
They’ll be like, it’s using 100,000 gallons of water per day. And the reader’s kind of left on their own to be like, “That sounds like so much. I don’t have anything to compare that to except how much water I personally drink.” And you know, I just drink a few glasses a day. So that sounds ridiculous. And in a lot of responses to my writing, I’ll see people be like, “100,000 gallons of water per day could be used for people, but instead it’s being used for AI.” And I think they’re kind of internally comparing this water use to just their everyday life.
And if you actually just step back and look at how much do data centers use compared to all other American industries—a lot of American industries and commercial buildings and recreational buildings will actually also use vast amounts of water for a lot of different reasons. Water is very useful for a lot of different things. It’s actually very cheap in America. America has some of the cheapest water rates of any very wealthy country.
And yeah, most places actually—the main issue they have isn’t that there’s a lack of raw access to water in America. It’s more that their infrastructure for delivering water is aging. And so if you compare data centers to golf courses—I don’t have the exact stats on me right now—but even if you include all data centers in America, not just AI, they seem to be using like 8% of the water of just golf courses specifically. And again, I’ll need to circle back on this. I don’t have the exact numbers in my head right now. But it’s very easy to just Google “how does this compare to other things?”
And there are some places where data centers are using a significant proportion of the water in a local region. There’s this one city in Oregon called I think it’s pronounced The Dalles—I’m not actually sure, should really know that—but they basically have a really large Google data center in their town that a lot of headlines will jump on as using 30% of the community’s water.
And that sounds really big until you read that this place is about 16,000 people. And if instead of reading this as “this weird new thing,” you just interpret this as “the main industry in the town,” it’s like—if there were a large college or a large factory in the town that were using 30% of the community’s water, I think the average person would say, “This is just a pretty normal thing.” But I think people have this internal sense that any water or any physical resource that’s used on a digital product like AI is wasted.
Like all of this valuable community water is just being blown into nothing because it’s being used on a digital product rather than something physical people can use. And so another general theme of my writing is that people really need to get over the sense that it’s always sinful and wrong to use a physical valuable resource like water on a digital resource. Separate from your beliefs about AI, digital resources more broadly are very valuable because information is valuable.
So yeah, a lot more to say about the water stuff. But I’m just pretty convinced that if you see a news article that’s scary about water and you literally just Google, “How much water is this compared to other industries?” Not compared to me personally, but just how does this compare to golf courses and farming and factories and things like that? It’s pretty easy to find, oh, there’s a car factory that uses more water, or there’s a golf course or other things. So yeah, a lot to say about that, going off in a bunch of different directions here.
Steel-Manning the “Climate Crisis” Objection
Dan Williams: I think it would be helpful if Henry and I throw some objections to you as a way of sort of clarifying and stress testing your perspective. I mean, the first one, this isn’t really an objection, but I think it’s probably going to be going through the minds of lots of people and why they’re going to be skeptical of this kind of take. And it’s just something like, look, there’s a climate crisis, there’s a climate emergency, there’s this huge industry, this growing industry where there’s a vast amount of investment going into it. It does use a lot of energy. We should expect that to just increase over the next few years and decades. And it sounds like what you’re saying is, just chill out everyone, there’s nothing to worry about here. Is that—so, I mean, this is not really an argument, but it’s just me trying to put myself into the headspace of someone who hasn’t read your blog posts, they’re listening to what you’re saying for the first time, they’ve heard that this is terrible for the environment. It seems like it must be terrible for the environment, just on the common sense ways in which we think about this topic. If I’ve got that kind of view, what would you say to me?
Andy Masley: Yeah, I mean, first of all, it’s totally understandable to be really concerned about this if you first hear about it. And also, again, usually in these conversations, I want to flag that it’s totally understandable that a lot of people are skeptical of me at first, because I’m a guy with a Substack, basically. So they should actually just examine the arguments for themselves, see the sources I’m using and see for themselves whether it makes sense.
But yeah, basically something I want to flag here is that a lot of people—first of all, 100% where I would describe us as living in a climate crisis. I’m quite concerned about climate change. There are a lot of little nuances there where I don’t think climate change is gonna end civilization tomorrow, but I also think there are a lot of tail risks that are obviously just really concerning and I don’t actually think we’re on a guaranteed path to do a good job with climate change in general. And so there’s a ton of reasons why I’m totally sympathetic to this take and if you see some new massive use of energy that you personally don’t think is valuable at all, totally makes sense to be really worried about that.
I do want to flag that I worry a lot about how people think about the idea of a new source of emissions basically, where I think for a lot of people they kind of see a lot of emissions that are currently happening as kind of in the past somehow or they’ve been locked in. So as an example, it seems like if a new data center pops up and it’s emitting some new amount that wasn’t there before, it seems like people kind of treat this as this special, unique, like, these emissions are much worse and they stand out much more than the emissions from cars because cars are kind of in the background.
And I don’t want to go out and make the claim that these emissions don’t matter so much, but I do want to make the claim that it doesn’t really matter which emissions are new and which aren’t because in some sense, every day we wake up and emit huge amounts of new carbon dioxide into the air, mostly from much more normalized everyday things that we do. And so I don’t really want—I don’t want AI to receive zero scrutiny, but I also want its emissions to receive the same scrutiny or the same level of proportional scrutiny as other normal things that we do.
Even though, say, a new AI data center might open up, if it’s only emitting one thousandth as much emissions as the cars in the local city, I think it should still only receive about one one thousandth of our attention or something like that. There’s another interesting argument here where you can say because emissions are new, they’re maybe more malleable and changeable than cars. But I kind of worry that this locks us in too much to old ways of doing things. And it’s in some ways kind of an inherent argument against starting any new industry in America whatsoever, where any new industry is going to come with quote unquote new emissions.
And so I think my claim isn’t that, yeah, again, it’s not that we shouldn’t worry about this at all, but I just want these to be compared to all the other ways we’re emitting. And when we’re thinking about what to cut, we should think about the things that will actually cut emissions the most. And the fact that some emissions are quote unquote new shouldn’t really blind us to all the ways that other emissions have been normalized, if that makes sense.
Henry Shevlin: So developing this line of objections, you mentioned golf courses, but golf courses to be fair do get a lot of hate already.
Andy Masley: They’re crazy. Yeah. I mean, to be clear, golf courses use unbelievable amounts of water. So I do want to flag that there are a lot of other comparisons you can make. But sorry, go ahead.
Henry Shevlin: No, it’s interesting, right? I think there’s lots of other stuff that people would say, “Okay, sure, we don’t need—maybe golf courses are even worse than AI, we should get rid of golf courses.” But there’s lots of other stuff that we really need. Obviously we need water for agriculture, we need cars and trucks to bring our food to supermarkets. We don’t really need this AI stuff in the same way. So its carbon footprint and water footprint is more concerning than the carbon and water footprint associated with haulage or farming or other core goods as they see it.
Andy Masley: Yeah. This is an interesting question that brings up how much we should worry about the emissions from something that we see as basically useless. Where I think a lot of people are really kind of thrown off in these conversations where they’ll be like, “No matter how little AI uses, and even if Andy says it only uses one 100,000th of my daily emissions or something like that, literally all of its emissions are wasted or bad because AI is useless.”
And I don’t really want to get into conversations and debates about whether AI is that useful because people just go in so many different directions here. So I kind of just want to bracket that and instead say that most of the problem with climate change is actually that—limiting this to climate change for now and can talk about water separately—but the big problem with climate change is mostly, most of the time, that the ways that we use fossil fuels are actually incredibly valuable to us. And it’s not that we’re wasting all these emissions on these things that don’t matter for us. It’s like driving my car makes my life much easier. Flying makes people’s lives much easier.
And it doesn’t really make sense to really, really hyper-focus on small ways that we use energy just because it’s all wasted. That obviously matters and it doesn’t count for nothing. And if we are wasting a lot of energy, yeah, if I thought AI were useless, I would want to generally shut it down and I would be more worried about AI and the environment to be clear. But I also don’t want us to lose sight of the fact that if I do something that’s useless like AI and it uses one 100,000th of my daily emissions, and then I drive my car and it’s very useful to me, I still think driving my car is actually worse for the environment by any meaningful measure, basically.
And then on the food thing, I got a little bit snippy about this, as a vegan of 10 years, where if I’m being lectured to about not using AI by someone who just ate a burger or something, I want to be like, well, something like half of the agriculture, half of the food that we grow in America is used to feed animals that we then eat. And so I’ve cut my personal agricultural water footprint by 50%, and that’s actually quite a large part of my total water footprint. And so a lot of this food that we see as inherently—”people really need to have access to steak and pork and chicken”—I don’t agree with. And I want to say that there are actually more promising cuts we can make there. I don’t talk about that so much just because I think the main problems with animal welfare are not its environmental impact, but it is kind of a nice win on the side.
And so, even there, if you think AI is completely useless and you also really like eating beef, those are both fine, but you need to be able to say the beef does actually matter more for climate, even though it’s providing you value in a way that AI doesn’t, if that makes sense. So that would be my general take. Again, not to not worry at all. If I thought that all these massive data centers were providing nothing of value, I would say, yeah, we should shut these down. These are bad for the environment because they’re adding nothing. But even then, they wouldn’t be as big of a climate problem for me as cars or animal products or other things.
Henry Shevlin: I guess I’d also—sorry, do you want to go ahead, Dan? I was going to just add, I think also when people are drawing this comparison between AI usage, ChatGPT usage versus food or vital infrastructure, right? Maybe that’s not the kind of marginal comparisons to be making. They should be thinking, well, you know, if we’re going to cut, what is the most expensive and least rewarding thing that I do? Maybe it’s that 200th short form video that you watch at 3 a.m. when you really should have gone to sleep an hour ago, maybe it’s not turning off your computer for the convenience of leaving it running so you don’t have to go through a startup sequence. So maybe again, that’s not an apples to apples comparison thinking about the most essential things we do versus AI.
I had another objection to throw your way though, which is how about just this worry that in a lot of cities around the world, but I guess particularly in places like Arizona, there’s just this real risk of running out of water. Home building projects are getting canceled or frozen because there’s just not enough water to go around. Should we really be building data centers when we are encountering these hard limits in terms of actual water supplies?
The Arizona Water Situation
Andy Masley: Yeah, a ton of stuff to say about that. I mean, first of all, I don’t want to lecture city planners and would defer to them on this stuff. Again, just some guy here and I definitely don’t think my position is we should just build data centers willy nilly. I kind of see these as equivalent to very large factories that have very specific resource and energy demands. And I wouldn’t say we should just plop a large factory down literally anywhere and not care about the environmental impacts.
I do have a lot of thoughts about Arizona specifically, just because this comes up a lot in the data center conversation because a lot of data centers are being built there and it’s an incredibly weird situation, because it’s in the middle of a desert and people will say we shouldn’t be building anything new that uses water in the middle of a desert like Arizona. That’s really bad for water access. I’m very liable to ramble on this stuff. So please stop me if I go too far here.
But something really weird about a lot of areas in Arizona where data centers are being built is that the water is already being pumped in from hundreds of miles away. The general situation with water is actually very weird there already. This is actually a place where water access and environmentalist concerns about water really, really come apart. Because if you care about the fragility of local water systems, the Phoenix area has grown really fast in the past 20 years or so. It has really rapid growth. If your goal is to make water for every new person who comes in as maximally cheap as possible, I think that’s admirable. I’m more on team “technology is good, people should be able to live where they want” and stuff. But I also will sign off that this will create some problems for all the water systems that we’re pumping water from a hundred miles away. This is not exactly the most environmentalist thing we could be doing to keep Phoenix’s water rates as low as possible.
So I think the first thing to say is that this is a place where environmentalists and equity actually really come apart. And I think if you’re a really hardcore water environmentalist, I think your first take should be, people should not be living in Phoenix. People should move somewhere else. So there’s that.
Secondly, a lot of other industries in Phoenix use a ton of water. Best guess I have is that all the data centers in that area are using something like 0.2% of the water in the Phoenix area. Again, I would need to go back on this. But again, Phoenix has a ton of golf courses which is crazy to me. I don’t know if either of you have ever been out in that area but it can be really eerie because you can be in this barren desert and just you stumble on this lush, almost sickly green amount of space that’s just been artificially created out of this water that’s being pumped from far away or pumped from local aquifers and stuff.
Again, what I’m saying, I could be getting some of this wrong. I don’t know exactly how Phoenix gets its water. So don’t quote me on this too much and people listening just do your own research on this.
But yeah, basically any argument that says that data centers should not be built in Arizona seems kind of like an argument that Arizona should not have any industry. And ultimately, if we want people to be able to live in these places, we need some kind of way of supporting the local tax base. Data centers don’t provide very many jobs. This is kind of another issue with them where they use a lot of resources relative to the jobs they provide. But at the same time, they’re usually providing very large amounts of tax revenue to whatever locality they’re in. My best guess right now is that data centers are using 1/50th of the water that golf courses use in that area of Arizona, but they’re actually providing more tax revenue on net than the golf courses are just because they’re part of the single most lucrative new industry in America right now. And so the locals are benefiting quite a bit.
And so my claim is that if you’re against data centers being built in the desert, you’re also against the city existing in the desert in the first place. And that is a legitimate take, but you need to be clear that people’s water bills should not be made as low as possible if that’s your goal ultimately. So yeah, I’ll have to say about that.
“Every Little Bit Counts” and Why That’s Wrong
Dan Williams: I really want to circle back to this point you made about the utility of AI and how people’s views about the utility of AI affect how they think about this topic. And I think that would be a nice segue into another set of issues. But just to sort of conclude this discussion specifically about AI and the environment, I think some people might say, “OK, let’s suppose you’re right that actually the environmental impact of using ChatGPT is much lower than many people have assumed. Isn’t it the case that every little bit counts?” I think you’ve got a kind of interesting response to that kind of intuition that many people have, which is that, okay, maybe it’s not such a big aspect of our energy use overall, but if there’s a climate crisis, then shouldn’t we pay attention to even small things like this?
Andy Masley: Yeah, this brings up—I very strongly recommend for anyone who is interested in doing more climate communication and thinking about proportions like this, there’s a really great book called Sustainable Energy Without the Hot Air by David MacKay. And the introduction of that book has this really great long rant by him of that exact quote, “every little bit counts” basically. And I think that the quote “every little bit counts” is actually really quite drastically bad for the climate movement overall for a lot of different directions.
Something weird about this conversation is that back when I was in college, at least in the conversations I was having, I feel like everyone was kind of on the same page about this where people are talking about climate change as this looming crisis. A lot of times they’ll compare it to being in a war where we’re in this war where we have to strategize and make all these really complicated moves in society to make it go well. And the quote “every little bit counts” basically implies that you can kind of just shoot off in random directions and be like “I’ll just cut this, this, this, and that, and I’ll feel good about climate change,” regardless of how much emissions I’m actually cutting altogether.
So one comparison I make is different people making different decisions about what to do for climate change, where one person cuts out—he’s addicted to ChatGPT, he uses it a thousand times a day, he can’t stop, and he makes the noble sacrifice for climate change where he cuts it out. Another person decides that she’s going to try to keep a nuclear power plant open for another 10 years and a third person goes vegan for a year.
And best rough estimate is, even if this other person who’s trying to keep a nuclear plant open for another 10 years, even if she’s working with 500 other people to do this and their impact is divided by 500, she’s still saving tens of thousands of times as much CO2 from being emitted as the vegan even, where there used to be this general idea that individual lifestyle changes don’t really add up compared to big systematic changes we can make to the grid overall.
And so if you were approaching these people, I think you should tell the vegan and the person addicted to ChatGPT, if you are going to focus this much attention on the environment, you really need to strategize about what you’re doing and what you’re actually cutting because the difference in what you can do is on the order of millions of times as much.
I think people sometimes mistakenly think that their impact on the environment is either a little, a medium amount or a lot. And using ChatGPT a lot can bump you up from a little to a medium. And what’s actually happening is there are huge orders of magnitude difference in the interventions we can make. And literally just giving people the advice, “every little bit counts. That’s what matters,” I’m worried kind of gives a lot of status to those really tiny moves that just don’t actually make a dent for climate change at all.
Like going back to the war comparison, if we’re in World War Two and we were thinking about how we can win and I was like, “I’m gonna do my own thing, every little bit counts. So if I’m strategizing about how to win, maybe I can just build a little wall out of sticks and that will...”—in no other situation that is an emergency, do we think “every little bit counts” basically. I think this is just a very bad way of thinking and I’m influenced a lot by basic effective altruist ideas here about you should really target interventions that do a lot of good because the differences in interventions are on the order of hundreds of thousands or millions of times as much good sometimes.
And so yeah, I basically worry that the “every little bit counts” thing is basically a way of peppering your life with random amounts of guilt that you then assuage and you mostly don’t emit less than you were before.
AI Utility and Hostility Toward the Technology
Dan Williams: I think it’s just on the point I mentioned about the utility of AI and coming back to that conversation. So I think it’s worth saying, I use ChatGPT and Gemini and Claude all of the time. I get enormous value from these systems. So do you. I know that Henry does as well. I mean, we’re pretty weird, I think, in the fact that we are very pro the utility of these systems. And I think many people not only do they think actually modern chatbots are kind of useless, they hate modern chatbots. They think they’re actively harmful, right? They’re plagiarism machines, they’re slop generating, stochastic parrots, they’re enriching these sociopathic tech billionaires. And if you’re coming with that web of associations when you’re approaching the topic, it’s gonna be a very different kind of conversation than if you’re like us, where actually we’re pretty positive about all of the different use cases of that.
So I’m interested in your view about that. I’m also interested in what Henry thinks about that. Many people are approaching this specific conversation, but AI more generally, with just loads of hostility towards this technology. And I think we need to kind of acknowledge that. And I’m wondering what you two think about that as a driver of how people are thinking not just about AI and the environment, but the entire topic.
Andy Masley: It’s definitely a noticeable part of almost every conversation that I have about it where I think if I talk about AI doesn’t use this much water. A lot of people read that as me saying there’s this sinful evil technology that’s ruining society. And I, Andy, want to sacrifice this precious resource and just burn it up forever for the sake of this evil thing. And I think that gets at something really primal, some basic reaction they have to “there’s this precious thing that keeps us alive and you want to sacrifice it to this massive social catastrophe basically.”
So most conversations about the environment I try to frame as saying we need to kind of separate out how bad you think AI is and instead just ask, if you were talking to an environmentalist and you were being honest with them and say, “Where in the world should you work to do the most good for climate? What’s actually going to really move the needle, regardless of how much evil you think AI is doing?” Its evilness isn’t really a good way of determining “this is the thing I should work on for climate.” I would almost always tell anyone doing this, you need to help with the solar scale out and you need to build out new storage systems for energy and things like that and just this is gonna be what moves the needle on climate not preventing one additional data center from coming up. It just in terms of impact it just seems to completely dwarf the other one.
But yeah in terms of the value, I do try to be clear in my writing that I’m trying to write this for a general audience. If you hate AI I want to convince you as well that this isn’t the biggest deal. But that being said, man, it’s really difficult to communicate just how useful these tools are to me. I think a lot of my personal blogging success has been because I have these little robot research assistants where—and when I say that I want to clarify that I’m double checking every source—but basically I think of ChatGPT as a little helper where I can be like “Okay you sometimes lie to me and that’s okay. I understand that you just have a compulsion to do this but I’m going to ask you to assemble all these external resources for me on a question so I can just use a thinking prompt and be like, assemble a ton of information on how much water Arizona uses.”
And it comes back with sources. And I’m like, “Thank you. I don’t totally trust you. So I’m going to read all these. But look, they said what you were saying. That’s good this time.” And that on its own, even if the chatbot lies to me one in every five times it does that, is still just such a phenomenal upgrade from what I was doing before. So I’m personally just like, I really think I wouldn’t have had nearly as much success as I’ve had if I didn’t have these little research assistants to help me with stuff.
That’s just one of the few things I use it for. But Henry, did you want to chime in about your thoughts on this?
Henry Shevlin: Yeah, I’m very much on the same page. I guess I find it frustrating that so many positions get bundled together that I think would be worth handling appropriately. So “AI is dangerous,” right? “AI is useless.” These are two positions that are often held by the same people. And there is maybe a way to make them work together. But I hear people talk about the massive harms that AI is doing. Also, “AI is completely undeserving of the term intelligence. It’s useless, doesn’t do anything valuable. AI is exploitative.”
These all seem like viable debates and important debates to have, but I don’t—it’s dispiriting when they’re all bundled together. And if you even start to push back against one sub clause in one of them, then it seems like you’re already pitching your—you’re already flying your flag as “you’re one of these pro AI, tech bro people.”
What I think, for example, one thing that I always found strange in debates around AI safety is many of the people who were most critical of AI X-risk as a topic, but also very dismissive of AI capabilities. So people—these evil tech bros pushing their AI safety agenda—there’s already some confused sociology informing some of these debates. And by the way, AI is useless. This sort of idea though that AI is useless, I do think it is fascinating how many people, how much variation there is in people I speak to about how useful they find AI. So even some really quite smart, relatively tech savvy people I know, say that they are quite dismissive of the value of LLMs. They say, “No, it ultimately takes more time to fact check everything AI’s put out. It doesn’t really save me time.”
And yet other people, I mean, I guess the three of us would say that, “No, this is incredibly useful.” So one of my pet projects for the next year or so is to try and get a better handle on why some people find so much more utility in LLMs in particular, and others find them borderline useless. I’m curious if Dan or Andy have thoughts on this.
Dan Williams: Yeah, I mean, I would have thought one factor explaining some of the variation is some people just hate AI. And if you hate AI, you’re not going to invest in trying to figure out how to use it. But the set of people who are smart, sort of intelligent, knowledgeable, informed people who aren’t just part of the tribal anti AI crowd, who also don’t get any utility from these chatbots—I’m personally baffled by that because I get such an enormous amount of value as someone who is a researcher, a blogger, et cetera. I find them just incredible research assistants for brainstorming, for getting information, as Andy says, as a kind of initial fact-checking filter. So many sources of positive utility from these systems. So I’m baffled and I’m sort of looking forward to where your project goes if you’re going to try to investigate what’s going on.
It occurs to me, Henry, I cut you off earlier when you were talking about the arguments as soldiers framing. And is this connected to what you were just saying about people bundling all of these different critiques into like a single thing? Is that the idea?
Henry Shevlin: Yeah, exactly. It’s like, I don’t know, let’s just say we’re debating the hobby of torturing kittens. And you’re throwing out all these arguments about—I’m also firmly against torturing kittens. But someone says, “And you know what? Torturing kittens has a massive climate impact.” And you’re like, “OK, hang on. I’m not sure it really does.” “Oh, you’re pro-torturing kittens. I see.” That’s the kind of essence of the problem.
Andy Masley: Yeah, it is really weird. I’ve noticed this quite a bit. Obviously, I want to flag, I’m director of Effective Altruism DC. I’m around a lot of people, including myself, who are really quite worried about the potential for future AI systems to do a lot of bad, even outside of the standard X-risk cases, just—I can see a lot of ways that more advanced systems could really benefit authoritarian governments and surveillance. And it’s just very odd to me. Sometimes I meet people who are very critical of AI but who will also have these same criticisms as well where “one, AI is useless and two, it’s totally gonna empower authoritarian governments and militaries and you’re sick for using it because maybe it’s already been involved in X or Y or Z war crimes and stuff and also it uses so much water.”
And my reaction to that is just like, if the authoritarian dictator’s troops are bearing down on me, my first thought isn’t like “each gun used a whole cup of water to make.” It’s just so disproportionate where it seems like a lot of people really underrate the fact that their really valid concerns about a lot of aspects of AI could actually really be helped if they stopped saying “this is destroying the social fabric and it uses a few drops of water every time it does.” And I’m like, “Well, let’s talk about the social fabric thing. This other thing just seems kind of silly.”
And I think that’s actually an especially powerful talking point. I found where a lot of people who viscerally hate AI and again can totally understand being worried—I am quite concerned about a lot of aspects of AI—but even for people who hate it, actually I find that it gets through to them to say that sprinkling on this thing about energy and water at the end just is actually really diluting your point and it’s really helpful to just talk directly about the potential harms rather than just adding on this little addendum.
Even for me, as someone who’s really into animal welfare, the way we treat chickens is just abysmal. I think this is one of the huge moral catastrophes happening right now and I actually don’t think chickens are especially bad for the environment. If you actually just read about how much impact they have, they’re super clustered together, they eat really basic food and stuff. I mostly don’t say when someone’s eating a chicken sandwich, “that’s really adding to your food emissions or water use.” I mostly just say, “there was a little guy who had a really bad time his whole life.”
And I think the environmental thing just really takes away from that. So yeah, I tend to say, I feel like there are a lot of funny comparisons between how I experience animal stuff and how a lot of people experience AI. Yeah, there are these big evil buildings that use a lot of energy and water called factory farms that a lot of our everyday things are coming from. So a lot of separate comparisons to make about that.
Henry Shevlin: And also produce massive amounts of animal suffering.
Andy Masley: Yeah, yeah, I’ve wanted to, at some point I want to write some short post where I’m like, let’s just compare data—I’ll totally indulge in whataboutism for a second and just be like, data centers have all these big computers in them. And there’s this other big evil building called factory farms and they all have pigs having just incredibly bad lives and they’re both using a ton of water, but the factory farms are using way more. And if you’re worried about one thing, please God, maybe just focus a little bit more on the factory farms. But I think this would correctly be read as a whataboutism thing that’s maybe a little bit distracting so I haven’t stooped to that level yet.
How to Make the AI Debate Better
Henry Shevlin: Just to throw one quick point as a reflection on the kind of arguments as soldiers idea, I think maybe—we’re all trained as philosophers for our sins. And one point that I really stress, particularly on undergrads and early grad students is that very often, including weaker arguments in your papers, drags the paper down, right? So your paper is only often as strong as your weakest argument. And this is particularly relevant if you’re sending publications for review. You just want to make sure you don’t include any subpar arguments in there at all, right? Just better to drop them entirely. But I think a lot of people just kind of have this kind of buffet idea. It’s like, “Well, why not? Well, let’s throw another argument on the pile.”
Andy Masley: Yeah. And yeah, it goes kind of back to the soldier thing where you can almost feel the author writing as if it’s like “and I got him with another one and another one.” And you stop feeling like the author is actually trying to build a comprehensive understanding of the world. And it’s just firing off individual like “this will really destroy the tech bros who like AI or whatever.” And yeah, totally buy the idea that—yeah, a lot of—it’s very easy for people—I personally have a lot of experience with people reading through my 20,000 word post and being like, “But this one thing he said is so stupid. So the whole rest of the thing is a waste.” I’m like, “God, it’s actually quite hard to write about this stuff.” And so yeah, I totally buy that. Just getting that across matters a lot.
Henry Shevlin: So maybe this is a good time to talk about another one of your posts that I found really helpful, which is all the ways in which you want the AI debate to be better. Could you give us just a quick overview of what you see as the—I know it’s a really long post. It’s like what, 10,000 words or something. But what are some examples, perhaps, is a better way to phrase that question, of ways in which you think the debate could be better.
Andy Masley: Yeah, I think there were just a lot of general tendencies that people would get really tribal really fast in general conversations about AI. I was noticing there were a lot of places where people would suddenly start to argue for things that I just don’t think they actually believed a year ago. A lot of people who were very secular would suddenly start talking about how there’s something fundamentally magic in the human mind that a machine can literally never replicate. Other really contradictory weird ideas where AI is both really dangerous and completely useless. Other things like I think people get too tribal in both directions about whether the overall arc of AI is going to be good or bad.
I’m again pretty wary and uncertain about the future of AI. I can totally see ways it makes society much worse overall. And I think my background goal there was to kind of just poke at people who were doing this really soldiery thing and just trying to throw a lot of arguments in both directions about how I would really like us to just step back and stop trying to be a soldier for our side and just engage with some of what I think are these pretty convincing arguments in a lot of different directions. I was trying to raise the standards of the conversation a little bit and just say we shouldn’t just be throwing out blindly these slogans that I don’t actually think cohere with everything else that we believe.
Is AI Just a “Stochastic Parrot”?
Dan Williams: Yeah, it’s a great blog post. I assign it as part of my reading list for my students when I teach philosophy of AI.
Andy Masley: Yeah, the dream for me, by the way, that was insanely flattered by that. So thank you. Yeah.
Dan Williams: No, no, no, it’s a great post. You mentioned there just in passing, so some people have come to believe, maybe they always believed, that there’s something kind of magical, supernatural, non-physical about the human mind, such that machines couldn’t in principle replicate the kinds of capacities that we see associated with human beings. I mean, I take it there are sort of, there are two issues in that general conversation. One is whether you think even in principle machines could replicate human competencies that we associate with intelligence and so on. The other is your assessment of current state of the art AI. Because I take it somebody could think, yeah, okay, in principle, there’s nothing magical about human intelligence and it’s just this very sophisticated information processing mechanism. But nevertheless, it’s also the case that chatbots are just glorified autocomplete or stochastic parrots and so on.
And you’ve got a really nice discussion in that blog post, which sort of attempts to address that latter skeptical position. You also look at the other more general skeptical position. I know that you’ve got loads of interesting views about this as well, Henry, but what’s your take in response to this idea that, meh, it’s just a stochastic parrot, it’s just glorified autocomplete, there’s nothing really intelligent when it comes to these systems?
Andy Masley: Yeah, I mean, part of my motivation for writing the environmental stuff, honestly, was I really want a lot of people who aren’t touching chatbots because of environmental reasons to just sit down and play with them and see for themselves. If this were a stochastic parrot, what would I expect them not to be able to do basically? I’m familiar with stochastic parrots. Back when I was in college, I was a physics major and a lot of my fellow physics majors and I were sometimes not entirely ethically using Wolfram Alpha. And sometimes that would also spit out just wildly incorrect things. And it was very limited in what it could do. And that I would say was kind of a stochastic parrot.
* Post-recording note from Andy Masley: “Wolfram Alpha is not a stochastic parrot. I use that a lot in examples of a chatbot-like thing I was using before ChatGPT and got my wires crossed, sorry!”
Obviously, this is actually very different. This might not be the best comparison but just more broadly, first defining for yourself what a stochastic parrot is. Ultimately, you can say about that stochastic parrots don’t actually do any internal thoughts. They just kind of assign probabilities to different things and spit out an answer kind of randomly. And there’s nothing going on inside. And so if you hold this view and you can engage with GPT-o1 and say “okay, these are the things I don’t expect it to be able to do” and if your category of things that you do expect it to be able to do include literally all useful cognitive tasks, at some point I’m like, it doesn’t really matter if it’s a stochastic parrot or not.
And then separately I think a lot of people really misunderstand what it means to say that AI models predict the next token blah blah blah where I think they mistakenly think that the AI model is just taking this one word at a time and just kind of rolling the dice. And it’s like, the last word was America. Maybe the next word seems very likely that it is. And I think what they’re misunderstanding is that to competently predict the next word, you often need a really powerful general world model inside of the AI model, which I think there’s been a lot of just interesting implications that AIs do have something that can be called a world model from their training that they’ve learned partly by just predicting so much text that over time you start to learn “okay, this idea is associated in vector space with all these other similar ideas” that you can start to draw some general web of meaning between them.
I’ll flag that I’m also not an expert on AI, so take everything I say with a grain of salt here too. But one example that I really like from Ilya Sutskever, formerly of OpenAI, just really massively influential AI scientist, is if you’re reading a detective novel and you get to the end of the novel and the detective is giving their spiel about who killed them and they say “and the killer was”—your ability to predict the next word there actually depends on your general world model of the entire book and if you could do that correctly, it’s a sign that you have what I would consider to be general understanding of the world.
So I definitely don’t think AIs are human level. There are surprising ways that they fail still and it still seems like there’s a lot about human intelligence that’s very mysterious to us. And I’m totally open to the idea that being just physically embodied maybe just matters a lot. And there’s just a limited amount of information things can get from text. But that limited amount of information also just contains a huge amount of stuff that’s very useful to me personally.
So even if they’re very far from human level, they’re already way beyond the level of “they’re personally useful to Andy.” I think sometimes people will be like “they’re either human level or they’re just stochastic parrots.” And there’s this huge middle ground where I’m like “well, they’re not human level, but they can act as basically useful research assistants to me.” And wildly useful research assistants actually. So just having more of that spectrum also matters a lot. But Henry you should go ahead. I know you have a lot of thoughts about this.
AI Capabilities and Moravec’s Paradox
Henry Shevlin: Yeah, okay, I’ll try and limit—I agree with everything you said, Andy. I think one of my particular frustrations, one that you call out in that post, is that I don’t see people make claims about what chatbots can and can’t do without just experimenting themselves.
Andy Masley: Yeah, I will, I just need to scream this really loud for just a moment. If you are making claims about a chatbot, you should try using the chatbot and see if it can do what you were claiming it can’t do. I’ve just seen—there have been articles in the New York Times where people will say they can’t do something and then you can literally just hop on and be like, “Do this please.” And it does it perfectly every time. But sorry, go ahead.
Henry Shevlin: Well, yeah, I was thinking, I think in the Chomsky piece written, I think in early 2023, there were confident claims about various things that ChatGPT can’t do. There was also the piece that I think you mentioned where it was suggested that chatbots can’t detect irony, which is particularly striking to me because it’s something that I kind of regularly probe ChatGPT on. I’ll often make slightly dry or sardonic asides and it’s amazing at picking up when I’m doing so.
In fact, I think that’s probably one of the most surprising capabilities. I think we’re used to operating with this sort of Commander Data mindset when thinking about AI, where it’s all the subtleties and nuances of human communication where they fail and they’re really good at the kind of logical, rigorous thinking. But actually these days it’s almost kind of the opposite. There’s this idea, long-standing idea called Moravec’s paradox, which is “what’s easy for humans is hard for AI and vice versa.” And the funny thing is LLMs kind of go in the opposite direction. They kind of violate the normal expectations around this insofar as so many of the kind of soft skills that we think of as very human, like sarcasm, wit, dry irony, and so forth, they do great at. And it’s exactly the same stuff that we find hard—very, quite complex, multi-step logical reasoning—that they find hard.
Another source of frustration for me here is sort of, I feel like people should be keeping score a lot more about what AI can do this year that it couldn’t do last year, rather than getting into these essentialist debates about “no, the nature of these systems is such that they can never do X.” And it’s like, when those debates are constantly changing and this year it’s X, next year it’s Y, the year after, the constant goalpost shifting, I think is frustrating and elides or obscures the actual really demonstrable progress across multiple capabilities that’s happened.
Andy Masley: Yeah, so much to say about that. I will say just super quick that it’s really overwhelming just how much popular commentary about AI is this really rapidly congealed common wisdom that people develop where they’re like, “Obviously, AI can make kind of okay art, but it’s never going to get hands right,” and just a few months later, it’s there. And people don’t seem to notice what’s happening there where some new capability will come out and just some new take on it—the popular “all intelligent people believe this” take will just snap into place and people will be like “this is what popular intelligent people say about these systems.” Like “yeah I can’t get hands right” and then Sora 2 comes out and it’s like “yeah, this one tiny thing if you have a gymnast flipping sometimes their leg is just slightly off place and that will never change.” And it’s just the rapidness at which people are willing to be like “this just changed, we’re not really gonna notice that massive change and just be like, ‘Well, all intelligent people know this is the limit.’” It’s been a little bit alarming to me, actually.
There’s this one guy on Twitter who will go unnamed but seems to have a pretty big following in AI. I remember I posted a while ago, “I predict within a year, image models will be able to make these shapes well.” And this guy swoops in and is like, “You’re anthropomorphizing the AI. You’re thinking there’s some magic in there, but it’s actually just a stochastic parrot, and that’s so silly.” And then my prediction comes true within five months or something and I’m just like “that was easy.” You can literally just—you don’t have to think there’s a little human hiding in the LLM to think that it seems like the capabilities are going to continue to improve. Just even going back a few years, I really want people to maybe experiment with GPT-2 a little bit just to see how far it’s come since 2021 or something.
* Post-recording note from Andy Masley: “GPT-2 was available in 2019 and GPT-3 came out in 2020.”
Dan Williams: I do think, maybe to steel man the other side a little bit. I mean, you might think that there’s this sort of moving the goalpost style strategy, but you might also think, it’s just very difficult to identify precisely what constitutes a test for the kinds of capabilities that we care about. And it’s just the case that when you specify, “A system won’t be able to do this,” and then it can do that, you might think, “What you’ve learned there was actually that was never a particularly strong test for what you really care about to begin with.” And I do have some sympathy for that. I think, like I said, I use these sorts of systems all of the time. I do think they’ve got this incredibly strange pattern of competencies and failures. And that’s why I would push back a little bit against this framing in terms of human level. I’d rather think of it as a kind of alien intelligence where it’s superhuman on certain kinds of things and it’s just nowhere near what human beings can do on certain other kinds of things.
And although I take the point that there are people who have that kind of perspective and they’ve tried to therefore say, “Well, there are these specific discrete tasks that they’re not going to be able to do,” and then when the next iteration comes out, they look like an idiot. But I think there’s a sort of more charitable framing of that where it’s just, it is just really, really difficult to state precisely what would constitute an adequate test for the kinds of capacities that we care about. And the fact that they just destroy so many of the benchmarks that we’ve been coming up with, that might be telling us something about how if we just get better and better at doing whatever they’re doing, we’re going to get to super intelligent AI systems. It might also just tell us something about how these benchmarks and these tests that we’re using actually aren’t all that reliable as a way of getting at the thing that we really, really care about. That would be my kind of steel man of the rival perspective.
Andy Masley: And I totally agree with that. I think that in the totally other direction, there’s a really unfortunate tendency by people who are very bullish on AI capabilities to sometimes imply that there’s going to be this perpetual smooth line and really underrate just how many different, very varied, complex capabilities we’re talking about. Where I’m totally open to the idea that the current quote unquote paradigm might just not produce really high levels of intelligence in the way that we would expect. It might just be that they are pulling from their vast amounts of training data to basically replicate what they’ve seen, but they can’t actually come up with new things yet.
It’s kind of another case where I feel like if you limit the spectrum to be “AI is incompetent versus it’s competent,” and you give a lot of arguments for the competent side, that really blurs just how varied and how huge of a gulf there is between different levels of competence. It could be that in five years, LLMs are much better at X or Y or Z things, but just don’t achieve the type of high level cognition that we’re expecting from really advanced AI systems. And I do want to make it clear that when I say LLMs I predict will get better I mean they’re getting better in very spiky ways and I’m really not confident at all that the current paradigm scales to AGI or something close to that. Really wildly unsure, mainly because I’m just some guy and have almost no technical background in this stuff so listeners please take everything I say with a huge grain of salt again.
The Challenge of AI Benchmarking
Henry Shevlin: I mean, I completely agree, Andy, and I agree, Dan. I think benchmarking is just really, really hard. And I think if you’re sticking your neck out and saying, “I think this is a good test,” right, there’s a very high likelihood in a rapidly changing technological paradigm that you’re going to make some bad calls. So just to give a personal case, back about 10 years ago, 2016, 2017, I got really excited by the Winograd Schema Challenge. So this was a proposed benchmark for AI that relies on the fact that a lot of pronouns in English are really ambiguous.
So if I say, “The trophy won’t fit into the suitcase because it’s too small,” right? There’s nothing in the grammar or syntax of English that tells you what the “it” refers to. It could be the trophy or the suitcase. But any competent speaker of English will know that if the sentence is, “The trophy won’t fit in the suitcase because it’s too small,” the “it” is referring to the suitcase. On the other hand, if the sentence was, “The trophy won’t fit into the suitcase because it’s too large,” then it refers to the trophy.
So in other words, we resolve these kind of ambiguities through common sense. I thought, “This will be a great test. Any AI system that could reliably disambiguate these pronouns would have to have something like genuine common sense.” But I mean, long before ChatGPT, already by the late 2010s, you had systems performing near human level, just using statistical analysis of what the most likely completions are.
So, I mean, I got that badly wrong and I still think that’s really fascinating for me to process what happened. And yeah, benchmarking is just nightmarishly hard. I think one, just to add to build also on something you said, Andy, I think, and Dan, I think there are some really striking areas where AI is just still so bad. And my favorite case here, there’s a great blog post by Steve Newman called “GPT-o1: The Case of the Missing Agent.”
Andy Masley: It’s crazy.
Henry Shevlin: Where he just really runs through the bizarre and baffling failures that our sort of still quite early stage AI agents are making. My favorite example is this experiment by Anthropic where they had Claude run a vending machine at Anthropic HQ, and the hallucinations quickly got really bad. I mean, maybe hallucinations isn’t quite the right word, but the kind of strategic failures and misunderstandings. Claude was saying it would show up and meet respective suppliers in person. And it’s like, “How are you going to build an effective AI agent if it doesn’t realize it’s not a human?” right? Which is not to say that I think that’s going to be a persistent problem, but I think we are very much in the early days of agency. Sorry, go on.
Andy Masley: I’m blanking out on who shared this, but there was some—I think my favorite personal example of this, I feel bad that I’m getting—I’m blanking out on who specifically it was, but someone on Twitter had shared a screenshot of GPT-o1 saying something about AI, something about deep learning that was wrong. And he was like, “Oh, where did you learn this?” And GPT-o1 very confidently said, “I went to a deep learning conference in 1997, and I remember overhearing this conversation at the deep learning conference in 1997” and just stuff like that is—it’s still wild to me.
And in some ways it makes sense. If you think about LLMs as being these piles of individual, kind of a soup of heuristics rather than a little guy inside, there’s no reason why these heuristics have to all be aligned to reality and it’s like if someone asks you where you learned something about AI you can tell them “I went to a deep learning conference”—that’s just a useful heuristic to use and it’s wild.
I guess on the other end they do also want to push people to think that maybe humans are also often just especially successful more aligned soups of heuristics rather than angels sitting in our brains manning things from behind.
Dan Williams: Yeah. Even when it comes to the next token prediction, I think people underestimate the extent to which many neuroscientists think that minimizing prediction error on incoming data is a fundamental learning mechanism within the cerebral cortex. It’s how animals and human beings learn an enormous amount about the world precisely because as you say, one really good way of building up sophisticated world models is just getting better and better at prediction. It’s this incredible kind of bootstrapping, self-supervised learning strategy.
I think one really good topic to end on would be to ask you a little bit about effective altruism. But maybe just before we get there, there’s another thing that you touch on in the “all the ways I want the AI debate to be better,” which is technological determinism. And I think this would be such a cool conversation to do a separate episode on, because I think it’s just a huge can of worms.
Technological Determinism and AI
Andy Masley: Sure, sure. Yeah.
Dan Williams: You say there that you’re—I mean, basically this falls into the category of things that you think aren’t necessarily obviously true, but should be taken seriously. So you say you’re a kind of technological determinist. Walk us through what that means and why you find that kind of view plausible.
Andy Masley: Sure, so I think that there are a lot of different social theorists who have put forward some basic version of technological determinism in the past. So I would attribute this in part to Marx—I’m very much not a Marxist to be clear—but I think a lot of his really useful insights actually come from some basically technological determinist insights where the basic idea is that new technology and the way that technology works is actually going to have a really huge outsized influence on everything else in society. The technology we have access to is really going to influence social relations and social relations can influence our own roles in the world. I totally buy some basic materialist story where a lot of our behavior is determined by our incentives, our status and things like that. And I think a lot of this is downstream of technology specifically.
So there are a bunch of individual examples of—I don’t think feudalist societies can exist now in these patchworks of how states relate to each other and really complex kind of weird social dynamics, partly because the weapons we have are just very different than what existed in feudal Europe.
Another example that I think a lot of people are really interested in is how farming impacted civilization where before farming, we were hunter gatherers, we were in these small tribes that may or may not have been deeply egalitarian—there’s a lot of unknown unknowns there, we don’t really know very much about that—but it does seem like in general the invention of farming both caused the human population to boom, made people much more dependent on big centralized hierarchical states and ways of relating to each other, and a lot of this just seems to flow from the nature of the technology itself. Farming just makes you dependent on a very specific way of relating to other people.
Another really obvious example is nuclear weapons. If someone goes and invents nuclear weapons, the world is just fundamentally changed. The way that states relate to each other is changed. In that piece I quote one of my favorite essays ever, George Orwell’s essay on the nuclear bomb specifically about how different states or different eras you can kind of predict how free or authoritarian they are, partly by the weapons that are available to everyday people. So he has some funny line about how if everyday people have very easy access to guns, they stand a chance against the state. And as soon as the state gets these really big, powerful new weapons, they can basically just clobber their citizens into submission.
As an aside, I’m not really interested in talking about gun control, separate issue. I don’t really have strong takes about that. But I think one thing that’s really interesting about Marxism is that Marx specifically was one of the first people to, I think, correctly see the full implications of the Industrial Revolution, where a lot of other economists were talking at the time about “this is just this new way of trading.” And meanwhile, Marx was writing all about how this is going to completely dissolve all hitherto existing social relations and make people see the bare structure of society and stuff. And I mostly agree with that, honestly. Everybody becoming very wealthy and becoming very specialized but at the same time a lot of workers in general becoming interchangeable—I don’t want to butcher Marxism or have Marxists get mad at me here so just want to say that his basic description of society where after the Industrial Revolution we should expect people to—a lot of social relations to really rapidly fall away and we can’t expect a lot of pre-existing power structures to continue. I think that’s the case.
And so when I look forward at AI and I’m like, something that could mimic most economically valuable cognition and automate a lot of labor and concentrate power in places—I’m not exactly excited about this. I assume that whatever could flow from this could actually be really quite bad. And we should assume that having access to advanced systems, even if you don’t buy the doom scenario of “AI just rises up and kills us,” there’s still a very high chance that it changes society in a way that leaves the average human in wildly less free or just in a really strange situation that we might not want in the same way that the average hunter-gatherer might not have wanted to suddenly live in a farming society or something.
So yeah I’m pretty into—I basically want technological determinism to be taken seriously as an idea and for people to engage with it and not just say “we can just choose how we use these systems.” I don’t actually think we can choose how we use nuclear weapons. We have some control over that, but ultimately they just create such radical new incentives that people are just really gonna be pushed in specific directions.
Henry Shevlin: Yeah, I think that’s—I completely agree. And I think it’s frustrating sometimes the way technological determinism functions as kind of, as a phrase, functions as kind of a thought terminating cliche, particularly in academic debates where it’s like, “No, that’s technological determinism.” And please go ahead. Yeah.
Andy Masley: Yeah. Well, I’ve been—there have been a few times I’ve really been shot down on this where people are like, “You’re a technological determinist—that’s so outdated.” And I’m like, “I don’t know. It seems like a basically useful insight here.” I haven’t been following how people in academia have thought about this. So again, just some guy here, but the basic impulse just seems to make sense to me and just shooting it off as “that’s this thing we don’t say anymore” just doesn’t really make sense to me. But go ahead.
Henry Shevlin: Yeah, and I think on the one hand, there are sort of the most extreme forms of technological determinism where sort of given technologies mandate or make inevitable certain kinds of outcomes. And I agree, that’s silly. There’s usually some cleavage, but it seems to me the sensible form of technological determinism and the one I think you’re endorsing, Andy, is that technology changes incentives. It changes affordances of states. For example, certain kinds of authoritarian control that would be massively hard to coordinate without things like CCTV cameras become a lot easier if you have CCTV cameras. Things like signals monitoring technologies, again, make certain kinds of authoritarianism more dangerous.
And I think to the extent—and I absolutely share with you, probably my single biggest worry about AI is its capacity for abuse by authoritarian governments. And I think that is, we shouldn’t just assume that all technology—because actually here’s another thought terminating cliche or another common line, people say “no technology is value neutral,” which I think is absolutely right, right, it’s getting there, but it does function as a bit of a platitude. To the extent that I think AI is likely to make certain kinds of authoritarianism more scary, I think we should be wary about shutting down discussions of this just through quick lines about technological determinism.
Dan Williams: Yeah, I think this just general question of whether liberal democracy, as we understand it today, can survive in a world with advanced AI—I feel like that’s such an important and underexplored question. I’m personally pretty pessimistic, but that’s a topic for another day.
Maybe we can end then with effective altruism.
Andy Masley: It’s very disturbing. Yeah, go ahead. Yeah.
Effective Altruism: State of the Movement
Dan Williams: So just as lots of people dislike AI, lots of people dislike effective altruism. I don’t think as many people—so you’re associated with contrarian, controversial opinions from many different areas. I mean, what’s your basic understanding? First of all, I mean, I think the philosophy of effective altruism is trying to do good in a way that’s evidence-based and quantitative in a way that I think came across in our conversation about AI and the environment. But I’m not really part of the EA community. I know people in EA, and lots of people say they’re EA adjacent, and I never know exactly what that means.
Andy Masley: Yeah, yeah, yeah, whoever’s organizing EA adjacent, the C is doing a great job. Let me tell you, they’re killing it. Yeah, yeah.
Dan Williams: Right, right. But what’s the state of EA today? I mean, I think loads of the kind of conversations that people are having about AI, especially in the X-risk debates and so on, to me at least, someone observing, it feels like that’s really connected to developments that have happened in EA, and EA has played a really big role in much of that. But you’re obviously professionally involved with effective altruism. What’s the state of the movement today?
Andy Masley: Ooh, a lot to say about that. I think overall, behind the scenes, I feel like EA in specific places, especially, is really quite healthy. For me, one of the things that I really love about it is it’s actually quite heterogeneous in terms of different people’s thinking on it. People aren’t really in lockstep about any one idea. Within EA DC, which is one of the largest EA communities anywhere, so I have a lot of access to how different people are thinking, people have really wildly different takes about basic questions about AI X-risk, a lot of other things.
We’re still getting a lot of just wildly interesting, competent people coming to the general movement in general, which I think is a sign of a lot of health. Obviously, the last few years have been a wildly bumpy ride. We can talk about that a lot as well. But I mostly—I don’t know, I’ve been pretty excited about basic EA ideas since I was a teenager. I remember I saw a Peter Singer video in 2009 that was one of a few really profoundly impactful videos on me where he’s just kind of walking around talking about global poverty and talking about how strangely people spend their money given all the problems in the world. And so I think in its full arc as I’ve seen it I’m quite bullish on it honestly.
Yeah, a ton to say basically. And I do obviously want to acknowledge that EA has produced both a lot of good and bad ways of thinking about AI. I definitely agree with a lot of its critics that there are a lot of places where people have kind of developed wildly overconfident, very specific world models and models of how AI works, especially. And I think at its worst, EA can be kind of a way of tricking yourself into feeling more confident about a very complicated topic than you actually do.
But I think for me personally, part of the reason why I’m so excited to put my face to it, especially the DC network, is that it’s still been a space where an incredible amount of new, very valuable ideas about the world and AI especially have been generated. I think in my own writing, I’ve been influenced a lot by basic EA sources in writing, where you try really hard to not be super teamy in what you write. You try to give people a complete overview of the issue and really try to push forward the idea that, yeah, we should be very quantitative and understand that there’s actually these huge gulfs between how different things impact the world. So yeah, I mean, I’m a big fan basically, but there’s a lot more to say about that, but happy to go in any specific direction there.
The Three Main EA Cause Areas
Henry Shevlin: Can I ask, by the way, Andy, so I would consider myself EA adjacent, or at least adjacent to EA adjacent, know a lot of EA adjacent people. So being really crude here, it seems like the three big cause areas throughout the history of EA, at least to my mind—tell me if I’m missing any—are animal welfare interventions, development interventions, things like bed nets, famously deworming and so forth, and third, X-risk. I’m curious if that seems like a good initial division of the cake and also if you have any thoughts on how the balance of those three have been evolving in the movement.
Andy Masley: Yeah, I tend to say—so basically the motivation for all of those three is that the specific guiding idea of EA is that you should spend time trying to figure out where you can do the very most good and what things you can do with your career or donations that will actually have the most total general positive impact on the world. And usually those correlate a lot with either where there’s the most suffering that can be very easily alleviated. So in global health, it’s really shockingly easy to donate to the right charities to permanently save a person’s life for like a few thousand dollars.
Animal welfare, obviously, if you ascribe any value to animal experience at all, there’s a gigantic moral catastrophe where hundreds of billions of conscious minds are suffering in really brutal ways that again, there’s a huge amount of low hanging fruit to fix because the funding for the total amount of funding in animal welfare is about a tenth of the funding of the admittedly large school district I used to work for. It’s incredibly small.
And then X-risk, the basic idea is that any small thing you can do to decrease X-risk has a huge outsized impact on both current people and potential people in the future. This is where it gets very controversial where you start to talk about speculative far future stuff and how the future can go.
Yeah, I definitely noticed in my time that AI has taken up more and more oxygen within EA. I think understandably, honestly, from the inside, I have access to a lot of high level people in the movement and I really don’t read their motivations for this as coming from some kind of “oh, AI is just the thing to talk about right now.” Because a lot of them were really, their hair was on fire about this back in 2015. And so, I don’t know, I was recently at this panel where someone who’s pretty critical of EA was saying “well have EAs just hopped on the bandwagon of AI?” And I was losing my mind a little bit because I was like, “I was doing this back when everyone was like, ‘Why are you guys so focused on AI? This does not matter at all.’ And suddenly we’re being accused of hopping on the bandwagon.”
So yeah, AI is taking up more and more oxygen. There’s a lot of sudden interest in how AI affects other things like AI and animal welfare or how AI will affect great power relations or very poor countries as well. There’s still—I think EA has worked surprisingly well as eight or nine kind of overlapping communities who mostly radically disagree with each other. In a lot of EA conferences I’ll go to, I’ll still meet a ton of global health and animal people who basically just don’t buy a lot of the basic AI X-risk case or just don’t feel there’s anything they could do about it.
And so it still feels pretty healthy in that way, but I do have to say that if you’re getting more involved in EA, it is very important to understand that AI is really being focused on now. Just a lot of people are much more convinced than they were a few years ago that very scary capabilities could arrive very soon basically. And there’s a lot to go off about that.
I would actually be curious, Henry, if you want to talk about it—I know a lot of people who identify as EA adjacent and I’ll be like “what do you mean by that?” where they’ll be like “I donate 10% to charity and I buy AI X-risk and I’m a vegan but I’m not an EA because I don’t buy this very specific view of utilitarianism.” And I’m like “well I don’t either.” So I’m curious about what makes you adjacent.
Henry Shevlin: I would actually say it’s almost a self-deprecating use of the term in the sense that there’s a lot of the EA value system and mission that I admire, but just fail to live up to in my own life. There are certain kind of interventions that are super easy for me. I’ve been vegetarian since I was quite young. I work on an area that I think does have high expected utility, namely AI ethics. But there are so many other areas. I’ve tried to go fully vegan multiple times and I try and—I think these days I’m reduceatarian, I think these days. I’ve managed to phase dairy out of my coffee. But yeah, so one reason I call myself EA adjacent rather than full EA is because I don’t feel like I’m not there yet. I’m not good enough at the implementing, turning my aspirations into reality.
Andy Masley: Yeah, I mean, I’m not living without sin. Most people I know—it’s actually only a minority of EAs who are fully vegan. It’s incredibly hard. And I think even within animal welfare actually—I’ve been meaning to write a blog post about this for a while actually, but I actually really worry about veganism as being the singular “this is what it means to care about animal welfare.” For the same reason that I don’t think you should be barred from being someone who was worried about climate because you personally drive a car or something like that. If anything, that would probably really blow up the movement. So I don’t know, I wouldn’t be so self-deprecating. Basically, I think if you’re doing AI ethics work, you’re thinking about X-risk and the suffering of AI systems and you’re vegetarian, I wouldn’t hold back on the label. But you know, totally understand if you don’t want to associate further reasons.
Henry Shevlin: Yeah, congrats. You have my blessing.
Andy Masley: Yeah, yeah. Love to hear it. Yeah, love to hear it. Cool, cool, cool.
Henry Shevlin: Nice. So I can call myself EA now. Okay, I’ll drop the adjacent. I’ll drop the adjacent. Yeah.
Dan Williams: You’re officially welcomed into the community. Okay, so that was so fun. There was so much stuff that I think we covered there. Do you two have any final things you wanted to touch on that you wanted to talk about? Andy, any questions that we had asked that we didn’t ask?
Closing Thoughts and Recommendations
Andy Masley: Yeah, this was a blast. I’m actually—I guess I would be curious about any of your recent takes, Dan, just because I know you’ve thought a lot about tribalism and polarization and just how people relate to expert consensus on stuff. And I guess would be interested if you have any additional thoughts on how the debate about LLMs have kind of evolved within that broader spectrum of how people think about deferring to experts and which experts to trust and stuff like that.
Dan Williams: Yeah, I mean, I think we’ve touched on some of this already. I would say I think the kind of work that you do is really valuable and really underrated, which is just putting in the work to persuade people with evidence and rational arguments. And to the extent that you do that in a good faith way, and the evidence that you’re citing is in fact the evidence of expert consensus on different views, I think people dramatically underestimate how impactful that kind of activity could be.
I think the issue of expert consensus in general is very, very challenging. I mean, you said a few times with AI, “I’m not an expert,” but I think it’s actually very difficult to say what precisely makes someone an expert when you’re talking about certain kinds of issues. I think Hinton is an expert when it comes to deep learning. I think he talks about a lot of other issues where I would say he’s not an expert, but he gets treated as one because it’s somehow connected to AI.
So that’s really complicated. I also think, and I’ve written a post about this recently, that there’s a lot of kind of high brow misinformation. This is a term that I’m taking from the philosopher Joseph Heath, which is, often we think about misinformation as being something associated with the low brow kind of dumb alternative media environment, Candace Owens, Tucker Carlson, et cetera. And I do think to be clear, there’s just a whole load of informational garbage in that space, as understandable that people focus on that. But even in highbrow information environments staffed by highly educated professionals with overwhelmingly center left progressive views, there’s a lot of just really low quality selective misleading content. I think we’ve touched on some of it today.
So I think this is a perfect case where when I talk to people in my social and professional circle about AI and the environment, I think they’re just really kind of misinformed about it. And I think you’ve done a great service in pointing that out. But I think even when it comes to climate change in general, there’s all of this bad right-wing denialism. Totally, I think that is bad. That needs to be called out. There’s also a lot of this kind of high brow catastrophism surrounding climate change, which is also not founded upon our best expert consensus on it.
So I think the heuristic many people have, which is “Oh, if I just kind of align my beliefs with what smart people affiliated with the institutions believe, then everything will be okay.” I don’t think that’s true at all. I think there’s a lot of incredibly misleading communication associated even with our kind of expert-based institutions. So sorry, that was waffling for a long period of time, but that’s my assessment of how that connects to what we’ve been talking about.
Andy Masley: No, that was great. No, that was good. Yeah, no, that was really good. Yeah, totally agree. Also a huge Joseph Heath fan in general. And yeah, I’ve definitely bumped into quite a bit of highbrow misinformation about climate in general where I am quite worried about climate. I think maybe even more so than a lot of EAs where I do probably ascribe more of a probability to quite bad things happening in the long run because of all this. And I’m not totally bought into the idea that technology is actually going to lead us on to the correct path on its own so we need a lot of policy and stuff.
But yeah, even there I just have so many individual memories of friends and people I would meet very confidently telling me basically that civilization would end by the mid-2020s or so. It was just a very common experience of my daily life throughout the 2010s. And so yeah, I have separately hoped that my pieces have been a small push back against that. I’ll pepper in where I can that you should actually just read the IPCC report summaries. Just try to understand what the actual science is like. Start there. Don’t start on a scary TikTok. Just go to the Wikipedia, understand what the IPCC says, and then run from that. So yeah, I totally agree on the issue of misinformation coming from multiple directions here.
Henry Shevlin: One thing that I love that happens sometimes on podcasts at the end is when people give me the opportunity to promote either work of mine or work of colleagues or just interesting cause areas. So maybe another nice way to close out would be to say, what are some stuff that you’d like more people to look at, whether it’s stuff from your own writing or other people or cause areas that people should be getting involved in?
Andy Masley: Oh man. I mean, there’s a lot. I would—let’s see. For EA stuff, I always like to plug 80,000 Hours. I think that if you enjoy my writing, a lot of my writing has actually been very directly influenced by their writing style, specifically where they try to give you a pretty comprehensive general overview using very non-teamy language about big, huge issues. I think their article on animal welfare especially is my single favorite place to drop on new people. If you’re interested in EA more, you should definitely check that out.
Climate stuff, definitely strongly recommend—and if you enjoy my writing—Sustainable Energy Without the Hot Air by David MacKay. I think, I’m pretty sure that’s the title. I read that in college and that was also just hugely influential. And if you read that book, you’ll be like, “Andy’s just doing the David MacKay thing.” This is literally just—Andy’s just doing this imitation of what he did in that book, basically. So I very strongly recommend that.
Yeah, but those would be two places to start to understand more of where I’m coming from specifically. And obviously, subscribe to my blog. I love to get subscribers, especially if you live in Nevada or South Dakota specifically. Those are the last two states before I’m on a 50-state blog, so I’m constantly on the hunt for those. Please, God, subscribe if you live in those states. But yeah, those would be my recommendations, basically.
Dan Williams: Fantastic. Well, thanks, Andy. This was great. And Henry and I will be back next time with another conversation about AI. Cheers.
Andy Masley: Yeah, yeah, this was so much fun guys, thank you. Yeah, cool, see you soon, bye.
Please note that this transcript is AI-generated and may contain errors.
Dan Williams: Okay, welcome everyone. I’m Dan Williams. I’m here with the great Henry Shevlin. And today we’re going to be continuing our series of conversations on artificial intelligence, some of the big picture philosophical questions that AI throws up. And today specifically, we’re going to be focusing on AI consciousness. So could machines be conscious? What the hell does it even mean to say that a machine is conscious? How would we tell whether a machine is conscious? Could ChatGPT-5 be conscious and so on? Before we jump into any of that, Henry, I’ll start with a straightforward question, or what seems like a straightforward question. What is consciousness?
Henry Shevlin: So it’s very hard to say anything about consciousness that is either not a complete platitude or rephrasing like consciousness is experience, consciousness is your inner light, consciousness is what it’s like. Those are the platitudes. Or saying something that’s really controversial, like consciousness is a non-physical substance or consciousness is irreducible and intrinsic and private. So very hard to say anything that is actually helpful without also being massively controversial. But probably let’s start with those kind of more platitudinous descriptions.
So I assume everyone listening to this, there is something it’s like to be you. When you wake up in the morning and you sip your coffee, your coffee tastes a certain way to you. When you open your eyes and look around, the world appears a certain way to you. If you’re staring at a rosy red apple, that redness is there in your mind in some way. And when you feel pain, that pain feels a certain way. And more broadly, you’re not like a rock or a robot, insofar as we can understand you purely through your behavior. There’s also an inner world, some kind of inner life that structures your experience, that structures your behavior.
All of which might sound very obvious and not that interesting, or not that revolutionary, but I think part of what makes consciousness so exciting and strange is it’s just very hard to integrate it with our general scientific picture of the world. And I’ll say in my own case, this is basically why I’m in philosophy. I mean, I was always interested in ethics and free will and these questions. But the moment where I was like, s**t, I’ve got to spend the rest of my life on this, came in my second year as an undergrad at Oxford studying classics. I was vaguely interested in brains and neuroscience, so I took a philosophy of mind module with Professor Anita Avramides. And I read an article that I’m sure many of the listeners at least will have heard of called “What is it Like to Be a Bat?” by Thomas Nagel, and it blew my mind.
And immediately afterwards, I read an article called “Epiphenomenal Qualia” by Frank Jackson, which is the article that introduces Mary’s room. And it blew my mind even more. And basically, I’d spent most of my life up until that point thinking the scientific picture of the world was complete. I spent most of my life until that point basically thinking the scientific picture of the world was complete. And you know, there was some stuff we didn’t understand, like what was before the Big Bang, maybe exactly what is time, but when it came to biological organisms like us, we had Darwin, we had neuroscience, it was basically all solved. And then reading more about consciousness, I realized, my god, we don’t even begin to understand what we are, what this is.
Dan Williams: Yeah. I think that’s... Let me just interrupt there to flag a couple of those things, because I think they’re really helpful in terms of structuring the rest of the conversation. The first is, when it comes to consciousness, it’s really, really difficult to articulate precisely in philosophically satisfying ways exactly what we’re talking about. You mentioned this classic article, “What is it Like to Be a Bat?” And I think it’s a fantastic article, actually. I’m teaching it at the moment. And one of the reasons I think it’s fantastic is because it does convey in quite a concise way, quite quickly, the sort of thing that we’re interested in.
So I’m talking to you, and I assume that there’s something it’s like to be you. Nagel’s famous example is with bats. They are these amazing animals. Their perceptual systems are very alien to ours, but we assume there’s something it’s like to be a bat. So it’s very difficult to state precisely exactly what we’re talking about, but you can sort of gesture at it—something to do with subjective experience, what it’s like to have an experience and so on.
And then the other thing that you mentioned, which I think is really interesting, and in a way, it’s sort of disconnected from the machine consciousness question specifically in the sense that even if we had never built AI, there would still be all of these profound mysteries, which is just how the hell do you integrate this thing called subjective experience into a scientific worldview? I mean, there are other sorts of things where people get worried about a potential conflict between, roughly speaking, a scientific worldview and a kind of common sense picture of the world. So maybe free will is another example, or maybe objective facts about how you ought to behave. Some people take that seriously. I’m not personally one of them, but some people do. But I think you’re right. Consciousness feels so much more mysterious as a phenomenon than these other cases that still seem to pose puzzles for a broadly scientific worldview.
Henry Shevlin: Also, unlike free will and unlike objective morality, I think it’s very, very hard to say that consciousness doesn’t exist. I mean, it’s pretty hard to say that free will doesn’t exist and painful perhaps to take the view that objective morality doesn’t exist. But these are just very well established positions. And there are some people out there, illusionists, who try and explain away consciousness. Maybe how successful they are is a matter of debate. But it’s very, very hard to just say, like, your experience, your conscious life—nah, it’s not there. It’s not real. It doesn’t exist.
Dan Williams: Yeah, right. Actually, I think that’s another nice place to go before we go to the specific issues connected to artificial intelligence. So there’s this metaphysical mystery, which is how does consciousness, how does subjective experience fit into a broadly scientific, we might even say physicalist picture of the world? And so then there are lots of metaphysical theories of consciousness.
I’ll run through my understanding of them, which might be somewhat inadequate, and then you can tell me whether it’s sort of up to date. Roughly speaking, you’ve got physicalist theories that say consciousness is or is realized by or is constituted by physical processes in the brain, in our case. You’ve got dualist theories that say consciousness is something over and above the merely physical. It’s a separate metaphysical domain, and then that comes in all sorts of different forms.
You’ve got panpsychism, which is, to me at least, strangely influential at the moment, or at least it seems to be among some philosophers, that says basically everything at some level is conscious, so electrons and quarks are conscious. And then you’ve got illusionism, and I suppose probably the most influential philosopher that’s often associated with illusionism would be Daniel Dennett. I understand that he had a sort of awkward relationship to the branding. But there the idea is something like, look, we take there to be such a thing as consciousness. We take there to be such a thing as subjective experience. But actually, it’s kind of just an illusion. It doesn’t exist. Is that a fair taxonomy? Is that how you view the different pictures of consciousness in the metaphysical debate?
Henry Shevlin: Yeah, I think that’s pretty much fair. A couple of tiny little things I’ll add. So panpsychism maybe doesn’t completely slot into this taxonomy in quite the way you might think. Because a lot of panpsychists would say, no, we’re just physicalists, right? We believe that everything is physical. There is only the physical world. But consciousness is just a basic physical property associated with all physical stuff. So that’s, for example, Galen Strawson’s view. He’s got a great paper called “Real Materialism.”
But there’s another way you can be a panpsychist, which was the Bertrand Russell sort of view and the view developed by people like Philip Goff that says the underlying nature of reality is neither physical nor mental. This is also sometimes called neutral monism. Also a view associated with Spinoza, the great historical philosopher. So underlying reality is neither physical nor mental, but everything has physical aspects and mental aspects. Spinoza’s version of this is called dual aspect theory.
So the other thing, other little qualification worth mentioning is if we go dualist, there is one very important, I think, pretty defining distinction. Okay, if consciousness is not physical, does it interact with the physical? Descartes, of course, famously thought that consciousness was a substance or the mind was a substance that interacted with the physical, which immediately runs into some really messy issues. But the other predominant form of dualism is epiphenomenalist dualism, where basically everything the world is made of is physical, but through complex or strange or basic metaphysical processes, there’s this conscious layer that sometimes emerges from reality, but doesn’t itself drive any causal processes. But that’s the basic metaphysical picture.
Dan Williams: Yeah. And I think that thing you said there at the end about emergence is really important. And I think many people coming to these debates to begin with, they don’t draw the distinction between a view that says consciousness in some sense just is or is constituted by physical goings on in the brain, for example, and a view that says consciousness emerges from physical goings on in the brain. They sound like they’re similar claims, but actually that latter claim about emergence is really very different from the ordinary physicalist view, because you’re basically saying there’s something over and above the physical. It might be related to the physical in a law-governed manner, but it’s a different kind of thing. Is that right as a distinction?
Henry Shevlin: Absolutely, yeah. And I think it’s one of the ways of problematizing the kind of knee-jerk physicalism that I think a lot of people have, or the kind of physicalism that doesn’t see consciousness as problematic. It’s like, yeah, consciousness emerges from the brain. It’s like, okay, but what we ultimately need is a theory that tells us how pain, orgasms, perceptions, everything, all of your experiences, how at some level they just are neurons, right? There’s nothing over and above neuronal activity or computational processes or information. The theory actually has to ultimately bottom out in neurons. If you say it emerges, and without being identical to it, without being identical to the brain stuff, that still does not neatly fit with the physicalist picture.
Dan Williams: Right, right. Yeah, and it’s very unsatisfying in a way, or at least I find it unsatisfying just to have brute emergence. By the way, I love that your first two examples there were pain and orgasms as the most salient examples of conscious experience that come to mind. It just occurred to me, something that we might be taking for granted as philosophers who know this literature well—you know it a lot better than I do, but we’re both immersed in it to some degree—which is we said consciousness poses a problem for, or at least a puzzle for a scientific worldview, we’ve gestured at why that is. Maybe it’s helpful to return to something you said, which is one of the things that you found most influential in your own philosophical journey was this article by Frank Jackson, which touches on what’s often called the knowledge problem.
So this is basically a thought experiment where you’re asked to imagine a color scientist called Mary who is color blind, or perhaps not color blind—in fact you can correct me if I’m misdescribing the story—but basically is an expert in color science but inhabits a black and white room, so she’s never actually seen, for example, the color red, but we’re supposed to imagine that she knows everything there is to know about the neuroscience, the neurophysiology, the physics of perception, of color perception, but she’s never actually herself experienced the color red. So she knows all of the physical facts, the facts about neurobiological mechanisms and so on, the facts about how light interacts with the visual system, knows all of the physical facts, never actually experienced red.
And then Jackson asks, okay, suppose that one day she does, let’s say, leave this purely black and white environment and encounters the color red for the first time. He says, well, it’s obvious that she’s going to learn something new, which is what it’s like to experience the color red. But by stipulation, she knew all of the physical facts already. So the fact that she’s learned something new suggests that the physical facts don’t exhaust the facts. Is that, roughly speaking, the thought experiment?
Henry Shevlin: Yeah, that’s exactly right. So yeah, in Frank Jackson’s original framing, she’s just in this black and white room, but typically when teaching undergrads, undergrads say, can’t she just rub her eyes really hard or bang her head or find some other way to generate phosphenes or something like that. So the way I usually frame it for pedagogical purposes is she’s got a condition called cerebral achromatopsia, which is a real condition, which is a form of neural deficit that means you can’t experience color. And then one day she gets her neural deficit fixed and she’s like, my god, now I know what all these colors are like. I spent my whole life researching color but I never knew what color actually looked like.
But yeah, that’s just my preferred framing but I think you’re absolutely right. The way the actual structure of the argument is by hypothesis, she knows all physical facts relevant to color vision at the neural level, the level of optics, the level of physics. She learns a new fact when she gets her vision fixed or leaves the room. Therefore, there are some facts that are not physical facts, or the domain of fact is not exhausted by purely physical facts.
And the interesting thing about this, I think—I mean, the argument is basically there in the Nagel article as well, maybe in slightly less clear form—one way of capturing one of the points that Nagel makes in that article is, look, no matter how much we ever knew about bat behavior, bat brains, about bat evolution, that’s never going to tell us what it’s actually like to see with ultrasound for bats. Presumably there is something it’s like, this phrase that Nagel popularizes in that article, but you’re never going to get to that purely from a neurophysiological or optical understanding of the world.
Dan Williams: Yeah, I mean, from what I remember in the Nagel paper—I reread it relatively recently—but I think Nagel says at the very least, this suggests that there’s this fundamental what philosophers would call epistemic challenge in the sense that even if you’re committed to the view that ultimately everything is physical, nevertheless, what these sorts of thought experiments and these sorts of examples demonstrate is we, at least at present, can’t even understand how that could be true.
Whereas I think Frank Jackson makes a more directly metaphysical conclusion, which is this thought experiment, at least when he was initially publishing these ideas—I think he’s later changed his mind—but this thought experiment demonstrates physicalism to be false. Maybe it’s worth also noting that another thing that philosophers will often bring up in this context is the zombie thought experiment, I guess, or the imagined situation where you’ve got a system which is behaviourally, functionally identical to a human being. So exactly as you are right now talking to me, we’re having a conversation about consciousness, and yet there’s just nothing going on inside. There’s nothing it’s like to be that system, no subjective experience.
And the point is not supposed to be that people actually are zombies, but the fact that we can allegedly coherently imagine that situation without running into any kind of logical contradiction demonstrates that consciousness is something over and above the merely physical, functional, behavioural. Roughly speaking, that’s the kind of idea. Before we move on to how these sorts of metaphysical puzzles and issues connect to AI, did you have anything else you wanted to add about any of that?
Henry Shevlin: Yeah, two very quick things. So just to give a name to the difference you are isolating between Nagel and Jackson, the difference between epistemic versus metaphysical arguments, and to round out our trifecta of guaranteed readings for class one of a philosophy of mind or consciousness session. So this is the idea of the explanatory gap, an idea coined by Joe Levine, who basically points out that, look, even if you think from inference to the best explanation, our best model of the world is that everything is ultimately made of physical matter and energy, right now, we have no idea how to integrate consciousness in that picture. This is the explanatory gap. And you can see that as a challenge that a good theory of consciousness, a good physicalist theory of consciousness should close. It should fill out and make obvious why consciousness exists and why things are the way they are.
And I think this is actually my preferred way of framing the zombie arguments, right? You can see the zombie argument as a challenge to physicalism that says, all right, give me a theory of consciousness that shows why zombies are logically impossible, right? In other words, why it is impossible, a contradiction in terms, which is what you ultimately need for a complete physical theory, to talk about beings who are not just behaviorally, but microphysically identical to us—beings that are microphysically identical to us that are non-conscious. That seems like something I could imagine, like something, someone’s walking around but there’s no lights on on the inside, so to speak. A good physicalist theory of consciousness, a complete physicalist theory, should close the explanatory gap and show why that is not just implausible but like an actual logical contradiction.
Dan Williams: Yeah, yeah, and I think that’s nice, and I think it connects to, in some ways, some actually quite complex philosophical ideas. The more you dig into this, you realize actually you need access to a whole complex philosophical machinery to really even articulate the core ideas and positions and debates and so on. But I think we’ve said enough to sort of frame this and give it context. Maybe now we can move on to artificial intelligence.
Before we get to the present day, I think it’s worth maybe just taking a little detour through some of the most important historical developments in thinking about artificial intelligence and consciousness. So famously Alan Turing, who’s a pioneer in computer science and digital computers and AI and so on, he published an article, I think it was 1950, “Computing Machinery and Intelligence.” And the opening line of that is something like, we’re going to consider the question, can machines think? And then he goes on to propose what’s subsequently referred to as the Turing test, which roughly speaking is, could a machine via conversation, text-based conversation, trick us into thinking that we’re talking to a human being? And if it could, then it passes the test. It’s imitated human behavior and Turing seems to kind of say that if a system did in fact pass that test then there’s some sense in which we could say that we can describe it as a thinking entity or we can describe it as intelligent.
I’m being sort of vague there because I think the article itself is a little bit vague in terms of how it expresses these different ideas, which is understandable in a way because obviously this was written in 1950. So I think there are kind of two issues there. One is that Turing was focused on, at least on the surface, thought and intelligence in the first instance, whereas we’ve been talking about consciousness. And then the second facet of this is Turing was proposing a kind of behavioral test for establishing whether a system, whether a computing machine can accurately be described in these sorts of psychological terms.
Let’s take the first of those first. This issue of intelligence, understanding, thought on the one hand and consciousness on the other. My sense is many people bundle these two things together, but you might think there’s an important distinction between them. Where do you come down there?
Henry Shevlin: Yeah. So I think one distinction that maybe is present in pretty much all debates in cognitive science, but is less present in public understanding of these issues is the distinction between cognition or the mind on the one hand and consciousness on the other. So basically, for the last hundred years or so, it has been not uncontroversial, but widely recognized that there are at least some forms of unconscious cognition. Whether you think that’s Freudian unconsciousness—we have unconscious drives and motivations—or the more cognitive neuroscience view of unconscious. Think about the stuff that happens in early vision or things that happen in linguistic processing or for that matter, we’ve got a rich literature now on unconscious perception. All the various ways your brain could register and interpret representations without thereby giving rise to any conscious experience.
And I think our language around the mind and cognition doesn’t need to be conscious in order to be useful. In particular, I think there’s another route to understanding what mental states are that doesn’t involve consciousness, which is they’re in some sense representational states. And as soon as we need to start talking about things like what you’re perceptually representing or what your unconscious beliefs are or what your unconscious motivations are, you’re talking about your brain managing and interpreting representations. At that point, as far as I’m concerned, you’re in the domain of the psychological, you’re talking about the mind, and a lot of that stuff happens unconsciously.
So I think, yes, we shouldn’t assume at the outset, certainly, that all mental states are conscious. So I think it’s entirely coherent to say, for example, I think LLMs think, I think LLMs understand, I think LLMs reason, but I don’t think LLMs are conscious. And I think we can keep those two apart, although of course, different theoretical positions are going to lead you to say, actually, maybe the only real form of understanding is conscious understanding. For example, that’s a position you can adopt.
Dan Williams: Right, right. Yeah, that’s really interesting. I definitely—it seems to be the case that people are kind of comfortable, well, they’re comfortable talking about artificial intelligence, which suggests that they don’t really object, for the most part, to the idea that these systems are at least in some sense intelligent. I think there’s also a tendency for people to use terms like thought and reasoning quite easily when it comes to using these systems. But people, I think, are much more suspicious of the idea that we should think of them as conscious systems when it comes to something like ChatGPT-5 or Claude or Gemini. And I think that’s kind of in line with basically what you’re saying, that there’s an important distinction between these two things. They might go together in the human case in the sense that we are intelligent animals and we’re also conscious animals. But in terms of thinking about this issue carefully, you do need to draw a distinction between them.
Henry Shevlin: Absolutely. I’d also add that I think this is something that’s really clear in any kind of comparative cognition. So if we’re looking at non-human animals, okay, maybe we all agree that chimpanzees, dogs and cats are both conscious and have a mind. But there’s a really rich field of insect cognition. People work on cognition in quite simple invertebrate organisms, even more simple than insects. There’s a growing field, believe it or not, of plant cognition that tries to understand plant behavior in cognitive terms and even—and so actually really quite interesting research on bacterial cognition, cognition in microbes.
Now, we can debate whether that’s really called cognition or whether it’s better understood in more basic terms. But the point is you don’t need to decide in advance if a given animal species is conscious in order to do useful cognitive science and comparative psychology about its behavior and its internal structures.
Dan Williams: Yeah, that’s interesting. It reminds me actually, when I teach AI consciousness, I would often start by putting some images of things up on the PowerPoint and asking people for a kind of snap judgment of whether it’s conscious or not. So it’s like, here’s a chimpanzee—conscious. Of course it’s conscious, people think. Here’s a squirrel. And then I’ll do things like, here’s bacteria or here’s a tree or here’s the entirety of planet earth. And when you get to those sorts of cases, I think people just fill up either it’s not conscious or they’re in this complex situation where they’re not quite sure what the question even means.
But funnily enough, when I then put up the logo of ChatGPT, and I realize this is not a representative sample necessarily, but I think basically always the response is no way is that conscious. And we should get to that later on because I think that’s an interesting intuition and I think you’ve also got views about how widespread that intuition is and how we should expect that to develop in the future. Maybe before moving on from this then, have you got any thoughts about the Turing test? Because it has been very, very historically influential. So we should probably say at least a couple of things about it before moving on.
Henry Shevlin: I think it’s a beautiful paper and it’s very accessible and I’d encourage everyone to read it. You don’t need any particular specialization to read it. And it’s just amazing how many arguments Turing successfully anticipates. And I quote that article on so many different issues. He talks about how even with the very simple computers he was using at the time, they regularly do things that he wasn’t expecting them to do. They regularly surprise him. He’s addressing this idea of a computer can never do anything except what it’s been programmed to do. It can never surprise us. And he’s like, no, of course. Just because you program in the initial instructions doesn’t mean it can’t surprise you in all sorts of interesting ways.
He also does—you’re right that it’s framed around this question of can machines think. But he explicitly addresses consciousness or the argument from consciousness, as he calls it. And he makes this slightly behaviorist move where he basically says, this is at least a lightweight behaviorist move where he says, look, none of us know whether anyone else is conscious, right? He calls it a polite convention, right? What we do is we look at each other’s behavior and assume, well, look, it’s behaving in relevantly similar ways to me, I’m conscious. So I guess he’s conscious too. This is what he calls the polite convention.
So he says, basically, look, the question is, under what circumstances, if we’re worried about consciousness, should we extend that convention to machines? And it seems like if they can successfully pass as humans—I’m doing some slight retro interpretation here, but I think one way of seeing the argument is if they can, to the extent that machines can do the same kind of behavioral capacities, relevantly similar behavioral capacities as humans, then we should extend the polite convention to them as well.
Dan Williams: Yeah, and I think just to connect that point about, can they exhibit the appropriate behavioral capacities? So there’s a kind of claim there about consciousness, but obviously connected to him proposing the Turing test is this broader view, which is that there must be some kind of behavioral test of whether a system can accurately be described as—his primary focus, as we’ve said, is on thought and intelligence, but also potentially consciousness as well.
My sense is most people these days who are experts in the field, they think that the specific test that he proposed was in a sense not very good, just because you can get systems that imitate human beings in such a way that they can trick people into attributing intelligence without actually being all that intelligent. We should also say that, I mean, my understanding of this, and I wouldn’t find this surprising at all, is that state of the art large language models can basically pass pretty stringent versions of the Turing test, as I understand.
Henry Shevlin: Yeah, yeah, so there have been some large scale—replication isn’t quite the right word—instantiations of the Turing test. One from earlier this year that involved five minute dialogues with state of the art language models, found that state of the art language models were actually judged to be human more often than their human competitors. But the slightly tricky thing here is that I think the Turing test is not a single highly defined experimental set of parameters, right? Because you can vary so many different dimensions. How long do you get to talk to the person for, for example, right? Are language models allowed to be deliberately deceptive and say things that you know to be false? So there’s a whole bunch of different rules parameters you can vary. So there’s not going to be like a single moment where it’s like, amazingly, machines have finally passed the one and only Turing test, right? There are lots of different variations of it.
And there was a long running prize called the Loebner Prize. I think it started in the 90s and ran for about 20 years, but basically pitted state of the art chatbots against humans in an instantiation of the Turing test. So a lot of people dismissed it as very showbiz rather than real science, but it led to some interesting journalism and some interesting chatbots that were developed. But I think the last one was held before, certainly before the release of ChatGPT. I think it was about seven or eight years ago. But yeah, so people have attempted to actually run Turing type tests for many years.
Dan Williams: Yeah, yeah. Okay, before we turn finally to state of the art AI and these deep questions about AI consciousness, maybe we should also just briefly touch on another kind of argument that I sort of hate, but I think has been quite historically influential, which comes from John Searle and it’s the Chinese room thought experiment. Very roughly speaking, Searle asks us to imagine a situation in which you’ve got somebody in a room. They don’t speak a word of Chinese, they don’t understand any Chinese. They’re receiving Chinese text from outside of the room. They are then consulting a sort of instruction manual, maybe a set of if-then instructions, about what to do when you’re given certain kinds of inputs. And then in response, they are producing, sending outside of the room, certain kinds of outputs according to this set of instructions.
And so he says, look, you could imagine the instruction book being designed in such a way that someone just mechanically following that procedure would, from the perspective of somebody outside of the room, come across as coherent, intelligible Chinese conversation. But by stipulation, the man inside the room doesn’t understand anything about Chinese. And then the move is something like, well, what’s going on within the room is in some ways relevantly similar to how a computer or a computing machine works. So what this is supposed to tell you is that doing whatever it is that a computer does in and of itself won’t produce genuine understanding. Is that right? Is that a fair representation of it?
Henry Shevlin: Yep, that’s a very good summary. Maybe just a couple of small points. So just to add extra clarity for listeners, the person in the room is not using a phrase book and translating from Chinese into English and back into Chinese. They have no idea what any of the characters mean. They just have basically a lookup table that says if you see pictures that look like this, if you see characters that look like this, respond with these characters in return. So the idea is there is zero semantic understanding happening inside the room. However, the room as a system seems to produce this appearance of understanding Chinese. And that is the core claim that you can have this appearance of understanding without any real semantic understanding. Hence, the Turing test is inadequate, is insufficient as a test for genuine understanding.
And the other thing to mention is that in the article or the chapter where Searle lays out this argument, “Minds, Brains and Programs,” he’s very much focused on understanding, but in his later work, he connects this to consciousness quite explicitly because his whole theory of what semantic understanding is, of what some real semantic content is, is that it’s ultimately grounded in consciousness. So there’s sometimes a little bit of slippage, but when people shift between understanding and consciousness, it’s very unclear to me that understanding requires consciousness. I don’t think it does. But it’s very clear for Searle, they are two sides of the same coin.
The other thing I’d really emphasize about the Chinese room, two things I’d emphasize, because it is probably worth noting—John Searle sadly died two weeks ago. One of the most influential philosophers of language and obviously the Chinese room was massively influential. Due respect to John Searle, but I will say it is one of my least favourite arguments in philosophy. I think in particular, it’s one of these arguments that’s easy to fit, nod along to without thinking through the details.
So just to give one little example of how it’s probably a lot more complicated than it sounds. Imagine I say to you, imagine we’re having a conversation and I say, “Hey Dan, how are you?” You might respond, “Not bad Henry, how about yourself?” If I say exactly the same thing again, “Hey Dan, how are you?” You might say, “Did you not hear me the first time? I’m fine.” If I say it again, you’d be like, “I’m sorry, is there a problem with the line?”
That’s a very simple example, but it illustrates the fact that language does not work in a series of neat input-outputs, but you actually need to keep track of the entirety of the conversation that’s gone before. So given this kind of combinatorial complexity, it’s a classic combinatorial explosion in even the most basic sentence, right? The kind of scale of the lookup table that would be required for managing even a basic conversation would exceed very quickly the amount of bits in the entire universe.
This is why you can easily say sentences that have never been said before. If I say, “The elephant looked forlornly at the cheese grater,” I can almost guarantee no one has ever said that exact sentence before because all natural languages have this insane combinatorial complexity. So that, I think, makes the idea that you could ever build a Chinese room that literally consisted of a lookup table—that makes it very implausible, as in a lookup table that consisted of just a set of one-to-one correspondences for every possible conversation and every possible extended conversation. So that’s one point I’d emphasize.
A second point is that ultimately, and I think we might come back to this in a second, it’s an argument from intuition. The argument is saying, do you reckon a system like this is genuinely understanding or in later versions is genuinely conscious? And I think basically what a lot of the argument is designed to do is designed to promote this feeling of disquiet or unease or skepticism that a system like that would be conscious. But if we’re approaching consciousness like a genuine scientific problem, for example, then it’s very questionable whether our intuitions have any epistemic or evidential weight at all. I mean, how are we supposed to know if this incredibly exotic, nomologically impossible to build machine—why should we trust our intuitions about whether such a system would genuinely understand or be conscious? I mean, there are theories about the semantics of our mental vocabulary about what words like conscious and understand mean that maybe give our intuition some weight, but you’ve got to do a lot of work to show why our intuitions would have any validity for these kind of cases at all.
Dan Williams: Yeah, no, I’m completely with you there. And it is maybe also worth saying, I actually don’t think this is that relevant, but it is worth saying when Searle was writing this, it was very much the era of what’s called good old fashioned AI, good old fashioned artificial intelligence, where the thought was intelligence in machines is rooted in a certain kind of rule-governed manipulation of symbols according to some program, which you might think very loosely maps on to what he was imagining in terms of a man following instructions in this lookup table.
Even there, I don’t think it makes any sense or is coherent. But these days when we’re thinking about state of the art AI, it’s really, for the most part, a completely different approach rooted in neural networks and the deep learning revolution where you’ve got these vast networks of, in state of the art AI today, hundreds of billions of neuron-like units and trillions of connections between them. And in terms of our actual examples of things which can produce conversation and hold conversation, that’s the kind of underlying architecture that we’re dealing with. Obviously, I’ve described that in a very simplified cartoonish way, but it’s so alien to anything that could even be confused on a foggy night for a man following an instruction book as imagined in Searle’s Chinese room thought experiment. Okay, let’s move on to state of the art AI now.
Henry Shevlin: Cool. Sorry, one quick thing before we do, if I may, because I think on that last point you mentioned, one of the most exciting conversations, one of the most striking conversations I’ve ever had, this was with GPT-4, was discussing the Chinese room with it. And just to give the full context, I was talking out loud, I was driving in my car, having an out loud conversation with GPT as I often do, and I was talking about this combinatorial explosion problem, the fact that in order to build any kind of nomologically possible version of the Chinese room, you would need something like a memory system to keep track of prior interactions. And as soon as you start to fill this out with any kind of remotely plausible architecture, like LLMs, rather than just this pure lookup table where everything is perfectly coded in advance, here is exactly what you should say in this instance—something a little bit more dynamic. At that point, I think our intuitions just get a lot murkier.
Anyway, I said this to ChatGPT, I was ranting and it replied out loud. I’ve got the quote here because I saved it. “You’re spot on, Henry. Searle’s thought experiment often gets simplified. But when you dig into the details, those rule books would have to be incredibly complex to account for context, syntax, previous conversation and so on. Essentially, the rule books would have to be some form of state or memory to handle a genuine conversation in Chinese or any language, really. So in a way, the rulebooks would resemble a state machine, keeping track of prior interactions to generate a meaningful current response. If you look at it this way, it starts to sound a lot like the algorithms that power language models, like… well, me.”
Dan Williams: Wow, yeah.
Henry Shevlin: And it said it with that exact intonation. And I almost stopped the car and I was so stunned at this seeming moment of—it looks a little bit like self-awareness because I should stress in this conversation, we had not discussed language models at all. We were just having a classic conversation about philosophy of mind.
Dan Williams: That is interesting. Okay, well, let’s bring it to these current systems like ChatGPT-5 or Claude or Gemini, state of the art large language models. I mean, in some ways they’re more than large language models as that concept was understood a few years ago because of all of these other aspects which they now involve, to do with the post-training that they receive and the multimodality of these systems and so on. But focusing on these systems, let’s return to this question of, okay, we can talk about what the test should be, but do you think there could be a purely behavioral test, one which doesn’t focus on the actual mechanisms by which a system produces that behavior, that could tell you whether or not that system is conscious?
Henry Shevlin: Oh my gosh, that’s a very tricky question. I think in short, given dominant assumptions about what consciousness is in consciousness science, the answer I would say is no. Because basically any behavioral test you’re going to adopt is going to have to come with certain metaphysical assumptions. And there is no straightforward or even probably possible way to test those metaphysical assumptions.
So let’s just take a view that a lot of people hold, which is that only biological systems can be conscious. That just as a matter of basic metaphysical fact, only systems that have metabolism that are alive can be conscious. No matter how good the behavior is, how complex the behavior is, even if we basically produce a perfect simulation of a human brain down to simulating individual neurons or even at the sub-neuronal level, that’s still not going to give rise to consciousness. So that is a perfectly possible metaphysical position you could hold. It’s one that I think is—that I don’t agree with. But equally, I can’t give like a clear refutation against it, right?
And I think there’s a whole bunch of different metaphysical assumptions you need to make to say anything about machine consciousness, its presence or absence or possibility. And those do not seem to be testable, number one, and two, they’re massively contentious. So you can propose various tests, but they’re only going to make sense given certain metaphysical assumptions.
Dan Williams: Yeah, okay, that’s good. Let’s actually—I mean, this is not necessarily where I was expecting this to go, but I think it’s actually really interesting—which is that position you’ve just mentioned that says there’s some kind of essential connection between the fact that we’re biological systems and maybe we’ve got a certain kind of material constitution, a kind of carbon-based material constitution and consciousness. I find this view baffling honestly, but there are very, very smart and intelligent people who argue for this view.
Maybe we could say something about what this view is positioned against, which is an alternative perspective, often connected to what philosophers call functionalism that says, roughly speaking, psychological states, including those connected to consciousness, should be understood functionally in terms of what a system could do. And if you think that, then you’re going to think psychological states are substrate neutral. So it might happen in the case of human beings that how we do the things that human beings do, like perceiving and imagining and reasoning and deliberating and so on, it might happen to be the case that those functions are performed by ultimately carbon based matter. But if you could build a silicon based system that could perform the same functions, then it would be accurate to describe that system in this psychological vocabulary of perceiving and understanding, and also potentially experiencing, if you think that you’ve got a functionalist view of experience.
Now, I find that view, I mean, as with philosophy, it’s difficult often to precisely articulate exactly the content of the appropriate formulation of that view, but I find the intuition very, very plausible. I find it weird the idea that if you could build a system that’s functionally identical to a human being, but it’s just made of different stuff, then we should deny that that system has consciousness. But as we’ve said, there are some smart people who disagree with that, who think that no, consciousness is essentially connected to the fact that we’re living systems with a carbon-based substrate. Why do they think that? So what’s the motivation for this biology-centric view of consciousness?
Henry Shevlin: So I should say I’m probably not the best person to steelman this view because frankly I’m also very sympathetic towards functionalism. Maybe one thing I’ll just quickly say to slightly problematize functionalism before we get into motivations here is it’s very hard to know—so you say, if a system does the same function, it’s functionally identical to humans, right? But there are different levels of granularity there that we can talk about. So I presume probably you would say that a system that’s functionally identical to humans just at the level of behavior, but works completely differently on the inside—it’s at least conceivable that that system is not conscious, right?
So you might say, okay, well, behavior is not quite enough to guarantee that the system is conscious in the same way that we are, at least. So we need to move below the level of function to something more like architecture. So people talk about microfunctionalism. Maybe the system needs discrete areas of like perception, maybe it needs something more like working memory, but you can still say, but how different could the system get while still being conscious, right? So functionalism is—I think I’m very sympathetic to some form of functionalism, but actually spelling out how similar a system would need to be in terms of its internal organization, in terms of its cognitive architecture is a really, really tough challenge, unless you just grasp the nettle and say, no, look, if it’s behaviorally identical to a human, that’s all that matters, right? So that’s the most extreme form of functionalism in one sense. So functionalism is—it’s not a simple slam dunk.
But as to why someone might think that, that the biological view that only living systems can be conscious, I think by far the most persuasive and sympathetic defender of this kind of view currently is Anil Seth. I think he’s written some great work on this. I’ve just written a response to him in Behavioral and Brain Sciences. And he says, look, there’s all sorts of very, very complex features of biological systems, stuff like they maintain homeostasis with their environment, they’re self-propagating. You have sequences of biological processes that he interprets through Friston’s free energy principle, which you definitely don’t want to get into now. But basically, biological systems do a hell of a lot more than just produce sentences of English or accomplish tasks, right?
So the idea that those are the only things that matter for consciousness is to beg massive theoretical questions, right? All we know for sure is that systems like us are conscious, and the fact that we can produce text and we can produce verbal outputs and accomplish goals is not even the most interesting thing about us. There are so many other relevant facts to our constitution, to the kind of beings that we are, all this kind of fine-grained biological stuff that really is critical to the kind of things that we are, and we should therefore have at least some reason to think that it might be instrumental in consciousness. As I say, I think we should get Anil on the show really, because I think he’ll do a much better job of steelmanning it, and I’m sure he’d love to come on as well.
Dan Williams: We should do, yeah. Yeah, no, he definitely would, yeah. Yeah, just quickly on that point, I mean, I think this is a bit nerdy and getting into the weeds of things a little bit, but it is an important distinction to make philosophically, which is I think there’s a position that says something like, we should be skeptical that it will be possible in practice to build silicon-based systems that could in fact perform all of the complex functions that our biology can perform. And it’s important to distinguish that view from another view that says even if you could build a system that was functionally identical, it wouldn’t have consciousness.
Those are two very, very different views. They might seem like they’re the same, and I think people often go back and forth between them. But they are different. The second one is making a much stronger metaphysical claim. The former one is ultimately an empirical claim about the capabilities, the capacities that you can get from certain specific forms of matter. And I’m not entirely sure which one Anil Seth is opting for, but I find, to be honest, the empirical claim about the limitations of, let’s say, silicon-based systems—I find that interesting and quite implausible, but I find it more plausible than the other claim, which is that even if you could replicate all of this stuff functionally, it wouldn’t be conscious.
Henry Shevlin: Yeah, I think the first challenge, first claim is basically an engineering challenge, right? It’s like saying, I don’t reckon you’ll be able to build something with a full range of human cognitive and behavioral capabilities just using silicon. There’s too much about our specific architecture that really matters for what we can do. And I mean, this has actually gotten me—it’s closely related to the ongoing debates about, for example, whether the whole architecture of LLMs can scale up to something like general intelligence. You might say, this is a source of current controversy and debate. Maybe we need to go back to something that is much closer to brain-inspired AI. Maybe, in fact, we need to be using literal biological neurons if we want to build AGI. It’s not a position I find super persuasive, but it’s an empirical question. It could turn out that just silicon architectures—and certainly silicon architectures like transformer models are just never going to give us the full range of behavioral capabilities that humans have. But yeah, it’s 100% an empirical question.
Where I really start to disagree though is when people say, well yeah, even if you could do that, exactly as you say, it still wouldn’t be conscious. Because at that point you’re really entering the domain not of engineering or science, but of metaphysics.
Dan Williams: Yeah, I mean, just to sort of double click on that point, I mean, not that I’m a great expert in this area or anything like it, but my sense is not only is it not the case that we’ve got certain kinds of material properties that means we’ll be able to do things that silicon-based systems won’t be able to do, I think rather it’s almost the opposite situation where I think we’ll be able to build forms of intelligence with silicon-based systems which are far more complex and impressive than the sorts that you get with carbon-based systems. But that’s a whole complex conversation and a bit of a digression. Let’s go back to... go on.
Henry Shevlin: Although I will say, I’ll just add one other thought here, which is, you sort of signposted quite helpfully an important distinction that gets blurred between the engineering version of the challenge to AI consciousness and the metaphysical version of the challenge to AI consciousness. But another line that sometimes gets blurred is the difference between the view that AI systems can’t be conscious at all and the view that AI systems can’t be conscious in exactly the same ways that we are.
I think the second claim is a lot more plausible. It may be that the precise qualities of pain or orgasms that humans have, that they would be very hard to instantiate in any purely silicon-based architecture. Or even if you did, you’d need to basically build an entire micro-scale functional model of a human mind or something like that. I find that view very plausible. So I think if AI is conscious and has a different architecture from ours, which it’s almost certainly going to do, then yeah, I imagine its conscious experience will be very different. But that is a move that I think sometimes gets glossed over, going from the claim that AIs won’t be conscious like us to AIs couldn’t be conscious at all.
Dan Williams: Yeah, and that’s such an important point. I mean, just to return also to what we were talking about earlier on, bats are not conscious in the way that human beings are conscious. They’ve got a very different, presumably a very different kind of set of conscious experiences, and yet we don’t think that means that they’re therefore not conscious. And I take the point that there might be even greater differences between the consciousness of the kinds of machines that we’re potentially building and us, but I think the point still stands.
You mentioned consciousness science, and I realize this is a huge can of worms, and it might be that we’ll need to do a whole other episode on this to really get into the weeds of this. So there are metaphysical views about what consciousness is in some very general abstract sense that we’ve already touched on. And then there are views which you find in neuroscience and psychology about the sort of—what’s the appropriate theory about consciousness, which presumably might be consistent with different metaphysical interpretations of the theory. What’s your sense of the big players when it comes to that area of like the theories of consciousness in that more specific non-metaphysically committal sense?
Henry Shevlin: Yeah, so this is a really important distinction between metaphysical theories of consciousness, like physicalism, dualism, and so on, and scientific theories of consciousness. Just to add a little bit of autobiographical detail here. So I spent four years banging my head against metaphysical problems in consciousness, and then was lured away into philosophy of cognitive science towards scientific approaches to consciousness, which seemed to me to be potentially a lot more—I won’t say interesting, but more fruitful or productive.
Now, to be clear, these are not trying for the most part to answer the same questions as the metaphysical theories. These are not trying to answer, to solve the hard problem, the problem of why there is consciousness at all. Instead, they’re for the most part trying to understand which kinds of brain dynamics or functional dynamics or informational dynamics are associated with conscious versus unconscious processing.
So I think probably modern consciousness science in many ways you can see—not the very beginning, but it really starts to come into its own in the early 90s with the work of people like Christof Koch. Actually, maybe I could push it a little bit back to the 80s and the work of people like Bernie Baars. But let’s talk about what was happening in the 90s. In particular, Francis Crick, one of the discoverers of DNA, has this lovely paper where he basically claims consciousness should be approached from a scientific angle. I’ve got a nice quote from this I can read: “No longer need one spend time attempting to understand the far-fetched speculations of physicists, nor endure the tedium of philosophers perpetually disagreeing with each other. Consciousness is now largely a scientific problem. It is not impossible that with a little luck we may glimpse the outline of the solution before the end of the century.”
And he was writing that in 1996. Suffice to say, that’s not how it worked. That’s not how it played out. But one of the projects that Crick and others contributed to was this search for neural correlates of consciousness. So we know about unconscious perception, for example, versus conscious perception. What’s going on in your brain that distinguishes the two cases? So if I show an image just below threshold, so as far as you’re concerned, you didn’t see anything, and I show it just above threshold, so you are aware that you’ve seen something, what’s going on in terms of brain dynamics that distinguishes those two cases?
And this question, this way of framing the question, it’s very relevant, for example, if we’re interested in things like predicting recovery of patients in persistent vegetative states or identifying lingering consciousness in people in minimally conscious states. And it’s given rise to a whole host of different theories of consciousness, scientific theories of consciousness. So there are many of these and new ones are being added all the time. I think it’s safe to say that none of them have won the room.
But the big ones are views like global workspace theory, which basically says consciousness is a kind of way that information gets shared across the brain. So information is processed in the brain—I should say more neutrally in intelligent systems. If information is localized, only available to some subsystems, it’s non-conscious. But when information is available to all subsystems, or a dedicated global workspace, as it’s called, basically a dedicated broadcasting network across the brain, that’s when information becomes conscious.
Another example of an influential theory is higher order thought theory that says information in the brain becomes conscious when it is the target of a further higher order thought. So at some level, your brain—you think to yourself, not consciously, although that’s a messy question—at some level your brain is representing, I’m having a perception of red right now, or I’m perceiving a red apple. So when you get a higher order state turning a spotlight as it were on a first order state, a first order mental state, that first order mental state becomes conscious. Now that’s a very crude...
Dan Williams: Can I just ask, just really quickly on that as a follow up, because I think this might be occurring to people. So in the global workspace theory, the conscious states are those that get broadcast to other cognitive systems within the system, within the brain in our case, but you can imagine AI systems working like this. Whereas the higher order thought theory says a psychological state becomes conscious when it’s the target of another thought. And the difference between those is in the global workspace theory, whether or not a state gets broadcast to these other cognitive systems is not a matter of whether it’s the target of another thought. Is that correct as a way of understanding the difference? Okay.
Henry Shevlin: Exactly. Exactly. Yeah. So on a global workspace theory, it doesn’t matter whether a thought is the target of another thought or a mental state is the target of a thought, a higher order representation. As long as it’s just available to everything, it’s conscious. Whereas for the higher order theorist, only those thoughts that are directly thought about or any mental states that are directly thought about are conscious.
Dan Williams: Yeah, okay, so those are two influential theories. As you’ve mentioned, there are a ton of theories, maybe not a ton of influential theories, and potentially people having their own theories that they’re introducing all of the time. I mean, from your perspective then, I suppose one way in which you could approach AI consciousness is you look at scientific theories of consciousness, which have been developed, presumably developed entirely by looking at examples of consciousness either in human beings or to a lesser extent maybe in other animals. You take those scientific theories of consciousness and then you sort of conditionalize and you say if such and such a theory is correct then what are the implications of that theory for thinking about AI consciousness? Is that how you think about this topic or do you approach things differently?
Henry Shevlin: Well, I think that’s a productive way to think about it. So there’s this wonderful report by Patrick Butlin, Robert Long and others from a couple of years ago called “Consciousness in Artificial Intelligence: Insights from the Science of Consciousness.” It basically does exactly that. It says, let’s take all of the leading theories and say, what would an AI system need to do in order to be conscious by the lights of these theories and do any current AI systems do it? And what they find—it’s a very long, very good report—but basically is that consciousness does not seem like the kind of thing that’s impossible for current architectures to realize. In some cases, not for every theory, but for many theories, even with fairly minor tweaks, existing kinds of architectures could give rise to conscious systems.
But there are a couple of problems with this. The first big one, of course, is that there are literally hundreds, if not thousands, of different scientific theories of consciousness. And I think they’re basically never refuted. New theories are constantly being added to the table. The second thing is that as they do in the report, right, there are different ways of operationalizing these theories of consciousness.
So for example, global workspace theory in its modern form, sometimes called global neuronal workspace theory, it’s associated with Stanislas Dehaene, a fantastic cognitive neuroscientist. And he’s got a great book called Consciousness and the Brain. He spells out what global workspace theory says, but he spells it out in a few subtly different ways. At one point he says something like, consciousness is any kind of system-wide information sharing. At other points he says, consciousness occurs when information from working memory is made globally available to modules including episodic memory, language and so on.
Hang on, those are two quite different claims, right? The first one suggests that even quite simple architectures might be conscious, maybe some existing architectures. The second one makes it sound like only creatures with our specific sorts of cognitive organization can be conscious. So even with our existing theories, there are different ways of spelling them out. This is something I go into in a paper called “Non-Human Consciousness and the Specificity Problem.” Different ways of unpacking or operationalizing them that have potentially very different conclusions for whether or not AI systems or for that matter non-human animals are conscious.
Dan Williams: Yeah, okay. I mean, yeah, that seems right. I suppose one issue here, which is hanging over the entire thing is understanding of consciousness, philosophically, metaphysically, scientifically, there’s so much uncertainty still that all of that is then carrying over to these issues of AI consciousness in a really significant way.
I mean, maybe we can just end with two overarching questions. The first of them, I think, follows pretty directly from what we’ve been saying. The second is a question which connects to, I guess, social, political, ethical stuff. So the first question I think we should look at is, okay, in light of all that we’ve said so far about the metaphysics of consciousness, the weirdness of consciousness in the scientific world, these different scientific theories, and so on, how should we actually think about state of the art AI systems? What are your views about that? And then the second question is what’s at stake here? Why is this an important issue? Because I think there are ultimately, you know, there are very interesting scientific, there are very interesting metaphysical questions here, but there are also presumably very, very important ethical questions when you’re dealing with the possibility of conscious machines that’s hanging over the conversation as a whole.
Taking that first question, I mean, what’s your sense, given your expertise in this area, given your views in this area, if you take a system like Claude, Gemini, ChatGPT, what’s your sense? Are these systems conscious in some sense, is there something it’s like to be them?
Henry Shevlin: So it’s a very reasonable question and it’s one I don’t have a good answer to. I think basically the only kind of answer I can give, given the massive uncertainty, is to hedge across so many different theories, so many different methodological approaches. Probably my conviction is that basically we don’t know our ass from our elbow when it comes to what consciousness is or how to measure it. Therefore, I think we are basically in a state of near total uncertainty when it comes to consciousness in AI systems.
That said, I’m a good Bayesian, I can deal with all this. So if I had to put numbers on it, they would come with huge error bars. But I think there’s a non-trivial chance that some existing AI systems have at least some minimal form of consciousness. And in particular, we don’t want to get too deep in the weeds here, but I don’t think it’s likely that any AI systems feel pain or have perceptual experience. But there’s a type of consciousness that’s sometimes called cognitive phenomenology. Think about the kind of experiences you have when you’re reasoning through a problem or come to a sudden insight or comparing two different ideas in your head without any accompanying visual imagery, just the raw processing of concepts. If you think there’s some kind of conscious experience associated with that, it doesn’t seem crazy to me to think there could be some kind of analog of that in AI systems.
And I guess one reason I’m a little bit more open minded about that than I think some people is because I’m pretty liberal about consciousness in biology. I think I’m very high credence, probably above 80%, that honey bees, for example, are conscious. I think it’s just the best way of understanding complex behavior in honey bees. There’s a whole big story there. But the point is I’m pretty liberal about where I think consciousness extends in nature. It can arise in quite simple systems, which I think pushes me towards being a bit open-minded about the possibility of consciousness in AI systems.
But it’s worth just really emphasizing the degree of uncertainty among experts on this. I’ve got some nice choice quotes here. So back in February 2022, Ilya Sutskever, former chief scientist at OpenAI said, “It may be that today’s large neural networks are slightly conscious.” This happened on Twitter, like all good philosophical discussions. And he got a reply from Yann LeCun, who’s obviously now head of AI research at Meta, or at least was until recently. Very prominent AI researcher at Meta.
Dan Williams: And also we should add a critic of the large language model based approach to AI. Sorry to interrupt.
Henry Shevlin: No, absolutely. It’s an important addendum. So he replied to Ilya saying, “No, not even true for small values of slightly conscious and large values of large neural nets.” But Murray Shanahan of DeepMind, I thought had the best reply here. He said, “They may be conscious in the same sense that a large field of wheat is slightly pasta.” Which I think is just brilliant and hilarious. So in other words, you’ve got the raw materials there, but it hasn’t been turned into the finished product. That’s one way of interpreting that.
And on the philosophers, philosophers are just as divided. Dave Chalmers has said, “Questions about AI consciousness are becoming ever more pressing. Within the next decade, even if we don’t have human level artificial general intelligence, we may have systems that are serious candidates for consciousness.” But Dave’s colleague at NYU, Ned Block, another one of the titans of modern consciousness, says, by contrast, “Every strong candidate for a phenomenally conscious being has electrochemical processing in neurons that are fundamental to its mentality.” And my old PhD supervisor, Peter Godfrey-Smith, also said in his book, “If the arguments in this book are correct, you cannot create a mind by programming some interactions into a computer, even if they’re very complicated and modeled on things that our brains do.”
So I think that just gives you a sense, these are all titans in their fields, just how divided opinion is on this issue.
Dan Williams: Yeah, right. You can’t just trust the experts, as people like to say with that slogan. I mean, I should say, I actually share your view, and there’s massive uncertainty, but it’s certainly not absurd to think that there’s something it’s like to be these systems that are state of the art, even though we shouldn’t think that what it’s likeness is anything like what it’s like to be a human being. But just to play the devil’s advocate view, I think lots of people think that is just kind of crazy. Actually, we should be certain there’s nothing it’s like to be these systems. They’re chatbots. They’re doing next token prediction on vast bodies of text. That’s not quite right, actually, for some of these systems, but that’s the kind of view, right? They’re stochastic parrots.
I saw a post the other day on another social media platform, Bluesky, with its own pathologies as a social media platform. And that was very much the spirit of the post. And from what I can gather, this is also the spirit of the popular opinion on Bluesky, which is just, it’s kind of absurd to even be talking about consciousness in a large language model. And maybe not even just absurd, but also potentially kind of dangerous or troubling or buying into the hype of these profit-seeking corporations and so on. So I don’t believe any of that, but I want to throw that at you and get your response.
Henry Shevlin: Yeah, so I find it a little bit baffling that people would see this as an offensive question. I mean, I’ve had people say, it’s offensive to think that an AI system could be conscious. And I just want to say, look, this is a scientific question, right? And it’s a philosophical question. And honestly, it doesn’t even necessarily have any direct normative implications. Maybe we can get to that in a second. But I mean, simply saying there might be some basic forms of cognitive phenomenology in an AI system—that doesn’t entail robot rights by itself, for example. So anyway, I think I don’t get the offensiveness angle.
One argument that I have engaged with, and I think it’s worth unpacking a bit, is this idea that it’s just matrix multiplication, or it’s just next-token prediction. And without wanting to go on too long a digression here, I think people really need to go away and read their David Marr. David Marr, one of my absolute heroes in cognitive science, died tragically young, but wrote a very influential book, his one and only book published posthumously in the 1980s on vision. And one of the basic insights he brings to bear is that look, almost any complex informational system, whether you’re talking about human vision or a computer, can be analyzed at multiple levels of explanation.
So you’ve got the high level functional, what he called computational explanation, which is what is this system or subsystem doing? So in the case of a part of your vision, it might be the system detects edges. Then moving down a level, you’ve got the algorithmic explanation. So how is that function accomplished in informational terms? What kinds of computations are being done to calculate where an edge is in your visual field? And finally, you’ve got the implementational level explanation, like what neurons are doing what, what circuits are doing what, how is this algorithm actually instantiated? And the point is almost any system is going to be analyzable at multiple levels right?
So at some level there’s going to be a mathematical gloss on what’s going on when I’m thinking through a problem in the human brain, at least if you accept even the most basic form of scientific naturalism, right? And even if you think there’s much more that’s going on, sure there may be in the human brain, but we’re also doing the computations, right? There’s going to be at least a computational level of description. And you can see this so clearly in a lot of perception, for example, where people produce very accurate predictive models, computational models of how early vision works or how early linguistic processing works.
And the fact that you can give the implementation level, algorithmic level, lower kind of functional level explanations of what any system is doing doesn’t exclude psychological or phenomenal level descriptions at all.
Dan Williams: Right, right, yeah. Such an important point. And I think one of the things that being a philosopher, or at least a good philosopher, trains you to do is whenever you encounter someone saying X is just Y, for alarm bells to go off, because I think that often smuggles a whole lot of very, very dubious ideas. Whether these systems are just minimizing prediction error on a next-token prediction task, or human beings are just complex biophysical machines or whatever. There’s a lot going on when people say a comment like that, a lot that’s getting smuggled in that needs to be actually thought about rigorously in a way that it’s often not.
Okay, we’ve touched on this already. Let’s end with this issue of what’s at stake. So I mean, this is, surprisingly to me at least, but it’s undeniably kind of a polarized area of discussion, both AI generally. I think there’s lots of heated, let’s say, conversations about how to make sense of this technology, but also specifically when it comes to these issues of AI consciousness or sentience, as people often describe consciousness in sort of popular conversation. What’s your sense of what’s at stake? Why does this matter? Why is this an important conversation?
Henry Shevlin: Yeah, so I think this is a really key point. And maybe we should have even mentioned this at the start of the program. One of the reasons why consciousness matters so much more, I think, than most other kinds of psychological states—we can argue about whether LLMs have beliefs or understand, but what makes consciousness so important is its connection to ethical issues, right? So there’s this famous passage in Peter Singer, one of the godfathers of modern utilitarianism and effective altruism, where he says, “A schoolboy kicking a stone along the road isn’t doing anything morally wrong because the stone can’t suffer. It’s not conscious and it therefore doesn’t have any interests, right?”
So one of the reasons we care about consciousness is because consciousness seems like a prerequisite on many views for having interests at all. If you can’t suffer, if you can’t feel pain or orgasms, if you can’t have positive or negatively valenced experiences, right? Then it’s very unclear whether you deserve any kind of moral consideration at all, or at least you get a lot of extra moral consideration by virtue of this.
And this is also one of the reasons why I think consciousness in animals is what actually got me so interested in animal consciousness, because one of my favorite essays of all time is David Foster Wallace’s “Consider the Lobster.” Fantastic read, highly recommended. And when a chef drops a lobster into a pot of boiling water, it seems to me to matter a great deal whether there’s something it’s like for that lobster to experience that, whether it suffers pain and suffering. That seems like a very important question. And equally, if we’re thinking about which animal welfare interventions to prioritize, right? Is it worth spending money on shrimp welfare, for example, to give a controversial area? Right? It seems to matter a great deal whether there’s anything it’s like to be shrimp, whether they can genuinely suffer. And so I think consciousness has this special normative connection that just isn’t clearly shared by any other psychological concepts. And that’s part of what makes it so important.
Dan Williams: Yeah, I completely agree with that. I’m going to maybe just end with a discussion potentially of the contrast between those cases where we’re thinking about non-human animals and these cases where we’re thinking about AI systems. I mean, there are two mistakes you can make here, right? There’s a kind of false positive where you attribute consciousness where it doesn’t exist or a false negative where you fail to acknowledge consciousness that does exist.
It seems to me that when it comes to, say, lobsters, we should err on the side of caution. I think there’s very, very good reason to think there is something it’s like to be a lobster. But fair enough, there’s uncertainty. It doesn’t seem like it’s the end of the world if we have a false positive here and that stops us from burning them alive. It seems, I mean, beyond egregious, on the other hand, to make the mistake of failing to take into consideration that they’re conscious.
With AI systems, I suppose it’s complicated because clearly there are real big issues here if we are in fact manufacturing conscious systems and we’re not recognizing them as conscious. But I can also see the argument from people who say there could also be big issues here if we’re just building—just to use that word—if we’re building machines that aren’t conscious, there’s nothing it’s like to be these systems, they’re sophisticated chatbots or whatever, and we’re treating them as conscious. To me at least, I can understand what the downsides might be in that kind of scenario. I think there is something a little bit troubling if people start treating systems that aren’t conscious, that is AI systems that aren’t conscious as if they are, in a way that when it comes to these other cases, I’m not so sure. Have you got any thoughts about that, just to sort of wrap things up?
Henry Shevlin: Yeah, absolutely. So I think another really important difference here is that there are some states that we undergo that we recognize as very, very bad, like extreme pain, starving to death, extreme nausea, that seem to have fairly straightforward physiological analogues in non-human animals. And I think that justifies a pretty strong precautionary attitude about inflicting those kinds of states on animals, right? Sticking me in a pot of boiling water, I can tell you would be pretty horrific. And so it’s probably a good idea not to do the same things to creatures that have relatively similar behavioral responses to pain to me, right?
But it’s just much less clear what it would even mean for an LLM to suffer, right? They don’t have bodies. They don’t have any kind of somatosensory processing. That doesn’t mean they can’t suffer, right? But it makes the question of what suffering in LLMs would look like a lot harder to answer. So I think one model for thinking about how LLMs work, that Murray Shanahan has popularized, is that they sort of role play. So if Claude says, like, this is awful, I’m really distressed right now, right? Is that more like an actor who’s portraying Romeo on stage? “My heart is shattered by the death of my beloved Juliet,” right? Or is it actually suffering? I think this is one of the things that makes AI welfare really hard.
My general sense here is that this is an area where we absolutely need better understanding, but also just better theoretical models of what it would even mean for an AI system to suffer in the first place. And I also, I think I am sympathetic to the idea that precautionary principles are at least easier to apply in fairly straightforward ways in the animal welfare case compared to the AI case.
That said, I also don’t think we can rest easy in the AI case. And partly, just to give one little example, suffering in biological organisms seems to be relatively biologically constrained. You have various kinds of negative feedback mechanisms like endorphins and so forth, just because suffering is generally not a particularly adaptive state to be in. It’s a powerful way that your body can send a signal that you’re injured or you desperately need to eat and so forth. But there are biological dampening mechanisms in most cases, not in all, but in that sort of create a kind of upper limit.
But it’s not clear that those would arise spontaneously or by design in AI systems. So the kind of theoretical upper limits of the extremes of suffering in AI systems may be less constrained than in biological systems. All that is very speculative, of course, but just to note that there could potentially be a lot of suffering associated with badly designed AI systems. And of course, if you’re dealing with systems that can have billions or trillions of instantiation simultaneously. That could quickly add up to some really messed up things that we’re doing.
I don’t think, for what it’s worth, I’ve not seen any compelling evidence, or I don’t think there’s much compelling reason to say any specific things we’re doing with AI right now are plausible candidates. There’s no AI equivalent to factory farming that’s obviously objectionable. Also, just to forestall one argument that I hear all the time, and I think is just so—it really frustrates me is when people say, the only reason people want you to think AI machines suffer is because AI companies want that extra recognition for the value of their products. I can tell you this is the last possible thing in the world that any tech company wants. The idea that their products that are making them billions of dollars right now, the idea that they might have rights, would be really, really bad for their business models.
Dan Williams: Right.
Henry Shevlin: Insofar as there are people pushing for greater awareness of AI welfare, they are not operating with any kind of commercial agenda in mind. And I think commercial agendas, in fact, push in the opposite direction.
Similarly, also just because—it’s so interesting how many people get really angry about this debate, the fact the debate is even happening. I meet people who’ve said it’s obscene to even debate the idea that AI systems might one day deserve rights. What I would say to these people is look, even if you think, like, even if your cognitive science of sentience or consciousness or theory of moral patency, even if you think there’s basically no chance that any AI systems are conscious, well, you should be engaged in this debate because a lot of people are going to take that seriously. And if you think they’re making a mistake, you need to engage with them and tell them why they’re making a mistake, right? It’s not a debate that I think you can just dismiss with a look of disgust, right? If you think that we’re in danger of making massive false positive ascriptions of moral status to AI systems, you need to tell people why and actually have that conversation, rather than dismissing it with a grimace of disgust.
Dan Williams: Yeah, yeah, just dismissing it doesn’t seem like it’s an option now. And I think as these systems develop in sophistication and their capabilities develop, that that conversation is going to become more and more important. And I think one thing that you alluded to there is there’s a question about how we should treat these systems. And there’s a question about how human beings will treat these systems. And I know that connects to some of your interests when it comes to things like social AI and so on. But we should postpone that to a future conversation. Henry, this was fantastic. Any final word, final comment that you want to end on?
Henry Shevlin: All right, so I’ll close on two reflections. The first is, on the one hand, as listeners who were not previously familiar with consciousness debates will have realized, it is possibly the messiest debate out there. Both scientifically, theoretically, metaphysically, it is an absolute snake pit of a debate. But don’t be put off, because I think it is also the most fascinating and rewarding question that I’ve ever worked on. I have happily dedicated basically most of my academic life to working on consciousness and I don’t regret it for a second.
