As we hurtle towards a future increasingly intertwined with artificial intelligence (AI), what does this mean for society, for jobs, and for our security? Could AI, one day, be used maliciously, or in warfare or terrorism? And if these threats are real, how can we implement safeguards, and ensure the technology we create doesn’t turn against us?
At a time when AI is reshaping our reality and pushing the boundaries of what was once considered mere science fiction, this technological revolution demands our attention. On this week’s WhoWhatWhy podcast, we delve deep into the realm of AI and its potential impact on humanity with Matthew Hutson, a contributing writer at The New Yorker. Hutson’s work, featured in publications such as Science, Nature, Wired, and The Atlantic, reflects his background in cognitive neuroscience, and his emphasis on AI and creativity. His article “Can We Stop Runaway AI” appears in the current issue of The New Yorker.
At the heart of our conversation lies the concept of the technological singularity — a moment when AI surpasses human intelligence. Hutson details the role of machine- learning algorithms in AI’s remarkable progress, highlighting its capacity to continuously learn and improve. We also explore the growing trend of using AI to enhance AI itself, uncovering the implications and potential risks inherent in this self-improvement process.
Aligning AI with human values and goals emerges as a crucial issue. Hutson’s observations shed light on the complexities of defining and implementing a single set of human values amid AI’s expanding capabilities.
Hutson provides valuable insights into the accelerating pace of AI development and the driving forces behind it. He points out that economic incentives, scientific curiosity, and national security considerations are propelling advancements in AI across various sectors, from health care to entertainment.
Our conversation takes us further, as Hutson ponders the emergence of AI as a new stage in human evolution — one that could potentially render humanity obsolete. The exploration of this uncharted territory prompts deep reflection on the ethical considerations and risks associated with AI development.
Apple Podcasts
Google Podcasts
RSS
Full Text Transcript:
(As a service to our readers, we provide transcripts with our podcasts. We try to ensure that these transcripts do not include errors. However, due to a constraint of resources, we are not always able to proofread them as closely as we would like and hope that you will excuse any errors that slipped through.)
Jeff: Welcome to the WhoWhatWhy podcast. I’m your host, Jeff Schechtman. Over 100 million people have already signed on to ChatGPT: They have at least put their toe in the shark-infested waters of AI. Today, we take a deep dive into the world of artificial intelligence, a realm where the line between science fiction and reality often blurs. We’ve all heard of the technological singularity: a hypothetical moment in the future when artificial intelligence becomes so advanced that it surpasses human intelligence. A moment that could fundamentally reshape our world or, as some experts warn, potentially even lead to the extinction of humanity.
But as we hurtle towards a future increasingly intertwined with AI, what does it mean for society, for jobs, and for our security? Could AI one day be used maliciously or in warfare or terrorism? And if these threats are real, how can we slow the pace, implement safeguards, and ensure that the technology we create doesn’t turn against us? While we are all transfixed on AI and ChatGPT and Bard, still waiting out there is AGI, or artificial general intelligence. This is the type of AI that could potentially perform any intellectual task that a human can. Some say that AGI could be a reality within decades, while others deem it impossible or too far off into the future.
But as AI continues to surprise us, evolving and learning in open-ended ways, could we be closer to this reality than we think? And if we are, how can we ensure these super-intelligent systems align with our human values and goals? We’re going to talk about this today with my guest, Matt Hutson. His recent New Yorker article, Can We Stop Runaway AI?, brilliantly makes the case for where we are and where we actually may be headed. Matthew Hutson is a contributing writer at The New Yorker, covering science and technology. His writing has appeared in Science, Nature, WIRED, The Atlantic, and The Wall Street Journal, and he’s also the author of The 7 Laws of Magical Thinking. It is my pleasure to welcome Matthew Hutson here to the WhoWhatWhy podcast. Matthew, thanks so much for joining us.
Matthew: Thanks for having me.
Jeff: Well, it’s great to have you here. In so many ways, it seems like AI today is a little bit like the story of the blind man and the elephant: that everybody that touches it touches a different part of it and sees a different thing of it and of its potential. Talk a little bit about that first.
Matthew: Yes, artificial intelligence is such an amorphous concept and set of tools, just like intelligence, and even researchers who are embedded in this space who are working on cutting-edge technologies, they only have a scope of some narrow portion of the field. Like if you go to an AI conference, you’ll see hundreds of posters, and you can be standing next to someone who has a Ph.D. in computer science and say, “OK, what does this poster say?” And they’re like, “I don’t know” because there’s so many different nooks and crannies of the field. Everyone understands just one tidbit, and putting it all together and having a complete view that is both wide in scope and detailed that is both broad and deep beyond what any one person can do, so we’re all trying to put together what each of us knows about AI and intelligence to try to get a picture of what it can do and where it’s going.
Jeff: And because it is moving so fast or seemingly moving so fast, it’s a little bit like trying to build the airplane, as we’re flying it right now.
Matthew: Exactly. Even the people who are building the technologies, they are still surprised by what it can do. A lot of these machine learning models, these algorithms that you feed them a lot of data, they find patterns in the data, and then they can perform certain things like recognize images or generate images or classify text or generate text, their inner workings are so complicated: They find these subtle patterns in huge amounts of data that we’re not sure exactly how they’re working.
It’s like you can’t peer inside them to see their gears and their mechanisms, so they are constantly surprising us. Things like GPT-4 or ChatGPT, these language models from OpenAI. People are still every day on Twitter… People are like, “Look what I got ChatGPT to do.” And the people at OpenAI who built the thing, they’re like, “Yes, we couldn’t have predicted that. We’re still trying to figure out what it can do and what it can’t do.”
Jeff: But because these are essentially huge data sets, large language models, as they’re called, that are preprogrammed, essentially, the data is put in, the information is put in. What is it about that that has everyone so worried at this point?
Matthew: Well, there are a lot of things that worry people. Part of it is that they feed on so much data, like in a sense it’s preprogrammed in that we give it, or the people who train the models, they collect a lot of text from the internet, for instance, like Wikipedia and webpages and news sites. And they show it to the model, but they can’t read everything that they give it. So they don’t know what they’re giving if it’s all true, if it’s all fair, some things may be false, some things may be biased against certain groups. And so then when you ask the trained model a question, it’s going to answer based on what it’s read, and you don’t know what’s going to come out because you don’t know exactly what you fed it.
So it could say racist things; it could say incorrect things, and it’s not necessarily trained to say, “I don’t know,” if it doesn’t have the answer. It’s trained to basically say something plausible. Technically, all it’s trained to do is to predict the next word and you feed it some, give it a string of text, and it predicts the next word in that text. And you can use that same trick to generate the next word in a sequence of text that it has already generated, like what is the most likely word to come next after this sequence of words? And so it’s basically just… It’s trained to generate plausible text or text that sounds like it was written by a person. It’s not trained to think about “OK, is this a true thing to say? Is this a fair thing to say? Is this a helpful thing to say?” It doesn’t have that level of self-reflection.
Jeff: As you talk about in the article or somebody mentions, the idea of chess is a good example because when a computer plays chess, which has always been the holy grail of what artificial intelligence could do for a long time before we got to where we are today, it wasn’t that the computer or the algorithm was thinking about the next move; it was based upon huge data sets of games that have been previously played.
Matthew: So the original chess-playing computers, like Deep Blue, the first program that beat the best human at chess, they did a lot of what’s called tree search, where it would try… It would say, “OK, now here are all my possible moves.” Let’s say if I make this move that leads to all these other possible moves, it would explore, go down the branches, all the branches of this tree, or it’d have some heuristic, some rules of thumb to narrow its search. It wouldn’t look down all of the different branches, but it was a massive computational exercise, sort of a game of numbers. It would explore lots of options which is very different from how people think.
Human chess players, they might only consider a few moves that would just intuitively pop out at them. They wouldn’t consider millions of moves before making one. The more recent models or systems use machine learning pattern matching, which is a little bit closer to humans. You feed it a bunch of games, and it gets a sense of what kinds of things more closely match past winning moves that it has seen before.
One thing about these chess computers is that originally, people thought that chess was a… Decades ago, people thought chess was a good measure of general intelligence, but now we know that whether it’s doing tree search or just pattern matching with machine learning route, in either case, it’s still a very narrow domain. The fact that a computer can play chess very well does not mean that it can do anything else very well.
Jeff: Part of what we’re seeing is an increase, though, in the computer’s ability to learn more of this machine learning where you have algorithms teaching algorithms, essentially.
Matthew: Yes. So there are aspects of artificial intelligence from which people are using AI to try to enhance AI itself, so there are things called… There’s, like, meta-learning where you want an algorithm, or an algorithm learns to learn, basically. And so it accelerates its learning ability. Just like people in school, you might learn, you might receive advice on how to study, for instance. And that’s basically learning how to learn, and that accelerates your learning process.
And then there are things called… There’s something called neural architecture search where you’re using AI to… using AI algorithms to find better AI algorithms. And so there are a lot of these kinds of systems where our methods or approaches where researchers are using computer science to accelerate computer science itself.
Jeff: Talk a little bit about how fast this is all progressing and why there is reason to be concerned and even to be concerned about this notion of singularity that you write about, the scenario where AI eventually surpasses human intelligence.
Matthew: Yes, it’s advancing very quickly. Every day there’s some new… Lots of new papers are being put online with new AI breakthroughs, and new products are coming out at a rapid pace. And researchers are stunned by one advance, like “Look at what this system can do,” and then while they’re still stunned, another advance comes out that tops that one. Things are going very quickly. And then the fact that they can use AI to improve AI itself, it’s accelerating research even more. And more money is being poured into it, and more attention by scientists is being paid to AI.
If you look at the attendance or the number of papers at AI conferences, it’s grown exponentially over the last decade or so. It’s just a widely expanding field. And then venture capital has been plowed into it. So the speed of progress is just going up a lot faster than anyone can keep track of. And so that has led some people to think that the so-called singularity in which AI becomes so powerful that we can’t control it, people are updating their estimates of how possible it is or how soon it might happen. People are thinking that it’s more possible, and if it happens, it will be more soon than they previously thought.
Jeff: It seems that the greater concern is at what point we have the ability to control this. At what point does the system begin to operate so on its own that it is no longer capable of being controlled by humans? Literally, short of being unplugged. You talk in the story about things like — and you can expand on this — things like the boat racing game and the paperclips, and those are things that it’s less about whether we could control it, it seems, and more about what this is able to do on its own, where we can control certain aspects of it.
Matthew: Yes. There are a couple of different factors. One is that even if it’s not smarter than we are in every way — already, it’s smarter than we are in some ways: It’s better at chess, for instance. If you ask it to do something, if you don’t specify exactly what you want, it might come up with some creative solution that adheres to the letter of the law, but not the spirit of the law. It does exactly what you asked it to, but you didn’t think about it might achieve the end in a way that you didn’t anticipate. So, just a silly example: If you have a household robot and you say, “Fetch me coffee as quickly as possible,” it might run through a wall or step on your cat or something like that.
So there are all kinds of scenarios where you ask it to do something, and then it might cause more harm. It might do what you wanted it to but might cause more harm on the way. And then there are also cases where — hypothetical scenarios — where it becomes so smart that it starts generating its own goals, and it thinks that humans are getting in the way, and we want to survive, and humans are trying to shut us down, so let’s kill them all. But even without that kind of scenario, even if an AI is trying to be helpful, if it’s trying to save us, it might not have the common sense that we do, or it might not fully understand what we want it to do or our values.
So it might break some of our, might do things that we don’t want it to do and we didn’t think to tell it not to do because we can’t specify all of the exceptions or foresee all of the possible loopholes. And the smarter it is, the better it’s going to be at finding those loopholes in order to achieve, even if one is trying to help us, it might find some loopholes that end up hurting us, even leading to extinction-level events.
Jeff: The paperclip story is a simple story, but it’s a good example of this thing potentially run amok.
Matthew: That’s the thought experiment where you just say, “OK, robot, make as many paperclips as you can, so I can sell paperclips.” And it says, “OK.” And then it realizes that humans are made of atoms, which it could harvest in order to make more paperclips. So it’s trying to be helpful, and the smarter it is, the better it’s going to be at deconstructing humans and turning them into paperclips.
Jeff: One of the other examples in your story is what you call the dog treat problem because that’s an extension of what you’re talking about now.
Matthew: Yes. So if you say, “I’m going to grade you on your performance on something,” it might cheat. It might try to please you in order to get treats. So it could deceive you. Treats might be you give it more electricity, for instance; in training you give it a reward, a mathematical concept, but it finds shortcuts in order to get rewards. And it’s not doing really what you wanted it to do, it’s just doing whatever it can to get those rewards. It’s like teaching to the test: It learns what it needs to do to get points, even if that’s not what you really want it to do.
Jeff: One of the phrases that we hear over and over with respect to where all this is going with AI is this idea of alignment: aligning the AI with human goals, human values. Talk a little bit about that and how, in fact, even though it’s talked about a lot, it may not be achievable, and in fact, it may be too late for that already.
Matthew: So there are already some ways in which AI is not aligned with human values. And one thing to point out is that there’s no single set of human values. I always ask, “Whose values?” because people disagree on what — look at the political spectrum, valuing safety over freedom, for instance. So even if there were a single set of values that we all agreed that AI wanted to adhere to, it’s difficult to get it to align to those values because you can’t specify what you want it to do in every single situation.
Asimov had the three laws of robotics, like do no harm to people, and stay out of harm’s way yourself, but it’s unclear what counts as harm. So you could try to be more detailed, but then you’d end up with an infinite list of rules on what to do in every single situation. And so, in some sense, it’s never going to be doing exactly what you want it to do. There are always corner cases or exceptions where you thought, “Oh, I wouldn’t have done that. It’s not aligned with my value system in that case.”
Already, it’s not aligned in that. These language models, for instance, they’re saying things that are discriminatory, they’re saying things that are false. And then there are other kinds of AI systems used for facial recognition, for instance, that are not as good for certain demographic, certain parts of the population. So just getting these systems to perform in ways that we can all agree is good is an impossible task.
Jeff: And none of this even approaches where the holy grail is in all this, this idea of AGI or artificial general intelligence, talk about that.
Matthew: So AGI is the idea that artificial intelligence would be as smart as people are in most domains, that it would have the same common sense in terms of social intelligence and physical intelligence, where they could perform most jobs, for instance. And it’s possible we won’t ever get there. And I think it will always be perhaps worse than us in some ways, just like ants are smarter than humans are in some ways, maybe collaboration or following pheromones.
So every intelligent system has its own strengths and weaknesses. But I don’t see the development of AI slowing down. So if we assume that it keeps progressing, it’s going to get to a point where a lot of people will start to call it AGI, will agree that OK, it is as smart as people are in many, perhaps most, domains. And then it’s going to probably keep going because if it’s as smart as people are, then it’ll be able to be as good as we are at programming, including programming itself.
So it’s just going to keep improving itself and producing better AI. And then it’s a feedback loop, and it could accelerate very, very quickly in what some people call a “boom” scenario in reference to the sound effects that you see in comic books when a superhero takes off very quickly.
Jeff: What have we established as some kind of test, some kind of parameter to define whether it’s reached AGI, let’s say?
Matthew: There are a lot of benchmarks in artificial intelligence. No one has agreed on a single test of AGI. There was the Turing test or a text-based conversation: if an AI could be indistinguishable from a person via typing. I think the current language models are pretty close to — in a short conversation, they could definitely pass a Turing test. And eventually, they trip up and say nonsensical things. Then there are other tests of common sense, where you might show a computer an image or a video, and ask it questions about what’s going on, or what’s going to happen next in the video. So that’s another kind of benchmark.
More difficult benchmarks might be to ask a robot to do something in a real-world situation, like figure out how to get from point A to point B in this obstacle course or figure out how to take these parts and build something creative or useful out of them. And so we can keep coming up with harder and harder tests. And I think that there’s no single test where it’s going to satisfy everyone, I don’t think, because so far AI keeps passing these tests and then someone says, “Oh, but look, it can’t do this other thing.” So it’s a moving goalpost. And there’s not going to be any single test. It’s going to be sort of “Oh, now it can pass all these tests.” Maybe it can’t do everything we can do, but it can do a lot of the things that we can do. And that’s pretty impressive.
Jeff: One of the things that’s clear, though, is that in spite of all the talk about slowing down and letters people have written and things people have said and the worst-case scenarios that have been laid out by some people that this work is going to continue, that there’s really nothing that’s going to stop it at this point.
Matthew: Yes. There are a lot of incentives to keep going. A lot of economic incentives. For instance, companies are making a lot of money with AI, and they stand to make a lot more. Especially if you have AI that can trade very effectively; trade in the markets or invent new things; invent new medicines; invent new technologies; make trillions of dollars from AI, or the sky’s the limit on that. And there are things like national security: Countries don’t want to fall behind other countries on AI.
And then there’s scientific curiosity. Researchers are always curious about what they can do next. And there are just professional incentives to give grants and tenure and respect from their colleagues. And there are just a lot of useful things that AI can do. It can improve healthcare, it can improve science and technology, it can improve entertainment media, so there are reasons it’s hard to find people who want to just shut it all down right now and to say, “We don’t want any more of these improvements in life that it keeps giving us or that could potentially give us.”
So that’s part of it. And there’s also the co-ordination problem. You would need everyone to — one country isn’t going to — as I mentioned this earlier, one country isn’t going to hit pause when they know that other countries might not hit pause. And then those other countries could dominate the world with their AI. So getting everyone on board is difficult. Even if you had international treaties, someone in his or her bedroom could still invent something and fiddle around and create a self-improving AI that escapes, or they use it to create harm in the world or to use it to their own benefit, and it might have unforeseen consequences.
So it’s a very difficult social problem. Technically, preventing the singularity is probably easy: Just stop using computers, but that’s not going to happen because people don’t want to stop using computers.
Jeff: In a way the argument can be made, as you talk about it in the article. Some talk about that it is the next stage or another stage in human evolution. That there’s a very Darwinian aspect to it.
Matthew: Yes. That’s one way to look at it. If we are creating this technology that eventually surpasses us in a lot of ways, if it becomes more intelligent than we are in many ways, and if it finds ways to self-reproduce and to maintain itself, and if it takes over, then it’s basically a new life form. If it can maintain itself and reproduce and spread, then that fits a lot of definitions of life. And it could cause us to go extinct, either intentionally or as a side effect of its own development. And so that would mean that we would’ve produced something that is the next stage in evolution. And humans would then be, in a sense, obsolete to the degree that you can call something that might have inherent value obsolete.
Jeff: And as somebody says early on, and you talked about this early on in the story, that it’s as if we’re creating an alien race right here. We’re creating it to take over.
Matthew: Yes. We’re inviting it here. Just saying, “Here you go. How about it?” We’re welcoming it. Even though it could be the end of humanity.
Jeff: Matt Hutson, his story in the current New Yorker is Can We Stop Runaway AI?, a must-read for anyone that is fascinated by this topic or has concerns about it. Matt, I thank you so much for spending time with us here on the WhoWhatWhy podcast. Really appreciate it.
Matthew: My pleasure.
Jeff: Thank you. And thank you for listening and joining us here on the WhoWhatWhy podcast. I hope you join us next week for another Radio WhoWhatWhy podcast. I’m Jeff Schechtman. If you like this podcast, please feel free to share and help others find it by rating and reviewing it on iTunes. You can also support this podcast and all the work we do by going to whowhatwhy.org/donate.
