Artificial Superintelligence, Safety and Security

Thanks a lot for inviting me.

I am a professor. I do research in computer science, more specifically on artificial intelligence, safety and security, and I work at a university. I teach and do research.

Well, I always wanted to work on cutting-edge technology and on something important. I don't think there is anything more important than creating artificial minds and making sure they behave.

I don't think it matters what we call it. It's just a label, and artificial just means that it's made. It's not naturally occurring. But you can call it whatever else you want. It wouldn't change a thing.

Well, intelligence has many definitions. One of the best I encountered talks about accomplishing goals in different environments. So whatever it is, a human or an artificial agent, I don't think it makes a difference in terms of what it is we are observing and measuring when it comes to intelligence.

Right, so that makes the whole subject a lot more complicated. You go beyond intelligence to things like consciousness, which we are not very good at detecting, measuring or in any way producing. So I think we could potentially have a very capable intelligence which was not conscious. That would not be a contradiction.

And I don't think at this point there is anyone, in terms of artificial agents, who would get offended by being called something less. They just don't have that capability yet.

So I agree. It's a very important question, a fundamental question about consciousness. And I have done some work showing that perhaps consciousness is a side effect of intelligence. You cannot avoid it. So it's very possible that our machines in fact do have some rudimentary internal qualia, and we'll have superintelligence with super-consciousness as a side effect.

A lot of what I do is not just about the narrow systems we have today, but about futuristic systems, and those are superintelligent systems. It's hard to show any more respect than calling something superintelligent, compared to us barely intelligent humans. So I think there is a possibility that they will be greater than us and more conscious, and have multiple streams of consciousness and deeper experiences.

In what sense? So there are other conscious beings out there. It doesn't mean anything for me. What are you asking about?

So if it's superintelligent, our concerns are about control: who is in charge, who's making decisions, who is surviving. Whether that intelligence is conscious or not is not as important. Let's say it's a very malevolent intelligence, and you have this Terminator chasing you, trying to kill you. I don't think you care if it feels anything or not. It's the same outcome.

With low-level AIs it matters, because if they are conscious, we cannot exploit them. We cannot use them in certain ways that would be unethical. But again, if it's superintelligent systems we're talking about, I think their intelligence dominates the conversation over their internal states.

Well, I think it's very realistic. We'll definitely get there. As soon as we get to human level, because of how computers operate in terms of memory and access to knowledge, it immediately becomes superintelligent in a certain sense.

As to timelines, I've seen predictions from seven years to hundreds of years. I don't know for sure. 2045 is a number a lot of people come up with, based on different data streams.

I would be surprised if by 2045 we didn't get there.

I would be surprised if we didn't have it by that point.

Well, we rarely discuss not just "Can we do it?" but "Should we do it?" Everyone's working on making the systems more capable, getting there as fast as possible. But so many people around the world never had a chance to understand what the issue is, or to consent to it. And this technology will definitely impact everyone.

So we're essentially performing an experiment on billions of people. We're starting it already with children today: we give them iPads. "Oh, let's see what happens in 20 years if you spend a couple of months a year sitting on an iPad. We'll find out." And likewise, if we do develop this more capable intelligence, maybe it will be controlled and beneficial, maybe not. But we haven't had this conversation with the general public.

I'm not sure we need quantum computing to get there. It might speed things up, if computers are a limiting factor. I think the most impact will be in areas like cryptography, where a lot of existing protocols will be broken. A lot of private information will be released. Cryptocurrencies will be impacted. So I think short term, that's where most of the action will happen.

As far as I can tell, I haven't seen a learning algorithm which would heavily depend on quantum computing in order to perform. But with quantum computing, it becomes possible to do integer factorization, and then it may be problematic at some point.

Specifically with security, we don't have much at all. We always work on developing a product and releasing it, but we rarely consider how insiders or outsiders can hack it or manipulate data. We're just starting to see a little bit of it with self-driving cars, but in general, AI researchers never considered safety and security questions, and many still don't.

If you release a product which has no security and you don't consider what can happen, we are already starting to see [the consequences], even with very narrow systems. For example, there is a famous Microsoft chatbot which was released, and they didn't consider the possibility of users manipulating its learning and feeding it bad data. It seems obvious to consider for something like that, but the system was learning from whoever was feeding it information.

Similarly with self-driving cars: we've seen examples of them being hacked. That makes it possible to take over the controls of the car, kill a person, kidnap a person. Those things are not considered at the time the design of the system is introduced.

Well, we have to include security experts on all projects. Just like in basic software development, where cybersecurity plays a larger and larger role, we have to do similar things with intelligent systems.

Usually after they have an accident, they start to go: "Oh, now it's time to invest in it!"

It seems to work with narrow AIs. In general, all systems have failed. We had cybersecurity breaches, data was stolen, but usually it's just a financial loss. You get a second chance, a third chance. The concern is that with superintelligent systems, you don't get a second chance. If you don't get it right the first time, it's too late.

Right. We saw examples with smart contracts, where a mistake in a smart contract led to the theft of hundreds of millions of dollars worth of value. So that's a great example where the systems are not very intelligent. They have tiny IQs, five points or something like that. But they have to control human users. And that's a similar challenge to us trying to control intelligent systems.

Blockchain, in the end, is just a glorified database. I'm not sure what magical abilities it would give to AI. Almost all of it can be done with other technologies, bypassing this approach altogether. Blockchain allows some decentralization, but with a single AI system, that's not a big issue.

Correct, but this is exactly how computer viruses work, right? You can't turn off a virus. You can turn off the internet if everyone chose to do it, but it's not some magical additional level of protection. Either you have the blockchain software installed and running, or you delete it. Those are the two options. And it's the same with the power supply or internet providers.

In general, based on some research we've done and others have done, you're not going to get a chance to turn off superintelligence. If it's smarter than you, it will predict such actions and strike first, essentially.

Well, you have to be realistic about it. It's not helpful to panic and be pessimistic, or to be optimistic and think, "Oh, it's so smart, it's going to be great." You have to look at those systems realistically. Do they have some sort of control mechanism? What are they capable of doing? Are we capable of controlling that behavior, predicting it, explaining it?

And the answer so far to almost all of those questions is: no, we don't have this technology today. We cannot control AI at current levels really well if it's not deterministic, if it's making novel decisions, if it's learning, if it's operating in new domains. And as it becomes more capable, our capabilities in terms of control go down. That's just the state of the art today. Hopefully it will change, but it doesn't look too promising.

It goes back to the question we brought up before. Should we be going forward with this? Should we put limits on capabilities, on the resources the system can have? Maybe it's not possible to limit it that way.

We are right now experiencing the coronavirus outbreak, and this is exactly the same debate people are having. Should we limit travel, limit meetings, limit our economic capabilities to control it? And there is debate. In some countries, it's very easy to say, okay, everyone stay home, and it works for them. In other countries, it may not work.

But in a way, intelligence is a different type of malevolent technology. We understand biological weapons, we understand chemical weapons, but intelligence can be weaponized just as well, and we don't have the same attitude towards it. Everyone goes: "Oh, it's just software. What can happen?" But it could be much worse.

Well, there is an interesting problem: the people who have the power to do something, the people in control of those major companies, have a very strong financial interest to go forward. And it's very hard to go against your own interests, no matter how smart you are. Even if you understand the problem, it's still very hard to say: "Well, no, we cannot do this. This is not a good idea."

So it would be ideal if we could separate the people making those decisions from the people benefiting from certain outcomes, so they're not combined. Then you'd have the freedom to make a decision that has no financial impact on you personally.

Almost everyone thinks of it as a tool, whereas more and more we're turning to the agent model of AI, where it has independent capabilities. That's a very different type of collaboration, and a different type of adversary. Most people don't see it as such, but once we get to AGI and superintelligence levels, it will not just be a tool for us to use. So don't think of it as a calculator or tax preparation software. Think of it as an alien intelligence from another planet.

If tomorrow aliens came, and they have the technology to travel, so they're smarter than us and more experienced, how would that interaction go? Can you guarantee that it's going to be beneficial for both sides? This is the type of analysis you can do today, and basically decide: do we want them to come? Let's say we have to invite them before they come. Do we want to invite them?

If you look at the history of human civilizations, different islands, the Americas versus the Europeans and so on: any time there was penetration from one group into the other, the one with more developed technology usually exterminated, wiped out, or did something horrible to the less technologically advanced civilization. So it seems to be a very common pattern. It doesn't have to be that way, but so far we don't have any technology to guarantee that it's not going to be that way.

So first, we're going to observe that people, despite being mostly religious around the world, are completely unsafe. There are no safe people, and how dangerous they are is usually in proportion to how much power they have, what they can get away with. Power corrupts. If you give someone absolute dictatorship powers, they become very, very dangerous. There's no exception to that rule that I know of.

We also don't really agree on those ethical principles. That's why we have wars between different religions, because people say: "Your rules are evil. My rules are better. Let's fight over it!" So we as humanity, with the same biological code, the same everything, don't agree on what the code should be.

The classic example with AI is the three laws of robotics, right? You can tell them: don't harm humans, do this and that. It always fails. Those are contradictory rules. You define the rules, and you're dealing with an intelligent lawyer. You can give it a thousand rules, a million rules, it will still find a loophole to do as much damage as it wants.

So this doesn't seem to work. A, you cannot model a safe agent [on humans]. B, we don't agree on the actual rules to follow. And C, even if we managed to come up with a list of rules, it would still find a loophole to bypass them. So this doesn't seem to work.

Not necessarily. A lot of it is miscommunication and undesired side effects. A simple classic example: you have a system which wants to do good, but is just not very aligned with human common sense. And I say something like: "Well, I don't want any people to have coronavirus in the world." One way to do it is to kill everyone who has coronavirus. And it follows the rules. It's doing what it was ordered to do. It's accomplishing exactly what is desired, it just doesn't have the common sense to understand: no, that's not what we had in mind.

And for any example you can give me where this is the rule, I can find a similar, maybe silly side effect, which you would never think about because you are human. You understand that's not what I meant, but a machine just looks for: "What's the easiest way to accomplish this?"

So as I said, it has some impact on robot rights and things like that. We cannot torture it and exploit it, assuming we are ethical beings ourselves.

It may in a way be dangerous for us, because if it understands those internal states of pain and suffering, the qualia, it may become much better at inflicting pain and suffering. It can do a lot of self-experimentation to determine: well, what's the worst thing I can do to you? So in a way it could actually be quite bad if it has this internal access.

So it looks like we would need a little more time to think about it, and a lot more people thinking about it. Right now we're talking about dozens, maybe. We need this to be a major effort. And maybe getting some more time, some sort of moratorium on certain types of technology, may buy us some time. I'm not sure it's possible or enforceable, but in principle, that would be a good thing.

It's happening at a lot of global organizations. The UN is one, the World Government Summit, the European Union: all of them have panels centered on AI, AI ethics, AI governance. But a lot of those things are so high level that they don't actually provide technological solutions. So you get this assurance: "We just signed this document saying AI should be good for humanity." But it has no impact. It's just a symbolic gesture.

Like with the pandemic we're experiencing, it seems like bad things happen, and people start to realize: "Oh, worse things can happen." So in a way, some of those very innocent accidents we have with AI are very good, in that they are starting to open people's minds to the possibility. Well, this silly chatbot can cause this much damage, and this self-driving car can cause this much. What happens when a defense AI in control of nuclear weapons has an accident? What happens when an AI designed for developing novel biological agents, novel biological entities, has an accident? That would be bad for us.

And so slowly people start considering something they never considered before. For many years, whether your technology would have a negative impact was not part of the debate. The question was who can do it first, who can get to higher levels. That's all that mattered.

It would help if we shifted our funding and resources from developing, specifically, militaristic, weaponized AI to working more on safety and security issues, at least for a while. It may help. It doesn't mean that there is an actual solution. Every solution I've seen so far has significant problems, and the ones which are problem-free are so futuristic that if I told you, it would sound like science fiction. It wouldn't be accepted by politicians or by regular people.

So first, it's important to figure out: is there a solution? Maybe the problem is not solvable. That's quite possible, and that's part of my research. I'm trying to understand: can we solve the problem? What does the solution look like? Is it desirable? But it's ongoing research. We don't have solutions. We don't have many answers.

Sure. It seems the main problem is that we don't agree. What you want, what I want, what 8 billion people want is very different. You mentioned the simulation hypothesis. If we do manage to control the superintelligent entity, perhaps all of us can get individual virtual universes, where whatever you want happens in yours. You can have other agents in it. You can be alone, you can be king, you can be a slave. It's up to you.

The point is: it's customized, and you don't have to negotiate with anyone. If you want to, you can bring your friends and family. But the point is, we don't have to give up what we consider important to satisfy someone else. That's the only solution I have found so far which allows for this merger of human values. Everything else creates conflict. At best, you have to sacrifice. Somebody is an atheist, somebody is religious, somebody is vegan, somebody likes barbecue. You cannot get all of them to follow the same rules and be happy.

So if you create separate entities, separate universes, personalized universes for everyone, it becomes a possibility. And if you think we live in a simulation, perhaps this is already some experiment by another civilization in that direction.

Correct, and that's what happens, you're right. You have virtual reality, and the final step is that it's not populated with everyone else in your community, and you don't have to compromise. Right now, if you go to something like Second Life, there are other agents, and you just transfer the existing problems to that virtual environment. You can still have the same arguments, whatever they are. If it's your universe, you decide what happens in it. You have complete control. You become godlike.

Sure. It's a scientific way of saying we live in a created world. Religion has been saying it for a while. This is a statistical argument. If we continue developing better and better video games, virtual worlds, at some point they will have very smart AIs in them, conscious AIs, and the graphics will be amazing. And if many people play many video games, many such virtual worlds, how do you know if you are in the real world or in one of millions of those games? Statistically, it's more likely that you are in a game.

And that's the argument: that in the future we'll do this, and that we are right now living in such a simulation. We are agents in that very realistic-looking world.

As I mentioned, I'm trying to understand what we can do, what is possible. So I'm looking at different limits and impossibility results in AI safety. Results like the unpredictability of the behavior of superintelligent systems: we cannot know what they're going to do. Unexplainability: they are black boxes, and even if they make a good decision, we don't understand how that decision has been reached. And of course the fundamental questions: is control possible? Can we control them? And what does it mean? Is it direct control, is it some sort of implicit control, and what are the limits to that?

Well, in a local context, for my family, I want to provide a better future for my kids. So if I can help create a world where we have beneficial intelligent systems helping us cure diseases and make new scientific breakthroughs, it will be a great success, as opposed to a malevolent superintelligence which does nobody knows what. So hopefully I can look back and say: "Okay, I helped steer research in the right direction."

Thank you so much.