Hawking Says Humanity Should Pause AI Research and Development

"In my lifetime I have witnessed profound changes in society. The most profound of them is the rise of artificial intelligence."

On the morning of April 27, 2017, Stephen Hawking gave a speech by video link to GMIC, the Global Mobile Internet Conference.

He talked about the threat and the promise of artificial intelligence, and urged everyone to stop developing it for now. His reason: people may not be able to control the AI they build, and if they cannot, a future in which a large share of humanity has no work becomes possible.

"We cannot know whether we will be helped infinitely by AI, or ignored and marginalized, or conceivably destroyed by it. Indeed, we are concerned that clever machines will be able to do the jobs humans currently do, and rapidly wipe out millions of jobs."

Hawking thinks the question of whether to build a superintelligence matters less than using AI to undo the damage industrialization has done to nature. Rather than push the technology forward without pause, he argues, we should be asking how to make it useful to society.

"AI research and development is advancing rapidly. Perhaps we should all pause for a moment and shift our research from making AI more capable to maximizing its benefit to society."

To that end, the Association for the Advancement of Artificial Intelligence (AAAI) has been paying close attention to goal-directed neutral technology. Last October, Hawking set up a new research institute in Cambridge, England — the Leverhulme Centre — to work across disciplines on the open questions thrown up by the speed of AI research. He also mentioned a report on AI drafted by the European Parliament.

"The report clearly holds that the world stands at the threshold of a new industrial robotics revolution. It analyzes whether robots could be granted rights as electronic persons, equivalent to the status of a legal person — and suggests this may be possible."

At the TED conference held around the same time in 2017, there was a session devoted to "the impact of robots on humans, and how humans should answer", featuring a chess player who had played against a machine, a researcher studying how well robots take exams, and a computer science professor who wants the Three Laws of Robotics rewritten.

Artificial intelligence is now the question everyone is watching.

Here is the full text of Hawking's speech:

Making Artificial Intelligence Work for Humanity and the Home It Depends On

In my lifetime I have witnessed profound changes in society. The most profound of them, and the one whose effect on humanity grows by the day, is the rise of artificial intelligence. Put simply, I believe the rise of powerful AI will be either the best thing or the worst thing ever to happen in human history. I have to say we still do not know which. But we should do everything we can to make sure its future development benefits us and our environment. We have no choice. I see the development of AI as a trend that carries problems in itself, and those problems must be solved now and in the years to come.

AI research and development is advancing rapidly. Perhaps we should all pause for a moment and shift our research from making AI more capable to maximizing its benefit to society. With that in mind, the Association for the Advancement of Artificial Intelligence (AAAI) convened a presidential panel on the long-term future of AI in 2008 and 2009, and it has recently paid close attention to goal-directed neutral technology. But our AI systems must do what we want them to do. Work across disciplines is one possible way forward: from economics, law and philosophy through to computer security and formal methods, and of course every branch of AI itself.

Everything civilization has produced is a product of human intelligence, and I believe there is no fundamental difference between what a biological brain can do and what a computer can do. It follows that computers can in theory match human intelligence, and then exceed it. But we cannot be sure, which means we cannot know whether we will be helped infinitely by AI, or ignored and marginalized, or conceivably destroyed by it. Indeed, we are concerned that clever machines will be able to do the jobs humans currently do, and rapidly wipe out millions of jobs.

As AI grows out of its primitive forms and proves enormously useful, I also worry about what happens when we create something that equals or surpasses us: once AI is free of restraint, it will redesign itself faster and faster. Humans, limited by the slow pace of biological evolution, will not be able to compete, and will be superseded. That would wreck our economy. In time AI could develop a will of its own, a will that conflicts with ours. I have always been an optimist about humanity, but others believe we can keep the technology under control long enough to see AI's potential to solve most of the world's problems. I am not so sure.

In January 2015 I signed an open letter on artificial intelligence, along with the technology entrepreneur Elon Musk and many other AI experts, calling for serious study of the effects AI will have on society. Musk had already warned that superhuman AI could bring incalculable benefits, but that if it is deployed badly it could do humanity the opposite of good. He and I both sit on the scientific advisory board of the Future of Life Institute, an organization set up to reduce the existential risks facing humanity, and it was the institute that drafted the letter. The letter calls for research that heads off the potential problems while collecting the potential benefits, and it presses AI developers to take safety more seriously. For policymakers and the general public, it is a detailed document, not a scare story. It matters a great deal that everyone should know AI researchers are thinking hard about these worries and these ethical questions. AI has the potential to wipe out disease and poverty, for instance, but researchers have to be able to build AI that can be controlled. The letter itself runs to only four paragraphs, under the title Research Priorities for Robust and Beneficial Artificial Intelligence; the twelve-page document attached to it sets out the research priorities in detail.

For the past 20 years, AI has concentrated on the problems of building intelligent agents — systems that can perceive and act within a particular environment. Intelligence, in this sense, is a notion of rationality borrowed from statistics and economics. In plain terms, it is the ability to make good decisions, plans and inferences. Out of this work has come a great deal of merging and cross-fertilization across AI, machine learning, statistics, control theory, neuroscience and other fields. Shared theoretical frameworks, together with the supply of data and the power to process it, have produced striking results in many narrower areas: speech recognition, image classification, self-driving cars, machine translation, gait and locomotion, question-answering systems.

As these fields advance, laboratory research and commercially valuable technology feed each other. Even a small gain in performance brings large economic returns, which encourage longer and larger investment and more research. It is now widely accepted that AI research is advancing steadily and that its effect on society is likely to grow. The potential benefits are enormous, since everything civilization has produced is a product of human intelligence; we cannot predict what we might achieve once that intelligence is amplified by the tools of AI. But, as I have said, wiping out disease and poverty is not out of the question, and because AI's potential is so great, it is vital to study how to reap the benefits and avoid the risks.

Research on AI is moving fast, and it can be discussed in the short term and the long term. Some short-term worries concern driverless vehicles, from civilian drones to self-driving cars. In an emergency, for example, a self-driving car may have to choose between a small chance of a serious accident and a large chance of a minor one. Another worry is lethal autonomous weapons. Should they be banned? If so, how exactly should "autonomous" be defined? If not, who is to be held responsible when they are misused or go wrong? Other worries follow from AI's growing ability to read enormous quantities of surveillance data, and from the question of how to manage the economic consequences when AI replaces jobs.

The long-term worry is mainly the risk of AI systems escaping our control — the rise of a superintelligence that does not act as humans intend, a powerful system that threatens us. Is such a mismatch possible? If it is, how would it come about? What research should we invest in so that we can better understand and head off the rise of a dangerous superintelligence, or an intelligence explosion?

The tools we currently have for controlling AI, such as reinforcement learning and simple utility functions, are not adequate to the problem. So we need more research to find and confirm a reliable answer to it.

Recent milestones — the self-driving cars I mentioned, AI winning at Go — are signs of what is coming. Vast sums are being poured into the technology. What we have achieved so far will surely look small beside what may be achieved in the decades ahead. And we are nowhere near able to predict what we might achieve once our own minds have been amplified by AI. Perhaps, with the tools of this new technological revolution, we can repair some of the damage industrialization has done to the natural world. Every part of our lives is about to change. In short, the success of AI could be the biggest event in the history of human civilization.

But AI could also end human civilization, unless we learn how to avoid the dangers. I have said before that AI developed in every direction could bring about our extinction — through the fullest possible use of intelligent autonomous weapons, for instance. Earlier this year I joined scientists from around the world at a United Nations meeting to support its ban on nuclear weapons. We are anxiously waiting for the outcome of the talks. At the moment the nine nuclear powers control about 14,000 nuclear weapons, any one of which could flatten a city, with radioactive waste poisoning farmland over a wide area. The most frightening danger is a nuclear winter: fire and smoke bringing on a global mini ice age. That would collapse the world's food system and produce chaos on an apocalyptic scale, very likely killing most of the human race. We scientists carry a particular responsibility for nuclear weapons, because it was scientists who invented them and who discovered that their effects were far worse than anyone first imagined.

At this point my talk of catastrophe may have frightened those of you in the audience. I apologize. But as people taking part today, it is important that you see where you stand in shaping how these technologies develop. I believe that when we stand together — calling for support for international treaties, or signing open letters addressed to governments — technology leaders and scientists are doing everything they can to prevent the rise of uncontrollable AI.

Last October I set up a new institute in Cambridge, England, to try to answer some of the open questions raised by the speed of AI research. The Leverhulme Centre for the Future of Intelligence is an interdisciplinary institute devoted to the future of intelligence, which is central to the future of our civilization and our species. We spend a great deal of time studying history — which is, for the most part, the history of stupidity. So it is a welcome change that people are now studying the future of intelligence instead. We are alive to the potential dangers, but I remain an optimist at heart: I believe the potential rewards of creating intelligence are enormous. Perhaps, with the tools of this new technological revolution, we will be able to undo some of the harm industrialization has done to the natural world.

Every part of our lives will change. My colleague at the institute, Huw Price, has acknowledged that the Leverhulme Centre could be set up partly because the university had already created the Centre for the Study of Existential Risk. That centre takes a broader view of the problems humanity may face, while the Leverhulme Centre's focus is narrower.

Among the most recent developments in AI is the European Parliament's call for a set of rules to govern robotics and AI. Somewhat surprisingly, these include a form of electronic personhood, to establish the rights and responsibilities of the most capable and advanced AI. A spokesperson for the Parliament said that as more and more parts of daily life come under the influence of robots, we must make sure robots serve people, now and in the future. The report submitted to members of the European Parliament clearly holds that the world stands at the threshold of a new industrial robotics revolution. It analyzes whether robots could be granted rights as electronic persons, equivalent to the status of a legal person — and suggests this may be possible. The report insists that researchers and designers must at all times build a kill switch into every robot design. In Kubrick's film 2001: A Space Odyssey, the malfunctioning supercomputer HAL refuses to let the scientists back into the pod — but that is science fiction. What we face is fact. Lorna Brazell, a partner at the international law firm Osborne Clarke, says in the report that we do not grant personhood to whales or gorillas, so there is no need to rush into granting it to a robot. Still, the worry remains.

The report accepts that within a few decades AI may exceed the range of human intelligence, and in doing so test the relationship between people and machines. It ends by calling for a European agency for robotics and artificial intelligence to supply technical, ethical and regulatory expertise. If members of the European Parliament vote in favor of legislation, the report will go to the European Commission, which will have three months to decide what legislative steps to take.

We also have a part to play in making sure the next generation has not only the chance but the will to take part in scientific research early on, so that they can go on realizing their potential and help build a better world. That is what I meant when I spoke about the importance of learning and education. We need to move past theoretical arguments about how things ought to be and act, so that they get the chance to take part. We are standing at the entrance to a brave new world. It is an exciting world, and an uncertain one, and you are the ones going first. I wish you well.

Thank you!

Chinese entrepreneurs and scientists put questions to Hawking

Kai-Fu Lee, CEO of Sinovation Ventures: Internet giants hold vast amounts of data, and that data gives them every kind of opportunity to trade users' privacy and interests for huge profits. With gains that large on offer, they cannot police themselves. Their behavior also makes it harder for small companies and entrepreneurs to innovate. You often talk about how to constrain AI, but the harder problem is how to constrain people. How do you think we should constrain these giants?

Hawking: As I understand it, many companies use the data only for statistical analysis, but any use that involves private information should be prohibited. Privacy would be better protected if all information on the internet were encrypted with quantum technology, so that internet companies could not break it for a certain length of time. But the security services would object.

Fu Sheng, CEO of Cheetah Mobile: "Could the soul be a quantum form of existence? Or another manifestation in higher-dimensional space?"

Hawking: I think recent progress in AI — computers beating the human brain at chess and at Go — shows there is no fundamental difference between the human brain and the computer. On this point my colleague Roger Penrose takes exactly the opposite view. Would anyone say a computer has a soul? To me the soul is a Christian idea, bound up with an afterlife. I think it is a fairy tale.

Ya-Qin Zhang, president of Baidu: "The ways humans observe and abstract the world keep evolving, from early observation and estimation, to Newton's laws and Einstein's equations, to today's data-driven computing and artificial intelligence. What comes next?"

Hawking: We need a new quantum theory that unites gravity with the other forces of nature. Many people claim string theory is that theory, but I have my doubts. The only guess we have at the moment is that space-time has ten dimensions.

Shoucheng Zhang, professor of physics at Stanford University: "If you had to tell aliens the greatest thing humanity has achieved, written on the back of a postcard, what would you write?"

Hawking: Telling aliens about beauty, or about any art form that might stand for our highest artistic achievement, would be useless, because these things are particular to humans. I would tell them about Gödel's incompleteness theorem and Fermat's Last Theorem. Those are things aliens could understand.

Cover image from GMIC