Recently, Ramez Naam posted What Do AI Researchers Think of the Risks of AI? while guest-blogging at Marginal Revolution. Naam quoted several risk skeptics like Ng and Etzioni, while conspicuously neglecting to mention any prominent AI people who take the risk seriously, such as Russell, Horvitz, and Legg. Scott Alexander at Slate Star Codex replied by quoting several prominent AI scientists past and present who seem to have taken the risk seriously. And let’s not forget that the leading AI textbook, by Russell and Norvig, devotes 3.5 pages to potential existential catastrophe from advanced AI, and cites MIRI’s work specifically.

Luckily we can get a clearer picture of current expert opinion by looking at the results of a recent survey which asked the top 100 most-cited living AI scientists when they thought AGI would arrive, how soon after AGI we’d get superintelligence, and what the likely social impact of superintelligence would be. ¹

But at the moment, I just want to mention one additional computer scientist who seems to have been concerned about AI risk for a long time: Ed Fredkin. ²

In Pamela McCorduck’s history of the first few decades of AI, Machines Who Think (1979), Fredkin is quoted extensively on AI risk. Fredkin said (ch. 14):

Eventually, no matter what we do there’ll be artificial intelligences with independent goals. In pretty much convinced of that. There may be a way to postpone it. There may even be a way to avoid it, I don’t know. But its very hard to have a machine that’s a million times smarter than you as your slave.

…And pulling the plug is no way out. A machine that smart could act in ways that would guarantee that the plug doesn’t get pulled under any circumstances, regardless of its real motives — if it has any.

…I can’t persuade anyone else in the field to worry this way… They get annoyed when I mention these things. They have lots of attitudes, of course, but one of them is, “Well yes, you’re right, but it would be a great disservice to the world to mention all this.”…my colleagues only tell me to wait, not to make my pitch until it’s more obvious that we’ll have artificial intelligences. I think by then it’ll be too late. Once artificial intelligences start getting smart, they’re going to be very smart very fast. What’s taken humans and their society tens of thousands of years is going to be a matter of hours with artificial intelligences. If that happens at Stanford, say, the Stanford AI lab may have immense power all of a sudden. It’s not that the United States might take over the world, it’s that Stanford AI Lab might.

…And so what I’m trying to do is take steps to see that… an international laboratory gets formed, and that these ideas get into the minds of enough people. McCarthy, for lots of reasons, resists this idea, because he thinks the Russians would be untrustworthy in such an enterprise, that they’d swallow as much of the technology as they could, contribute nothing, and meanwhile set up a shadow place of their own running at the exact limit of technology that they could get from the joint effort. And as soon as that made some progress, keep it secret from the rest of us so they could pull ahead… Yes, he might be right, but it doesn’t matter. The international laboratory is by far the best plan; I’ve heard of no better plan. I still would like to see it happen: lets be active instead of passive…

…There are three events of equal importance, if you like. Event one is the creation of the universe. It’s a fairly important event. Event two is the appearance of life. Life is a kind of organizing principle which one might argue against if one didn’t understand enough — shouldn’t or couldn’t happen on thermodynamic grounds, or some such. And, third, there’s the appearance of artificial intelligence. It’s the question which deals with all questions… If there are any questions to be answered, this is how they’ll be answered. There can’t be anything of more consequence to happen on this planet.

Fredkin, now 80, continues to think about AI risk — about the relevance of certification to advanced AI systems, about the race between AI safety knowledge and AI capabilities knowledge, etc. ³ I’d be very curious to learn what Fredkin thinks of the arguments in Superintelligence.

Footnotes:

1. The short story is:
   1. The median estimate was that there was a 50% chance of AGI by 2050, and a 90% chance of AGI by 2070.
   2. The median estimate on AGI-to-superintelligence timing was that there was a 10% chance of superintelligence within 2 years of AGI, and a 75% chance of superintelligence within 30 years of AGI.
   3. When asked whether the social impact of superintelligence would be “extremely bad” or “extremely good” or somewhere in-between, the experts tended to think good outcomes were more likely than bad outcomes, but not super-confidently. (See section 3.4 of the paper.)

   [↩]

2. This isn’t to say Fredkin had, in 1979, anything like the Bostrom-Yudkowsky view on AI risk. For example he seems to have thought that most of the risk is during a transition period, and that once machines are superintelligent they will be able to discern our true motives. The Bostrom-Yudkowsky school would reply that “the genie knows but doesn’t care” (also see Superintelligence, p. 121).[↩]
3. I learned this via Yudkowsky, who had some communication with Fredkin in 2013.[↩]