Check the original source

From Segerstrale (2000), p. 27:

In 1984 I was able to shock my class of well-intended liberal students at Smith College by giving them the assignment to compare [Stephan] Chorover’s [critical] representation of passages of [E.O. Wilson’s] Sociobiology with Wilson’s original text. The students, who were deeply suspicious of Wilson and spontaneous champions of his critics, embarked on this homework with gusto. Many students were quite dismayed at their own findings and angry with Chorover. This surely says something, too, about these educated laymen’s relative innocence regarding what can and cannot be done in academia.

I wish this kind of exercise was more common. Another I would suggest is to compare critics’ representations of Dreyfus’ “Alchemy and Artificial Intelligence” with the original text (see here).

The first AI textbook, on the control problem

The earliest introductory AI textbook I know about — excluding mere “paper collections” like Computers and Thought (1963) — is Jackson’s Introduction to Artificial Intelligence (1974).

It discusses AGI and the control problem starting on page 394:

If [AI] research is unsuccessful at producing a general artificial intelligence, over a period of more than a hundred years, then its failure may raise some serious doubt among many scientists as to the finite describability of man and his universe. However, the evidence presented in this book makes it seem likely that artificial intelligence research will be successful, that a technology will be developed which is capable of producing machines that can demonstrate most, if not all, of the mental abilities of human beings. Let us therefore assume that this will happen, and imagine two worlds that might result.

[First,] …It is not difficult to envision actualities in which an artificial intelligence would exert control over human beings, yet be out of their control.

Given that intelligent machines are to be used, the question of their control and noncontrol must be answered. If a machine is programmed to seek certain ends, how are we to insure that the means it chooses to employ are agreeable to people? A preliminary solution to the problem is given by the fact that we can specify state-space problems to require that their solution paths shall not pass through certain states (see Chapter 3). However, the task of giving machines more sophisticated value systems, and especially of making them ‘ethical,’ has not yet been investigated by AI researchers…

The question of control should be coupled with the ‘lack of understanding’ question; that is, the possibility exists that intelligent machines might be too complicated for us to understand in situations that require real-time analyses (see the discussion of evolutionary programs in Chapter 8). We could conceivably always demand that a machine give a complete output of its reasoning on a problem; nevertheless that reasoning might not be effectively understandable to us if the problem itself were to determine a time limit for producing a solution. In such a case, if we were to act rationally, we might have to follow the machine’s advice without understanding its ‘motives’…

It has been suggested that an intelligent machine might arise accidentally, without our knowledge, through some fortuitous interconnection of smaller machines (see Heinlein, 1966). If the smaller machines each helped to control some aspect of our economy or defense, the accidental intelligent might well act as a dictator… It seems highly unlikely that this will happen, especially if we devote sufficient time to studying the non-accidental systems we implement.

A more significant danger is that artificial intelligence might be used to further the interests of human dictators. A limited supply of intelligent machines in the hands of a human dictator might greatly increase his power over other human beings, perhaps to the extent of giving him complete censorship and supervision of the public…

Let us now paint another, more positive picture of the world that might result from artificial intelligence research… It is a world in which man and his machines have reached a state of symbiosis…

The benefits humanity might gain from achieving such a symbiosis are enormous. As mentioned [earlier], it may be possible for artificial intelligence to greatly reduce the amount of human labor necessary to operate the economy of the world… Computers and AI research may play an important part in helping to overcome the food, population, housing, and other crises that currently grip the earth… Artificial intelligence may eventually be used to… partially automated the development of science itself… Perhaps artificial intelligence will someday be used in automatic teachers… and perhaps mechanical translators will someday be developed which will fluently translate human languages. And (very perhaps) the day may eventually come when the ‘household robot’ and the ‘robot chauffeur’ will be a reality…

In some ways it is reassuring that the progress in artificial intelligence research is proceeding at a relatively slow but regular pace. It should be at least a decade before any of these possibilities becomes an actuality, which will give us some time to consider in more detail the issues involved.

“Beyond the scope of this paper”

From AI scientist Drew McDermott, in 1976:

In this paper I have criticized AI researchers very harshly. Let me express my faith that people in other fields would, on inspection, be found to suffer from equally bad faults. Most AI workers are responsible people who are aware of the pitfalls of a difficult field and produce good work in spite of them. However, to say anything good about anyone is beyond the scope of this paper.

Books, music, etc. from April 2016



Music I most enjoyed discovering this month:


Ones I really liked, or loved:

Media I’m looking forward to, May 2016 edition


* = added this round


(only including movies which AFAIK have at least started principal photography)

  • Russo & Russo, Captain America: Civil War (May 2016)
  • Stanton, Finding Dory (Jun 2016)
  • Liu, Batman: The Killing Joke (Jul 2016)
  • Cianfrance, The Light Between Oceans (Sep 2016)
  • Lonergan, Manchester by the Sea (Nov 2016)
  • Nichols, Loving (Nov 2016)
  • Scorcese, Silence (Nov 2016)
  • Edwards, Rogue One (Dec 2016)
  • Villeneuve, Story of Your Life (TBD 2016)
  • Dardenne brothers, The Unknown Girl (TBD 2016)
  • Swanberg, Win It All (TBD 2016)
  • Farhadi, The Salesman (TBD 2016)
  • Reeves, War for the Planet of the Apes (Jul 2017)
  • Unkrich, Coco (Nov 2017)
  • Johnson, Star Wars: Episode VIII (Dec 2017)
  • Payne, Downsizing (Dec 2017)

Scaruffi on art music

From the preface to his in-progress history of avant-garde music:

Art Music (or Sound Art) differs from Commercial Music the way a Monet painting differs from IKEA furniture. Although the border is frequently fuzzy, there are obvious differences in the lifestyles and careers of the practitioners. Given that Art Music represents (at best) 3% of all music revenues, the question is why anyone would want to be an art musician at all. It is like asking why anyone would want to be a scientist instead of joining a technology startup. There are pros that are not obvious if one only looks at the macroscopic numbers. To start with, not many commercial musicians benefit from that potentially very lucrative market. In fact, the vast majority live a rather miserable existence. Secondly, commercial music frequently implies a lifestyle of time-consuming gigs in unattractive establishments. But fundamentally being an art musician is a different kind of job, more similar to the job of the scientific laboratory researcher (and of the old-fashioned inventor) than to the job of the popular entertainer. The art musician is pursuing a research program that will be appreciated mainly by his peers and by the “critics” (who function as historians of music), not by the public. The art musician is not a product to be sold in supermarkets but an auteur. The goal of an art musician is, first and foremost, to do what s/he feels is important and, secondly, to secure a place in the history of human civilization. Commercial musicians live to earn a good life. Art musicians live to earn immortality. (Ironically, now that we entered the age of the mass market, a pop star may be more likely to earn immortality than the next Beethoven, but that’s another story). Art music knows no stylistic boundaries: the division in classical, jazz, rock, hip hop and so forth still makes sense for commercial music (it basically identifies the sales channel) but ever less sense for art music whose production, distribution and appreciation methods are roughly the same regardless of whether the musician studied in a Conservatory, practiced in a loft or recorded at home using a laptop.

Medical ghostwriting

From Mushak & Elliott (2015):

Pharmaceutical companies hire “medical education and communication companies” (MECCs) to create sets of journal articles (and even new journals) designed to place their drugs in a favorable light and to assist in their marketing efforts (Sismondo 2007, 2009; Elliott 2010). These articles are frequently submitted to journals under the names of prominent academic researchers, but the articles are actually written by employees of the MECCs (Sismondo 2007, 2009). While it is obviously difficult to determine what proportion of the medical literature is produced in this fashion, one study used information uncovered in litigation to determine that more than half of the articles published on the antidepressant Zoloft between 1998 and 2000 were ghostwritten (Healy and Cattell 2003). These articles were published in more prestigious journals than the non-ghostwritten articles and were cited five times more often. Significantly, they also painted a rosier picture of Zoloft than the others.

Books, music, etc. from March 2016



Music I most enjoyed discovering this month:


Ones I really liked, or loved:

  • [none this month]

Some books I’m looking forward to, April 2016 edition

* = added this round

CGP Grey on Superintelligence

CGP Grey recommends Nick Bostrom’s Superintelligence:

The reason this book [Superintelligence]… has stuck with me is because I have found my mind changed on this topic, somewhat against my will.

…For almost all of my life… I would’ve placed myself very strongly in the camp of techno-optimists. More technology, faster… it’s nothing but sunshine and rainbows ahead… When people would talk about the “rise of the machines”… I was always very dismissive of this, in no small part because those movies are ridiculous… [and] I was never convinced there was any kind of problem here.

But [Superintelligence] changed my mind so that I am now much more in the camp of [thinking that the development of general-purpose AI] can seriously present an existential threat to humanity, in the same way that an asteroid collision… is what you’d classify as a serious existential threat to humanity — like, it’s just over for people.

…I keep thinking about this because I’m uncomfortable with having this opinion. Like, sometimes your mind changes and you don’t want it to change, and I feel like “Boy, I liked it much better when I just thought that the future was always going to be great and there’s not any kind of problem”…

…The thing about this book that I found really convincing is that it used no metaphors at all. It was one of these books which laid out its basic assumptions, and then just follows them through to a conclusion… The book is just very thorough at trying to go down every path and every combination of [assumptions], and what I realized was… “Oh, I just never did sit down and think through this position [that it will eventually be possible to build general-purpose AI] to its logical conclusion.”

Another interesting section begins at 1:46:35 and runs through about 1:52:00.

Tetlock wants suggestions for strong AI signposts

In my 2013 article on strong AI forecasting, I made several suggestions for how to do better at forecasting strong AI, including this suggestion quoted from Phil Tetlock, arguably the leading forecasting researcher in the world:

Signposting the future: Thinking through specific scenarios can be useful if those scenarios “come with clear diagnostic signposts that policymakers can use to gauge whether they are moving toward or away from one scenario or another… Falsifiable hypotheses bring high-flying scenario abstractions back to Earth.”

Tetlock hadn’t mentioned strong AI at the time, but now it turns out he wants suggestions for strong AI signposts that could be forecast on GJOpen, the forecasting tournament platform.

Specifying crisply formulated signpost questions is not easy. If you come up with some candidates, consider posting them in the comments below. After a while, I will collect them all together and send them to Tetlock. (I figure that’s probably better than a bunch of different people sending Tetlock individual emails with overlapping suggestions.)

Tetlock’s framework for thinking about such signposts, which he calls “Bayesian question clustering,” is described in Superforecasting:

In the spring of 2013 I met with Paul Saffo, a Silicon Valley futurist and scenario consultant. Another unnerving crisis was brewing on the Korean peninsula, so when I sketched the forecasting tournament for Saffo, I mentioned a question IARPA had asked: Will North Korea “attempt to launch a multistage rocket between 7 January 2013 and 1 September 2013?” Saffo thought it was trivial. A few colonels in the Pentagon might be interested, he said, but it’s not the question most people would ask. “The more fundamental question is ‘How does this all turn out?’ ” he said. “That’s a much more challenging question.”

So we confront a dilemma. What matters is the big question, but the big question can’t be scored. The little question doesn’t matter but it can be scored, so the IARPA tournament went with it. You could say we were so hell-bent on looking scientific that we counted what doesn’t count.

That is unfair. The questions in the tournament had been screened by experts to be both difficult and relevant to active problems on the desks of intelligence analysts. But it is fair to say these questions are more narrowly focused than the big questions we would all love to answer, like “How does this all turn out?” Do we really have to choose between posing big and important questions that can’t be scored or small and less important questions that can be? That’s unsatisfying. But there is a way out of the box.

Implicit within Paul Saffo’s “How does this all turn out?” question were the recent events that had worsened the conflict on the Korean peninsula. North Korea launched a rocket, in violation of a UN Security Council resolution. It conducted a new nuclear test. It renounced the 1953 armistice with South Korea. It launched a cyber attack on South Korea, severed the hotline between the two governments, and threatened a nuclear attack on the United States. Seen that way, it’s obvious that the big question is composed of many small questions. One is “Will North Korea test a rocket?” If it does, it will escalate the conflict a little. If it doesn’t, it could cool things down a little. That one tiny question doesn’t nail down the big question, but it does contribute a little insight. And if we ask many tiny-but-pertinent questions, we can close in on an answer for the big question. Will North Korea conduct another nuclear test? Will it rebuff diplomatic talks on its nuclear program? Will it fire artillery at South Korea? Will a North Korean ship fire on a South Korean ship? The answers are cumulative. The more yeses, the likelier the answer to the big question is “This is going to end badly.”

I call this Bayesian question clustering because of its family resemblance to the Bayesian updating discussed in chapter 7. Another way to think of it is to imagine a painter using the technique called pointillism. It consists of dabbing tiny dots on the canvas, nothing more. Each dot alone adds little. But as the dots collect, patterns emerge. With enough dots, an artist can produce anything from a vivid portrait to a sweeping landscape.

There were question clusters in the IARPA tournament, but they arose more as a consequence of events than a diagnostic strategy. In future research, I want to develop the concept and see how effectively we can answer unscorable “big questions” with clusters of little ones.

(Note that although I work as a GiveWell research analyst, my focus at GiveWell is not AI risks, and my views on this topic are not necessarily GiveWell’s views.)

The silly history of spinach

From Arbesman’s The Half-Life of Facts:

One of the strangest examples of the spread of error is related to an article in the British Medical Journal from 1981. In it, the immunohematologist Terry Hamblin discusses incorrect medical information, including a wonderful story about spinach. He details how, due to a typo, the amount of iron in spinach was thought to be ten times higher than it actually is. While there are only 3.5 milligrams of iron in a 100-gram serving of spinach, the accepted fact became that spinach contained 35 milligrams of iron. Hamblin argues that German scientists debunked this in the 1930s, but the misinformation continued to spread far and wide.

According to Hamblin, the spread of this mistake even led to spinach becoming Popeye the Sailor’s food choice. When Popeye was created, it was recommended he eat spinach for his strength, due to its vaunted iron-based health properties.

This wonderful case of a typo that led to so much incorrect thinking was taken up for decades as a delightful, and somewhat paradigmatic, example of how wrong information could spread widely. The trouble is, the story itself isn’t correct.

While the amount of iron in spinach did seem to be incorrectly reported in the nineteenth century, it was likely due to a confusion between iron oxide—a related chemical—and iron, or contamination in the experiments, rather than a typographical error. The error was corrected relatively rapidly, over the course of years, rather than over many decades.

Mike Sutton, a reader in criminology at Nottingham Trent University, debunked the entire original story several years ago through a careful examination of the literature. He even discovered that Popeye seems to have eaten spinach not for its supposed high quantities of iron, but rather due to vitamin A. While the truth behind the myth is still being excavated, this misinformation — the myth of the error — from over thirty years ago continues to spread.

Time to proof for well-specified problems

How much time usually elapses between when a technical problem is posed and when it is solved? How much effort is usually required? Which variables most predict how much time and effort will be required to solve a technical problem?

The main paper I’ve seen on this is Hisano & Sornette (2013). Their method was to start with Wikipedia’s List of conjectures and then track down the year each conjecture was first stated and the year it was solved (or, whether it remains unsolved). They were unable to determine exact-year values for 16 conjectures, leaving them with a dataset of 144 conjectures, of which 60 were solved as of January 2012, with 84 still unsolved. The time between first conjecture statement and first solution is called “time to proof.”

For the purposes of finding possible data-generating models that fit the data described above, they assume the average productivity per mathematician is constant throughout their career (they didn’t try to collect more specific data), and they assume the number of active mathematicians tracks with total human population — i.e., roughly exponential growth over the time period covered by these conjectures and proofs (because again, they didn’t try to collect more specific data).

I didn’t try to understand in detail how their model works or how reasonable it is, but as far as I understand it, here’s what they found:

  • Since 1850, the number of new conjectures (that ended up being listed on Wikipedia) has tripled every 55 years. This is close to the average growth rate of total human population over the same time period.
  • Given the incompleteness of the data and the (assumed) approximate exponential growth of the mathematician population, they can’t say anything confident about the data-generating model, and therefore basically fall back on Occam: “we could not reject the simplest model of an exponential rate of conjecture proof with a rate of 0.01/year for the dataset (translating into an average waiting time to proof of 100 years).”
  • They expect the Wikipedia dataset severely undersamples “the many conjectures whose time-to-proof is in the range of years to a few decades.”
  • They use their model to answer the question that prompted the paper, which was about the probability that “P vs. NP” will be solved by 2024. Their model says there’s a 41.3% chance of that, which intuitively seems high to me.
  • They make some obvious caveats to all this: (1) the content of the conjecture matters for how many mathematician-hours are devoted to solving it, and how quickly they are devoted; (2) to at least a small degree, the notion of “proof” has shifted over time, e.g. the first proof of the four-color theorem still has not been checked from start to finish by humans, and is mostly just assumed to be correct; (3) some famous conjectures might be undecidable, leaving some probability mass for time-to-proof at infinity.

What can we conclude from this?

Not much. Sometimes crisply-posed technical problems are solved quickly, sometimes they take many years or decades to solve, sometimes they take more than a century to solve, and sometimes they are never solved, even with substantial effort being targeted at the problem.

And unfortunately, it looks like we can’t say much more than that from this study alone. As they say, their observed distribution of time to proof must be considered with major caveats. Personally, I would emphasize the likely severe undersampling of conjectures with short times-to-proof, the fact that they didn’t try to weight data points by how important the conjectures were perceived to be or how many resources went into solving them (because doing so would be very hard!), and the fact that they didn’t have enough data points (especially given the non-stationary number of mathematicians) to confirm or reject ~any of the intuitively / a priori plausible data-generating models.

Are there other good articles on “time to proof” or “time to solution” for relatively well-specified research problems, in mathematics or other fields? If you know of any, please let me know!

Geoff Hinton on long-term AI outcomes

Geoff Hinton on a show called The Agenda (starting around 9:40):

Interviewer: How many years away do you think we are from a neural network being able to do anything that a brain can do?

Hinton: …I don’t think it will happen in the next five years but beyond that it’s all a kind of fog.

Interviewer: Is there anything about this that makes you nervous?

Hinton: In the very long run, yes. I mean obviously having… [AIs] more intelligent than us is something to be nervous about. It’s not gonna happen for a long time but it is something to be nervous about.

Interviewer: What aspect of it makes you nervous?

Hinton: Will they be nice to us?

Some books I’m looking forward to, March 2016 edition

* = added this round

Books, music, etc. from February 2016



Music I most enjoyed discovering this month:


Ones I really liked, or loved:

If you want to write about intelligence explosion…

Toby Walsh has published a short new paper on the likelihood of intelligence explosion. Unfortunately, it doesn’t engage with three of the most detailed and thoughtful previous analyses on the topic.

If you want to write about the likelihood and nature of intelligence explosion, I consider the following sources required reading, in descending order of value per page (Walsh’s paper misses 2, 3, and 5):

  1. Bostrom (2014), chapter 4
  2. Yudkowsky (2013)
  3. AI Impacts‘ posts on intelligence explosion: one, two (both 2015)
  4. Chalmers (2010)
  5. Hanson & Yudkowsky (2013)

There are many other sources worth reading, e.g. Hutter (2012), but they don’t make my cut as “required reading.”

(Note that although I work as a GiveWell research analyst, my focus at GiveWell is not AI risks, and my views on this topic are not necessarily GiveWell’s views.)

Bill Gates on AI timelines

On the latest episode of The Ezra Klein Show, Bill Gates elaborated a bit on his views about AI timelines (starting around 24:40):

Klein: I know you take… the risk of creating artificial intelligence that… ends up turning against us pretty seriously. I’m curious where you think we are in terms of creating an artificial intelligence…

Gates: Well, with robotics you have to think of three different milestones.

One is… not-highly-trained labor substitution. Driving, security guard, warehouse work, waiter, maid — things that are largely visual and physical manipulation… [for] that threshold I don’t think you’d get much disagreement that over the next 15 years that the robotic equivalents in terms of cost [and] reliability will become a substitute to those activities…

Then there’s the point at which what we think of as intelligent activities, like writing contracts or doing diagnosis or writing software code, when will the computer start to… have the capacity to work in those areas? There you’d get more disagreement… some would say 30 years, I’d be there. Some would say 60 years. Some might not even see that [happening].

Then there’s a third threshold where the intelligence involved is dramatically better than humanity as a whole, what Bostrom called a “superintelligence.” There you’re gonna get a huge range of views including people who say it won’t ever happen. Ray Kurzweil says it will happen at midnight on July 13, 2045 or something like that and that it’ll all be good. Then you have other people who say it can never happen. Then… there’s a group that I’m more among where you say… we’re not able to predict it, but it’s something that should start thinking about. We shouldn’t restrict activities or slow things down… [but] the potential that that exists even in a 50-year timeframe [means] it’s something to be taken seriously.

But those are different thresholds, and the responses are different.

See Gates’ previous comments on AI timelines and AI risk, here.

UPDATE 07/01/2016: In this video, Gates says that achieving “human-level” AI will take “at least 5 times as long as what Ray Kurzweil says.”

Reply to LeCun on AI safety

On Facebook, AI scientist Yann LeCun recently posted the following:

I have said publicly on several occasions that the purported AI Apocalypse that some people seem to be worried about is extremely unlikely to happen, and if there were any risk of it happening, it wouldn’t be for another few decades in the future. Making robots that “take over the world”, Terminator style, even if we had the technology. would require a conjunction of many stupid engineering mistakes and ridiculously bad design, combined with zero regards for safety. Sort of like building a car, not just without safety belts, but also a 1000 HP engine that you can’t turn off and no brakes.

But since some people seem to be worried about it, here is an idea to reassure them: We are, even today, pretty good at building machines that have super-human intelligence for very narrow domains. You can buy a $30 toy that will beat you at chess. We have systems that can recognize obscure species of plants or breeds of dogs, systems that can answer Joepardy questions and play Go better than most humans, we can build systems that can recognize a face among millions, and your car will soon drive itself better than you can drive it. What we don’t know how to build is an artificial general intelligence (AGI). To take over the world, you would need an AGI that was specifically designed to be malevolent and unstoppable. In the unlikely event that someone builds such a malevolent AGI, what we merely need to do is build a “Narrow” AI (a specialized AI) whose only expertise and purpose is to destroy the nasty AGI. It will be much better at this than the AGI will be at defending itself against it, assuming they both have access to the same computational resources. The narrow AI will devote all its power to this one goal, while the evil AGI will have to spend some of its resources on taking over the world, or whatever it is that evil AGIs are supposed to do. Checkmate.

Since LeCun has stated his skepticism about potential risks from advanced artificial intelligence in the past, I assume his “not being really serious” is meant to refer to his proposed narrow AI vs. AGI “solution,” not to his comments about risks from AGI. So, I’ll reply to his comments on risks from AGI and ignore his “not being really serious” comments about narrow AI vs. AGI.

First, LeCun says:

if there were any risk of [an “AI apocalypse”], it wouldn’t be for another few decades in the future

Yes, that’s probably right, and that’s what people like myself (former Executive Director of MIRI) and Nick Bostrom (author of Superintelligence, director of FHI) have been saying all along, as I explained here. But LeCun phrases this as though he’s disagreeing with someone.

Second, LeCun writes as though the thing people are concerned about is a malevolent AGI, even though I don’t know anyone is concerned about malevolent AI. The concern expressed in Superintelligence and elsewhere isn’t about AI malevolence, it’s about convergent instrumental goals that are incidentally harmful to human society. Or as AI scientist Stuart Russell put it:

A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable.  This is essentially the old story of the genie in the lamp, or the sorcerer’s apprentice, or King Midas: you get exactly what you ask for, not what you want. A highly capable decision maker – especially one connected through the Internet to all the world’s information and billions of screens and most of our infrastructure – can have an irreversible impact on humanity.

(Note that although I work as a GiveWell research analyst, my focus at GiveWell is not AI risks, and my views on this topic are not necessarily GiveWell’s views.)