Hillary Clinton on AI risk

From What Happened, p. 241:

Technologists like Elon Musk, Sam Altman, and Bill Gates, and physicists like Stephen Hawking have warned that artificial intelligence could one day pose an existential security threat. Musk has called it “the greatest risk we face as a civilization.” Think about it: Have you ever seen a movie where the machines start thinking for themselves that ends well? Every time I went out to Silicon Valley during the campaign, I came home more alarmed about this. My staff lived in fear that I’d start talking about “the rise of the robots” in some Iowa town hall. Maybe I should have. In any case, policy makers need to keep up with technology as it races ahead, instead of always playing catch-up.

Monkey classification errors

More Wynne & Udell (2013):

Michael D’Amato and Paul van Sant (1988) trained Cebus apella monkeys to discriminate slides containing people from those that did not. The monkeys readily learned to do this. Then the monkeys were presented with novel slides they had never seen before which contained either scenes with people or similar scenes with no people in them. Here also the monkeys spontaneously classified the majority of slides correctly. So far, so good – clear evidence that the monkeys had not just learned the particular slides they had been trained on but had abstracted a person concept from those slides that they then successfully applied to pictures they had never seen before.

Or had they? D’Amato and van Sant did not stop their analysis simply with the observation that the monkeys had successfully transferred their learning to novel slides – rather they went on to look carefully at the kinds of errors the monkeys had made. Although largely successful with the novel slides, the monkeys made some very puzzling mistakes. For example, one of the person slides that the monkeys had failed to recognize as a picture of a human being had been a head and shoulders portrait – which, to another human, is a classic image of a person. One of the slides that the monkeys had incorrectly classified as containing a human had actually been a shot of a jackal carrying a dead flamingo in its mouth; both the jackal and its prey were also reflected in the water beneath them. What person in her right mind could possible confuse a jackal with a flamingo in its mouth with another human being?

The explanation for both these mistakes is the same: the monkeys had generalized on the basis of the particular features contained in the slides they had been trained with rather than learning the more abstract concept that the experimenters had intended. The head and shoulders portrait of a person lacked the head-torso-arms-legs body shape that had been most common among the images that the monkeys had been trained with, and consequently, they had rejected it as not similar enough to the positive image they were looking for. Similarly, during training, the only slides that had contained flashes of red happened to be those of people. Three of the training slides had contained people wearing a piece of red clothing, whereas none of the nonperson slides had contained the color red. Consequently, when the jackal with prey slide came along during testing, it contained the color red, and so the monkeys classified it as a person slide.

Adversarial examples for pigeons

From Wynne & Udell (2013):

Michael Young and colleagues carried out experiments that add to a sense that the pigeon’s perception of pictures of objects is not identical to our own. They trained pigeons to peck in different locations on a computer-controlled touch screen, depending on which of four different objects was presented: an arch, a barrel, a brick, and a triangular wedge (Young et al., 2001). The objects were initially presented to the pigeons as images shaded to suggest light shining on them from one direction. Next, Young and colleagues tested the pigeons with pictures of the same objects, but this time illuminated from a different direction… To the experimenters’ surprise, the pigeons’ ability to recognize the objects was disturbed by changes in lighting that human observers were barely able to perceive… [see below]

pigeons study

How long does it take to identify, mitigate, and remediate a major problem?

Baiocchi & Welser (2010):

…we conducted a literature survey on each of the [problems comparable to the problem of space debris]. We then determined the length of time spent in each stage (problem identification, establishment of normative behaviors, mitigation, and remediation) based on research from periodical sources, legislative records, and court rulings… Finally, we inspected each timeline and made a judgment about the approximate year in which each problem entered a new stage… The result is shown in Figure 6.1, and it provides a notional comparison that shows how each of the problems progressed through the four stages.

Figure 6.1

Could be interesting to see this kind of analysis for a greater range of societal challenges, or sets of challenges chosen for how similar they are to a different target case. (The target case for this report was space debris.)

A utilitarian foundation?

The introduction to Jacobson (1984) makes it sound as though the John A. Hartford Foundation was roughly cause-neutral and utilitarian in its approach, at least for some of its history:

The 1958 annual report of the Hartford Foundation describes its starting point:

Neither John Hartford nor his brother George, in their bequests to the organization, expressed any wish as to how the funds they provided should be used… Our benefactors’ one common request was that the Foundation strive always to do the greatest good for the greatest number.

…If available funds are to be used effectively, it is necessary to carve from the whole vast spectrum of human needs one small band that the heart and mind together tell you is the area in which you can make your best contribution.

The first task of the Foundation was thus to define the greatest good. Basing its decision on the pattern of John Hartford’s previous giving, the Foundation chose to support biomedical, largely clinical, research. Between 1954 and 1979, the Hartford Foundation participated in some of the most important advances in modern medicine, supplied hospitals and medical centers with equipment that reflected those advances, provided for the training of a generation of researchers, saved countless lives, and involved itself deeply in the burgeoning of the current health care crisis. In that period, the Foundation spent close to $175 million [presumably this is 1984 dollars, i.e. $408 million in 2016 dollars].

…Many modern research-supporting institutions have chosen to bear the costs of close supervision and peer review in order to ensure the quality of projects supported either directly or indirectly by the public. But both the trustees and the staff of the Hartford Foundation came from a background that stressed minimizing administrative costs so as to maximize benefits to the public. During the Foundation’s first seven years as a leading source of funds for biomedical research, the full-time staff consisted of one person. To achieve quality control at low cost, the Foundation adopted a policy of hiring consultants as they were needed to review particular grant applications.

As a matter of policy, too, the Foundation tried to fund projects and types of research that could not obtain funding from other sources. For example, the Hartford Foundation was the first to pay for the patient-bed costs of clinical research. Filling this gap was clearly desirable. But the Foundation also supported some researchers whose theories or personalities inspired skepticism in their colleagues. These grants were calculated risks. Many of the projects thus supported were unsuccessful; a few have produced major advances in clinical medicine.

When these successes occurred, the Hartford Foundation could have chosen to publicize its role in them. But John and George Hartford disliked publicity. The trustees and staff made this family trait a matter of policy. They believed that being in the public eye was tasteless, a waste of time, and likely to produce an excess of grant requests unmanageable by a small staff. As a result, the pool of grant applicants was limited largely to those who heard about the Foundation by word of mouth — from past grantees or consultants.

Probably the truth is more complicated; I haven’t investigated the foundation’s history closely. Note also that the foundation seems to have cared a lot about the overhead ratio, whereas today’s effective altruists tend to think overhead ratio considerations should be subordinate to impact per dollar.

Have any of my readers heard of any other charitable foundations aspiring to be (roughly) cause-neutral and utilitarian in their approach?

Bill Koch, romancer

Pretty sure my friends’ nerdy-romantic messages are cleverer than Bill Koch’s:

[Bill Koch’s lover] referred to herself in a separate fax as a “wet orchid” who yearned for warm honey to be drizzled on her body. In another, she wrote: “My poor nerve endings are already hungry. You are creating such a wanton woman. I can feel those kisses, and every inch of my body misses you.”

Bill’s far-less-sensuous facsimiles displayed the MIT-trained engineer’s geeky side: “I cannot describe how much I look forward to seeing you again,” he wrote. “It is beyond calculation by the largest computers.” In another fax, he jotted an equation to express his devotion, ending with a hand-drawn heart and, within it, the mathematical symbol for infinity.

Friedman on economics chairs

Funny comment in a 1990 letter penned by Milton Friedman, quoted in Blundell (2007), p. 47:

I have personally been impressed by the extent to which the growing acceptability of free private-market ideas has produced a lowering of the average intellectual quality of those who espouse those ideas. This is inevitable, but I believe it has been fostered by… the creation of free-enterprise chairs of economics. I believe that they are counterproductive.

Scott Aaronson on order and chaos


One of my first ideas was to write about the Second Law of Thermodynamics [in response to Edge.org’s Annual Question], and to muse about how one of humanity’s tragic flaws is to take for granted the gargantuan effort needed to create and maintain even little temporary pockets of order. Again and again, people imagine that, if their local pocket of order isn’t working how they want, then they should smash it to pieces, since while admittedly that might make things even worse, there’s also at least 50/50 odds that they’ll magically improve. In reasoning thus, people fail to appreciate just how exponentially more numerous are the paths downhill, into barbarism and chaos, than are the few paths further up. So thrashing about randomly, with no knowledge or understanding, is statistically certain to make things worse: on this point thermodynamics, common sense, and human history are all in total agreement. The implications of these musings for the present would be left as exercises for the reader.

Or, in cartoon form:


So apparently this is why we have positive psychology but not evidence-based psychological treatment

Here’s Marty Seligman, past president of the American Psychological Association (APA):

APA presidents are supposed to have an initiative and… I thought mine could be “evidence-based treatment and prevention.” So I went to my friend, Steve Hyman, the director of [National Institute of Mental Health]. He was thrilled and told me he would chip in $40 million dollars if I could get APA working on evidence-based treatment.

So I told CAPP [which owns the APA] about my plan and about NIMH’s willingness. I felt the room get chillier and chillier. I rattled on. Finally, the chair of CAPP memorably said, “What if the evidence doesn’t come out in our favor?”

…I limped my way to [my friend’s] office for some fatherly advice.

“Marty,” he opined, “you are trying to be a transactional president. But you cannot out-transact these people…”

And so I proposed that Psychology turn its… attention away from pathology and victimology and more toward what makes life worth living: positive emotion, positive character, and positive institutions. I never looked back and this became my mission for the next fifteen years. The endeavor… caught on.

My post title is sort-of joking. Others have pushed on evidence-based psychology while Seligman focused on positive psychology, and Seligman certainly wouldn’t say that we “don’t have” evidence-based psychological treatment. But I do maintain that evidence-based psychology is not yet as well-developed as evidence-based medicine, even given EBM’s many problems.

Karpathy on nukes

OpenAI deep learning researcher Andrej Karpathy on Rhodes’ Making of the Atomic Bomb:

Unfortunately, we live in a universe where the laws of physics feature a strong asymmetry in how difficult it is to create and to destroy. This observation is also not reserved to nuclear weapons – more generally, technology monotonically increases the possible destructive damage per person per dollar. This is my favorite resolution to the Fermi paradox.

As I am a scientist myself, I was particularly curious about the extent to which the nuclear scientists who conceived and designed the bomb influenced the ethical/political discussions. Unfortunately, it is clearly the case that the scientists were quickly marginalized and, in effect, told to shut up and just help build the bomb. From the very start, Roosevelt explicitly wanted policy considerations restricted to a small group that excluded any scientists. As some of the more prominent examples of scientists trying to influence policy, Bohr advocated for establishing an “Open World Consortium” and sharing information about the bomb with the Soviet Union, but this idea was promptly shut down by Churchill. In this case it’s not clear what effect it would have had and, in any case, the Soviets already knew a lot through espionage. Bohr also held the seemingly naive notion that scientists should continue publishing all nuclear research during the second world war as he felt that science should be completely open and rise above national disputes. Szilard strongly opposed this openness internationally, but advocated for more openness within the Manhattan project for sake of efficiency. This outraged Groves who was obsessed with secrecy. In fact, Szilard was almost arrested, suspected to be a spy, and placed under a comical surveillance that mostly uncovered his frequent visits to a chocolate store.

Henry Kissinger on smarter-than-human AI

Henry Kissinger, speaking with The Economist:

It is undoubtedly the case that modern technology poses challenges to world order and world order stability that are absolutely unprecedented. Climate change is one of them. I personally believe that artificial intelligence is a crucial one, lest we wind up… creating instruments in relation to which we are like the Incas to the Spanish, [such that] our own creations have a better capacity to calculate than we do. It’s a problem we need to understand on a global basis.

For reference, here is Wikipedia on the Spanish conquest of the Inca empire.

Henry Kissinger also addressed artificial intelligence in a recent interview with The Atlantic, though in this case he probably was not referring to smarter-than-human AI:

A military conflict between [China and the USA], given the technologies they possess, would be calamitous. Such a conflict would force the world to divide itself. And it would end in destruction, but not necessarily in victory, which would likely prove too difficult to define. Even if we could define victory, what in the wake of utter destruction could the victor demand of the loser? I am speaking of not merely the force of our weapons, but the unknowability of the consequences of some of them, such as cyberweapons. Traditional arms-control negotiations necessitated that each side tell the other what its capabilities were as a prelude to limiting those capacities. Yet with cyber, each country will be extremely reluctant to let others know its capabilities. Thus, there is no self-evident negotiated way to contain cyberwarfare. And artificial intelligence compounds this problem. Machines that can learn from their own experience and communicate with one another on their own raise both a practical and a moral imperative to find a way to keep mankind from destroying itself. The United States and China must strive to come to an understanding about the nature of their co-evolution.

How German nuclear scientists reacted to the news of Hiroshima

As part of Operation Epsilon, captured German nuclear physicists were secretly recorded at Farm Hall, a house in England where they were interned. Here’s how the German scientists reacted to the news (on August 6th, 1945) that an atomic bomb had been dropped on Hiroshima, taken from the now-declassified transcripts (pp. 116-122 of this copy):

Otto Hahn (co-discoverer of nuclear fission): I don’t believe it… They are 50 years further advanced than we.

Werner Heisenberg (leading figure of the German atomic bomb effort): I don’t believe a word of the whole thing. They must have spent the whole of their £500,000,000 in separating isotopes: and then it is possible.

In a margin note, the editor points out: “Heisenberg’s figure of £500 million is accurate. At the then-official exchange rate it is equal to $2 billion. President Truman’s account of the expense, released on August 6, stated: ‘We spent $2,000,000,000 on the greatest scientific gamble in history — and won.’ …Isotope separation accounted for a large share but by no means the whole of that…”

Hahn: I didn’t think it would be possible for another 20 years.

Karl Wirtz (head of reactor construction at a German physics institute): I’m glad we didn’t have it.

Carl Friedrich von Weizsäcker (theoretical physicist): I think it is dreadful of the Americans to have done it. I think it is madness on their part.

Heisenberg: One can’t say that. One could equally well say “That’s the quickest way of ending the war.”

Hahn: That’s what consoles me.

Heisenberg: I don’t believe a word about the bomb but I may be wrong…

Hahn: Once I wanted to suggest that all uranium should be sunk to the bottom of the ocean. I always thought that one could only make a bomb of such a size that a whole province would be blown up.

Weizsäcker: How many people were working on V1 and V2?

Kurt Diebner (physicist and organizer of the German Army’s fission project): Thousands worked on that.

Heisenberg: We wouldn’t have had the moral courage to recommend to the government in the spring of 1942 that they should employ 120,000 men just for building the thing up.

Weizsäcker: I believe the reason we didn’t do it was because all the physicists didn’t want to do it, on principle. If we had all wanted Germany to win the war we would have succeeded.

Hahn: I don’t believe that but I am thankful we didn’t succeed.

There is much more of interest in these transcripts. It is fascinating to eavesdrop on leading scientists’ unfiltered comments as they realize how badly their team was beaten to the finish line, and that the whole world has stepped from one era into another.

Hanson on intelligence explosion, from Age of Em

Economist Robin Hanson is among the most informed critics of the plausibility of what he calls a “local” intelligence explosion. He’s written on the topic many times before (most of it collected here), but here’s one more take from him on it, from Age of Em:

…some people foresee a rapid local “intelligence explosion” happening soon after a smart AI system can usefully modify its own mental architecture…

In a prototypical local explosion scenario, a single AI system with a supporting small team starts with resources that are tiny on a global scale. This team finds and then applies a big innovation in AI software architecture to its AI system, which allows this team plus AI combination to quickly find several related innovations. Together this innovation set allows this AI to quickly become more effective than the entire rest of the world put together at key tasks of theft or innovation.

That is, even though an entire world economy outside of this team, including other AIs, works to innovate, steal, and protect itself from theft, this one small AI team becomes vastly better at some combination of (1) stealing resources from others, and (2) innovating to make this AI “smarter,” in the sense of being better able to do a wide range of mental tasks given fixed resources. As a result of being better at these things, this AI quickly grows the resources that it controls and becomes more powerful than the entire rest of the world economy put together, and so it takes over the world. And all this happens within a space of days to months.

Advocates of this explosion scenario believe that there exists an as-yet-undiscovered but very powerful architectural innovation set for AI system design, a set that one team could find first and then keep secret from others for long enough. In support of this belief, advocates point out that humans (1) can do many mental tasks, (2) beat out other primates, (3) have a common IQ factor explaining correlated abilities across tasks, and (4) display many reasoning biases. Advocates also often assume that innovation is vastly underfunded today, that most economic progress comes from basic research progress produced by a few key geniuses, and that the modest wage gains that smarter people earn today vastly underestimate their productivity in key tasks of theft and AI innovation. In support, advocates often point to familiar myths of geniuses revolutionizing research areas and weapons.

Honestly, to me this local intelligence explosion scenario looks suspiciously like a super-villain comic book plot. A flash of insight by a lone genius lets him create a genius AI. Hidden in its super-villain research lab lair, this genius villain AI works out unprecedented revolutions in AI design, turns itself into a super-genius, which then invents super-weapons and takes over the world. Bwa-ha-ha.

Many arguments suggest that this scenario is unlikely (Hanson and Yudkowsky 2013). Specifically, (1) in 60 years of AI research high-level architecture has only mattered modestly for system performance, (2) new AI architecture proposals are increasingly rare, (3) algorithm progress seems driven by hardware progress (Grace 2013), (4) brains seem like ecosystems, bacteria, cities, and economies in being very complex systems where architecture matters less than a mass of capable detail, (5) human and primate brains seem to differ only modestly, (6) the human primate difference initially only allowed faster innovation, not better performance directly, (7) humans seem to have beat other primates mainly via culture sharing, which has a plausible threshold effect and so doesn’t need much brain difference, (8) humans are bad at most mental tasks irrelevant for our ancestors, (9) many human “biases” are useful adaptations to social complexity, (10) human brain structure and task performance suggest that many distinct modules contribute on each task, explaining a common IQ factor (Hampshire et al. 2012), (11) we expect very smart AI to still display many biases, (12) research today may be underfunded, but not vastly so (Alston et al. 2011; Ulku 2004), (13) most economic progress does not come from basic research, (14) most research progress does not come from a few geniuses, and (15) intelligence is not vastly more productive for research than for other tasks.

(And yes, the entire book is roughly this succinct and dense with ideas.)

Rockefeller’s chief philanthropy advisor

Frederick T. Gates was the chief philanthropic advisor to oil tycoon John D. Rockefeller, arguably the richest person in modern history and one of the era’s greatest philanthropists. Here’s a brief profile from Rockefeller biography Titan (h/t @danicgross):

Like Rockefeller himself, Gates yoked together two separate selves—one shrewd and worldly, the other noble and high-flown…

After graduating from the seminary in 1880, Gates was assigned his first pastorate in Minnesota. When his young bride, Lucia Fowler Perkins, dropped dead from a massive internal hemorrhage after sixteen months of marriage, the novice pastor not only suffered an erosion of faith but began to question the competence of American doctors — a skepticism that later had far-reaching ramifications for Rockefeller’s philanthropies…

Eventually Gates became Rockefeller’s philanthropic advisor, and:

What Gates gave to his boss was no less vital. Rockefeller desperately needed intelligent assistance in donating his money at a time when he could not draw on a profession of philanthropic experts. Painstakingly thorough, Gates combined moral passion with great intellect. He spent his evenings bent over tomes of medicine, economics, history, and sociology, trying to improve himself and find clues on how best to govern philanthropy. Skeptical by nature, Gates saw a world crawling with quacks and frauds, and he enjoyed grilling people with trenchant questions to test their sincerity. Outspoken, uncompromising, he never hesitated to speak his piece to Rockefeller and was a peerless troubleshooter.

For some details on Rockefeller’s philanthropic successes, see here.

Stich on conceptual analysis

Stich (1990), p. 3:

On the few occasions when I have taught the “analysis of knowledge” literature to undergraduates, it has been painfully clear that most of my students had a hard time taking the project seriously. The better students were clever enough to play fill-in-the-blank with ‘S knows that p if and only _____’ … But they could not, for the life of them, see why anybody would want to do this. It was a source of ill-concealed amazement to these students that grown men and women would indulge in this exercise and think it important — and of still greater amazement that others would pay them to do it! This sort of discontent was all the more disquieting because deep down I agreed with my students. Surely something had gone very wrong somewhere when clever philosophers, the heirs to the tradition of Hume and Kant, devoted their time to constructing baroque counterexamples about the weird ways in which a man might fail to own a Ford… for about as long as I can remember I have had deep…misgivings about the project of analyzing epistemic notions.

Social justice and evidence

Galileo’s Middle Finger has some good coverage of several case studies in politicized science, and ends with a sermon on the importance of evidence to social justice:


When I joined the early intersex-rights movement, although identity activists inside and outside academia were a dime a dozen, it was pretty uncommon to run into evidence-based activists… Today, all over the place, one finds activist groups collecting and understanding data, whether they’re working on climate change or the rights of patients, voters, the poor, LGBT people, or the wrongly imprisoned…

The bad news is that today advocacy and scholarship both face serious threats. As for social activism, while the Internet has made it cheaper and easier than ever to organize and agitate, it also produces distraction and false senses of success. People tweet, blog, post messages on walls, and sign online petitions, thinking somehow that noise is change. Meanwhile, the people in power just wait it out, knowing that the attention deficit caused by Internet overload will mean the mob will move on to the next house tomorrow, sure as the sun comes up in the morning. And the economic collapse of the investigative press caused by that noisy Internet means no one on the outside will follow through to sort it out, to tell us what is real and what is illusory. The press is no longer around to investigate, spread stories beyond the already aware, and put pressure on those in power to do the right thing.

The threats to scholars, meanwhile, are enormous and growing. Today over half of American university faculty are in non-tenure-track jobs. (Most have not consciously chosen to live without tenure, as I have.) Not only are these people easy to get rid of if they make trouble, but they are also typically loaded with enough teaching and committee work to make original scholarship almost impossible… Add to this the often unfair Internet-based attacks on researchers who are perceived as promoting dangerous messages, and what you end up with is regression to the safe — a recipe for service of those already in power.

Perhaps most troubling is the tendency within some branches of the humanities to portray scholarly quests to understand reality as quaint or naive, even colonialist and dangerous. Sure, I know: Objectivity is easily desired and impossible to perfectly achieve, and some forms of scholarship will feed oppression, but to treat those who seek a more objective understanding of a problem as fools or de facto criminals is to betray the very idea of an academy of learners. When I run into such academics — people who will ignore and, if necessary, outright reject any fact that might challenge their ideology, who declare scientific methodologies “just another way of knowing” — I feel this crazy desire to institute a purge… Call me ideological for wanting us all to share a belief in the importance of seeking reliable, verifiable knowledge, but surely that is supposed to be the common value of the learned.

…I want to say to activists: If you want justice, support the search for truth. Engage in searches for truth. If you really want meaningful progress and not just temporary self-righteousness, carpe datum. You can begin with principles, yes, but to pursue a principle effectively, you have to know if your route will lead to your destination. If you must criticize scholars whose work challenges yours, do so on the evidence, not by poisoning the land on which we all live.

…Here’s the one thing I now know for sure after this very long trip: Evidence really is an ethical issue, the most important ethical issue in a modern democracy. If you want justice, you must work for truth.

Naturally, the sermon is more potent if you’ve read the case studies in the book.

Check the original source

From Segerstrale (2000), p. 27:

In 1984 I was able to shock my class of well-intended liberal students at Smith College by giving them the assignment to compare [Stephan] Chorover’s [critical] representation of passages of [E.O. Wilson’s] Sociobiology with Wilson’s original text. The students, who were deeply suspicious of Wilson and spontaneous champions of his critics, embarked on this homework with gusto. Many students were quite dismayed at their own findings and angry with Chorover. This surely says something, too, about these educated laymen’s relative innocence regarding what can and cannot be done in academia.

I wish this kind of exercise was more common. Another I would suggest is to compare critics’ representations of Dreyfus’ “Alchemy and Artificial Intelligence” with the original text (see here).

The first AI textbook, on the control problem

The earliest introductory AI textbook I know about — excluding mere “paper collections” like Computers and Thought (1963) — is Jackson’s Introduction to Artificial Intelligence (1974).

It discusses AGI and the control problem starting on page 394:

If [AI] research is unsuccessful at producing a general artificial intelligence, over a period of more than a hundred years, then its failure may raise some serious doubt among many scientists as to the finite describability of man and his universe. However, the evidence presented in this book makes it seem likely that artificial intelligence research will be successful, that a technology will be developed which is capable of producing machines that can demonstrate most, if not all, of the mental abilities of human beings. Let us therefore assume that this will happen, and imagine two worlds that might result.

[First,] …It is not difficult to envision actualities in which an artificial intelligence would exert control over human beings, yet be out of their control.

Given that intelligent machines are to be used, the question of their control and noncontrol must be answered. If a machine is programmed to seek certain ends, how are we to insure that the means it chooses to employ are agreeable to people? A preliminary solution to the problem is given by the fact that we can specify state-space problems to require that their solution paths shall not pass through certain states (see Chapter 3). However, the task of giving machines more sophisticated value systems, and especially of making them ‘ethical,’ has not yet been investigated by AI researchers…

The question of control should be coupled with the ‘lack of understanding’ question; that is, the possibility exists that intelligent machines might be too complicated for us to understand in situations that require real-time analyses (see the discussion of evolutionary programs in Chapter 8). We could conceivably always demand that a machine give a complete output of its reasoning on a problem; nevertheless that reasoning might not be effectively understandable to us if the problem itself were to determine a time limit for producing a solution. In such a case, if we were to act rationally, we might have to follow the machine’s advice without understanding its ‘motives’…

It has been suggested that an intelligent machine might arise accidentally, without our knowledge, through some fortuitous interconnection of smaller machines (see Heinlein, 1966). If the smaller machines each helped to control some aspect of our economy or defense, the accidental intelligent might well act as a dictator… It seems highly unlikely that this will happen, especially if we devote sufficient time to studying the non-accidental systems we implement.

A more significant danger is that artificial intelligence might be used to further the interests of human dictators. A limited supply of intelligent machines in the hands of a human dictator might greatly increase his power over other human beings, perhaps to the extent of giving him complete censorship and supervision of the public…

Let us now paint another, more positive picture of the world that might result from artificial intelligence research… It is a world in which man and his machines have reached a state of symbiosis…

The benefits humanity might gain from achieving such a symbiosis are enormous. As mentioned [earlier], it may be possible for artificial intelligence to greatly reduce the amount of human labor necessary to operate the economy of the world… Computers and AI research may play an important part in helping to overcome the food, population, housing, and other crises that currently grip the earth… Artificial intelligence may eventually be used to… partially automated the development of science itself… Perhaps artificial intelligence will someday be used in automatic teachers… and perhaps mechanical translators will someday be developed which will fluently translate human languages. And (very perhaps) the day may eventually come when the ‘household robot’ and the ‘robot chauffeur’ will be a reality…

In some ways it is reassuring that the progress in artificial intelligence research is proceeding at a relatively slow but regular pace. It should be at least a decade before any of these possibilities becomes an actuality, which will give us some time to consider in more detail the issues involved.

“Beyond the scope of this paper”

From AI scientist Drew McDermott, in 1976:

In this paper I have criticized AI researchers very harshly. Let me express my faith that people in other fields would, on inspection, be found to suffer from equally bad faults. Most AI workers are responsible people who are aware of the pitfalls of a difficult field and produce good work in spite of them. However, to say anything good about anyone is beyond the scope of this paper.