- Bensinger, Groundwork for AGI Safety Engineering.
- The Good Country Index.
- The Economist reviews Superintelligence.
- New from Tetlock et al., “Forecasting tournaments: tools for increasing transparency and improving the quality of debate.”
Poor Economics is the best book I’ve read on poverty reduction. The book ends with a summary of its key lessons. Here they are:
As this book has shown, although we have no magic bullets to eradicate poverty, no one-shot cure-all, we do know a number of things about how to improve the lives of the poor. In particular, five key lessons emerge.
First, the poor often lack critical pieces of information and believe things that are not true. They are unsure about the benefits of immunizing children; they think there is little value in what is learned during the first few years of education; they don’t know how much fertilizer they need to use; they don’t know which is the easiest way to get infected with HIV; they don’t know what their politicians do when in office. When their firmly held beliefs turn out to be incorrect, they end up making the wrong decision, sometimes with drastic consequences — think of the girls who have unprotected sex with older men or the farmers who use twice as much fertilizer as they should. Even when they know that they don’t know, the resulting uncertainty can be damaging. For example, the uncertainty about the benefits of immunization combines with the universal tendency to procrastinate, with the result that a lot of children don’t get immunized. Citizens who vote in the dark are more likely to vote for someone of their ethnic group, at the cost of increasing bigotry and corruption.
We saw many instances in which a simple piece of information makes a big difference. However, not every information campaign is effective. It seems that in order to work, an information campaign must have several features: It must say something that people don’t already know (general exhortations like “No sex before marriage” seem to be less effective); it must do so in an attractive and simple way (a film, a play, a TV show, a well-designed report card); and it must come from a credible source (interestingly, the press seems to be viewed as credible). One of the corollaries of this view is that governments pay a huge cost in terms of lost credibility when they say things that are misleading, confusing, or false.
Second, the poor bear responsibility for too many aspects of their lives. The richer you are, the more the “right” decisions are made for you. The poor have no piped water, and therefore do not benefit from the chlorine that the city government puts into the water supply. If they want clean drinking water, they have to purify it themselves. They cannot afford ready-made fortified breakfast cereals and therefore have to make sure that they and their children get enough nutrients. They have no automatic way to save, such as a retirement plan or a contribution to Social Security, so they have to find a way to make sure that they save. These decisions are difficult for everyone because they require some thinking now or some other small cost today, and the benefits are usually reaped in the distant future. As such, procrastination very easily gets in the way. For the poor, this is compounded by the fact that their lives are already much more demanding than ours: Many of them run small businesses in highly competitive industries; most of the rest work as casual laborers and need to constantly worry about where their next job will come from. This means that their lives could be significantly improved by making it as easy as possible to do the right thing — based on everything else we know — using the power of default options and small nudges: Salt fortified with iron and iodine could be made cheap enough that everyone buys it. Savings accounts, the kind that make it easy to put in money and somewhat costlier to take it out, can be made easily available to everyone, if need be, by subsidizing the cost for the bank that offers them. Chlorine could be made available next to every source where piping water is too expensive. There are many similar examples.
Third, there are good reasons that some markets are missing for the poor, or that the poor face unfavorable prices in them. The poor get a negative interest rate from their savings accounts (if they are lucky enough to have an account) and pay exorbitant rates on their loans (if they can get one) because handling even a small quantity of money entails a fixed cost. The market for health insurance for the poor has not developed, despite the devastating effects of serious health problems in their lives because the limited insurance options that can be sustained in the market (catastrophic health insurance, formulaic weather insurance) are not what the poor want.
In some cases, a technological or an institutional innovation may allow a market to develop where it was missing. This happened in the case of microcredit, which made small loans at more affordable rates available to millions of poor people, although perhaps not the poorest. Electronic money transfer systems (using cell phones and the like) and unique identification for individuals may radically cut the cost of providing savings and remittance services to the poor over the next few years. But we also have to recognize that in some cases, the conditions for a market to emerge on its own are simply not there. In such cases, governments should step in to support the market to provide the necessary conditions, or failing that, consider providing the service themselves.
We should recognize that this may entail giving away goods or services (such as bed nets or visits to a preventive care center) for free or even rewarding people, strange as it might sound, for doing things that are good for them. The mistrust of free distribution of goods and services among various experts has probably gone too far, even from a pure cost-benefit point of view. It often ends up being cheaper, per person served, to distribute a service for free than to try to extract a nominal fee. In some cases, it may involve ensuring that the price of a product sold by the market is attractive enough to allow the market to develop. For example, governments could subsidize insurance premiums, or distribute vouchers that parents can take to any school, private or public, or force banks to offer free “no frills” savings accounts to everyone for a nominal fee. It is important to keep in mind that these subsidized markets need to be carefully regulated to ensure they function well. For example, school vouchers work well when all parents have a way of figuring out the right school for their child; otherwise, they can turn into a way of giving even more of an advantage to savvy parents.
Fourth, poor countries are not doomed to failure because they are poor, or because they have had an unfortunate history. It is true that things often do not work in these countries: Programs intended to help the poor end up in the wrong hands, teachers teach desultorily or not at all, roads weakened by theft of materials collapse under the weight of overburdened trucks, and so forth. But many of these failures have less to do with some grand conspiracy of the elites to maintain their hold on the economy and more to do with some avoidable flaw in the detailed design of policies, and the ubiquitous three Is: ignorance, ideology, and inertia. Nurses are expected to carry out jobs that no ordinary human being would be able to complete, and yet no one feels compelled to change their job description. The fad of the moment (be it dams, barefoot doctors, microcredit, or whatever) is turned into a policy without any attention to the reality within which it is supposed to function. We were once told by a senior government official in India that the village education committees always include the parent of the best student in the school and the parent of the worst student in the school. When we asked how they decided who were the best and worst children, given that there are no tests until fourth grade, she quickly changed subjects. And yet even these absurd rules, once in place, keep going out of sheer inertia.
The good news, if that is the right expression, is that it is possible to improve governance and policy without changing the existing social and political structures. There is tremendous scope for improvement even in “good” institutional environments, and some margin for action even in bad ones. A small revolution can be achieved by making sure that everyone is invited to village meetings; by monitoring government workers and holding them accountable for failures in performing their duties; by monitoring politicians at all levels and sharing this information with voters; and by making clear to users of public services what they should expect—what the exact health center hours are, how much money (or how many bags of rice) they are entitled to.
Finally, expectations about what people are able or unable to do all too often end up turning into self-fulfilling prophecies. Children give up on school when their teachers (and sometimes their parents) signal to them that they are not smart enough to master the curriculum; fruit sellers don’t make the effort to repay their debt because they expect that they will fall back into debt very quickly; nurses stop coming to work because nobody expects them to be there; politicians whom no one expects to perform have no incentive to try improving people’s lives. Changing expectations is not easy, but it is not impossible: After seeing a female pradhan in their village, villagers not only lost their prejudice against women politicians but even started thinking that their daughter might become one, too; teachers who are told that their job is simply to make sure that all the children can read can accomplish that task within the duration of a summer camp. Most important, the role of expectations means that success often feeds on itself. When a situation starts to improve, the improvement itself affects beliefs and behavior. This is one more reason one should not necessarily be afraid of handing things out (including cash) when needed to get a virtuous cycle started.
- Resolutions of mathematical conjectures over time.
- Peter McCluskey on Superintelligence and AI takeoff.
- Humor: Wrong Hands (visual puns).
- Michael Barr’s slides on the software defects which caused unintended acceleration in the 2005 Toyota Camry provide a good illustration of how easy it is for well-resourced projects to nevertheless fail to achieve high assurance software in numerous ways.
- Visualizing Africa’s progress.
Several decent, enjoyable books:
- Pennebaker’s The Secret Life of Pronouns
- Levy’s In the Plex
- Ellenberg’s How Not to Be Wrong
- Heller’s Ayn Rand and the World She Made
- Stone’s The Everything Store
- Stross’ The Launch Pad
- Hoffman’s The Dead Hand
Quammen’s Spillover was not particularly “enjoyable” given it’s subject matter, but it was informative and engaging.
Banerjee & Duflo’s Poor Economics is one of the most persuasive books I’ve read on the subject of poverty reduction.
Murray’s Curmudgeon’s Guide to Getting Ahead was short and entertaining but a mixed bag.
Lochbaum et al’s Fukushima was a helpful overview of exactly what happened at Fukushima, in what order, what the policy response was, and what the health and political fallout was.
- “Approval-seeking”: Christiano’s latest post on indirect normativity for AGI.
- “Accuracy of forecasts in strategic intelligence” (in PNAS).
- Some long-term effective altruism advice by Nick Beckstead that I generally agree with: Working in AI or synbio and Finding a job at an organization focused on existential risk.
- Paul Christiano & Katja Grace made some amendments to MIRI’s AI predictions dataset. They explain their amendments and compute the new statistics here.
- Orseau’s two new AGI papers: “Teleporting universal intelligent agents” and “A formal model for multiple, copiable AIs.”
- Bostrom’s new Superintelligence talk, accompanying his book.
- Lilienfeld et al – “Why ineffective psychotherapies appear to work, a taxonomy.”
- A visualization of Superintelligence.
Richard Dawkins and Jon Stewart discussed existential risk on the Sept. 24, 2013 edition of The Daily Show. Here’s how it went down:
STEWART: Here’s my proposal… for the discussion tonight. Do you believe that the end of our civilization will be through religious strife or scientific advancement? What do you think in the long run will be more damaging to our prospects as a human race?
In reply, Dawkins said that Martin Rees (of CSER) thinks humanity has a 50% chance of surviving the 21st century, and one cause for such worry is that powerful technologies could get into the hands of religious fanatics. Stewart replied:
STEWART: … [But] isn’t there a strong probability that we are not necessarily in control of the unintended consequences of our scientific advancement?… Don’t you think it’s even more likely that we will create something [for which] the unintended consequence… is worldwide catastrophe?
DAWKINS: That is possible. It’s something we have to worry about… Science is the most powerful way to do whatever you want to do. If you want to do good, it’s the most powerful way to do good. If you want to do evil, it’s the most powerful way to do evil.
STEWART: … You have nuclear energy and you go this way and you can light the world, but you go this [other] way, and you can blow up the world. It seems like we always try [the blow up the world path] first.
DAWKINS: There is a suggestion that one of the reasons that we don’t detect extraterrestrial civilizations is that when a civilization reaches the point where it could broadcast radio waves that we could pick up, there’s only a brief window before it blows itself up… It takes many billions of years for evolution to reach the point where technology takes off, but once technology takes off, it’s then an eye-blink — by the standards of geological time — before…
STEWART: … It’s very easy to look at the dark side of fundamentalism… [but] sometimes I think we have to look at the dark side of achievement… because I believe the final words that man utters on this Earth will be: “It worked!” It’ll be an experiment that isn’t misused, but will be a rolling catastrophe.
DAWKINS: It’s a possibility, and I can’t deny it. I’m more optimistic than that.
STEWART: … [I think] curiosity killed the cat, and the cat never saw it coming… So how do we put the brakes on our ability to achieve, or our curiosity?
DAWKINS: I don’t think you can ever really stop the march of science in the sense of saying “You’re forbidden to exercise your natural curiosity in science.” You can certainly put the brakes on certain applications. You could stop manufacturing certain weapons. You could have… international agreements not to manufacture certain types of weapons…
- Müller & Bostrom, Future progress in artificial intelligence: a poll among experts.
- Ord, The timing of labour aimed at reducing existential risk.
- In the USA, avionics software must be certified by designated specialists beholden to (e.g.) the FAA. But when it comes to software for self-driving cars, Google is pushing hard for a system of self-certification.
- So apparently there was a hit TV show in the late 90s, with more viewers than Game of Thrones, about a naked guy in a small room who had to survive entirely on sweepstakes winnings (e.g. dog food) for more than a year and who didn’t know he was on TV the whole time. Obviously, this happened in Japan.
Nick Bostrom’s new book — Superintelligence: Paths, Dangers, Strategies — was published today in the UK by Oxford University Press. It opens with a fable about some sparrows and an owl:
It was the nest-building season, but after days of long hard work, the sparrows sat in the evening glow, relaxing and chirping away.
“We are all so small and weak. Imagine how easy life would be if we had an owl who could help us build our nests!”
“Yes!” said another. “And we could use it to look after our elderly and our young.”
“It could give us advice and keep an eye out for the neighborhood cat,” added a third.
Then Pastus, the elder-bird, spoke: “Let us send out scouts in all directions and try to find an abandoned owlet somewhere, or maybe an egg. A crow chick might also do, or a baby weasel. This could be the best thing that ever happened to us, at least since the opening of the Pavilion of Unlimited Grain in yonder backyard.”
The flock was exhilarated, and sparrows everywhere started chirping at the top of their lungs.
Only Scronkfinkle, a one-eyed sparrow with a fretful temperament, was unconvinced of the wisdom of the endeavor. Quoth he: “This will surely be our undoing. Should we not give some thought to the art of owl-domestication and owl-taming first, before we bring such a creature into our midst?”
Replied Pastus: “Taming an owl sounds like an exceedingly difficult thing to do. It will be difficult enough to find an owl egg. So let us start there. After we have succeeded in raising an owl, then we can think about taking on this other challenge.”
“There is a flaw in that plan!” squeaked Scronkfinkle; but his protests were in vain as the flock had already lifted off to start implementing the directives set out by Pastus.
Just two or three sparrows remained behind. Together they began to try to work out how owls might be tamed or domesticated. They soon realized that Pastus had been right: this was an exceedingly difficult challenge, especially in the absence of an actual owl to practice on. Nevertheless they pressed on as best they could, constantly fearing that the flock might return with an owl egg before a solution to the control problem had been found.
It is not known how the story ends, but the author dedicates this book to Scronkfinkle and his followers.
(Last updated Feb. 11, 2015.)
What could an economics graduate student do to improve our strategic picture of superintelligence? What about a computer science professor? A policy analyst at RAND? A program director at IARPA?
In the last chapter of Superintelligence, Nick Bostrom writes:
We find ourselves in a thicket of strategic complexity and surrounded by a dense mist of uncertainty. Though many considerations have been discerned, their details and interrelationships remain unclear and iffy — and there might be other factors we have not thought of yet. How should we act in this predicament?
… Against a backdrop of perplexity and uncertainty, [strategic] analysis stands out as being of particularly high expected value. Illumination of our strategic situation would help us target subsequent interventions more effectively. Strategic analysis is especially needful when we are radically uncertain not just about some detail of some peripheral matter but about the cardinal qualities of the central things. For many key parameters, we are radically uncertain even about their sign…
The hunt for crucial considerations… will often require crisscrossing the boundaries between different academic disciplines and other fields of knowledge.
Bostrom does not, however, provide a list of specific research projects that could illuminate our strategic situation and thereby “help us target subsequent interventions more effectively.”
Below is my personal list of studies which could illuminate our strategic situation with regard to superintelligence. I’m hosting it on my personal site rather than MIRI’s blog to make it clear that this is not “MIRI’s official list of project ideas.” Other researchers at MIRI would, I’m sure, put together a different list. [Read more…]
I read Thiel’s Zero to One (2014) in May, but forgot to mention it in the May books post. I enjoyed it very much. His key argument is that progress comes from monopolies, not from strong competition, so we should encourage certain kinds of monopolies. I generally agree. I also agree with Thiel that technological progress has slowed since the 70s, with the (lone?) exception of IT.
The Info Mesa (2003), by Ed Regis, is fine but less interesting than Great Mambo Chicken (which I’m currently reading) and Nano (which I finished last month).
The Atomic Bazaar (2003), by William Langewiesche, tells the story of nuclear trafficking and the rise of poor countries with nuclear weapons programs, and especially the activities of Abdul Qadeer Khan. It was pretty good, though I wish it had done a better job of explaining the limits, opportunities, and incentives at play in the nuclear arms trade.
Age of Ambition (2014), by Evan Osnos, is a fantastically rich portrait of modern China. Highly recommended.
Human Accomplishment (2003), by Charles Murray, is a fine specimen of quantitative historical analysis. The final chapters are less persuasive than the rest of the book, but despite this terms like magisterial and tour de force come to mind. Murray does an excellent job walking the reader through his methodology, its pros and cons, the reasons for it, and the conclusions that can and can’t be drawn from it. You’ll probably like this if you enjoyed Pinker’s Better Angels of Our Nature.
Superintelligence (2014), by Nick Bostrom, is a fantastic summary of the last ~15 years of strategic thinking about machine superintelligence from (largely) FHI and MIRI, the two institutes focused most directly on the issue. If you want to get a sense of what’s been learned during that time, first read Bostrom’s 1997 paper on superintelligence (and other topics), and then read his new book. It comes out in the UK on July 3rd and in the USA on September 3rd. Highly recommended.
The Honest Truth about Dishonesty (2013), by Dan Ariely, is as fun and practical as the other Ariely books. Recommended.
When you’re not sure what to think about something, or what to do in a certain situation, do you instinctively turn to a successful domain expert, or to someone you know who seems generally very smart?
I think most people don’t respect individual differences in intelligence and rationality enough. But some people in my local community tend to exhibit the opposite failure mode. They put too much weight on a person’s signals of explicit rationality (“Are they Bayesian?”), and place too little weight on domain expertise (and the domain-specific tacit rationality that often comes with it).
This comes up pretty often during my work for MIRI. We’re considering how to communicate effectively with academics, or how to win grants, or how to build a team of researchers, and some people (not necessarily MIRI staff) will tend to lean heavily on the opinions of the most generally smart people they know, even though those smart people have no demonstrated expertise or success on the issue being considered. In contrast, I usually collect the opinions of some smart people I know, and then mostly just do what people with a long track record of success on the issue say to do. And that dumb heuristic seems to work pretty well.
Yes, there are nuanced judgment calls I have to make about who has expertise on what, exactly, and whether MIRI’s situation is sufficiently analogous for the expert’s advice to work at MIRI. And I must be careful to distinguish credentials-expertise from success-expertise (aka RSPRT-expertise). And this process doesn’t work for decisions on which there are no success-experts, like long-term AI forecasting. But I think it’s easier for smart people to overestimate their ability to model problems outside their domains of expertise, and easier to underestimate all the subtle things domain experts know, than vice-versa.
- Beckstead, Will we eventually be able to colonize other stars?
- GiveWell conversation with Tom Dietterich about long-term AI safety.
- Two professional soccer players vs. 55 children (video).
- Sean Carroll: Physicists should stop saying silly things about philosophy.
- BPS Research Digest: How can we increase altruism towards future generations?
In general and across all instances I can think of so far, I do not agree with the part of your futurological forecast in which you reason, “After event W happens, everyone will see the truth of proposition X, leading them to endorse Y and agree with me about policy decision Z.”
Example 2: “As AI gets more sophisticated, everyone will realize that real AI is on the way and then they’ll start taking Friendly AI development seriously.”
Alternative projection: As AI gets more sophisticated, the rest of society can’t see any difference between the latest breakthrough reported in a press release and that business earlier with Watson beating Ken Jennings or Deep Blue beating Kasparov; it seems like the same sort of press release to them. The same people who were talking about robot overlords earlier continue to talk about robot overlords. The same people who were talking about human irreproducibility continue to talk about human specialness. Concern is expressed over technological unemployment the same as today or Keynes in 1930, and this is used to fuel someone’s previous ideological commitment to a basic income guarantee, inequality reduction, or whatever. The same tiny segment of unusually consequentialist people are concerned about Friendly AI as before. If anyone in the science community does start thinking that superintelligent AI is on the way, they exhibit the same distribution of performance as modern scientists who think it’s on the way, e.g. Hugo de Garis, Ben Goertzel, etc.
My own projection goes more like this:
As AI gets more sophisticated, and as more prestigious AI scientists begin to publicly acknowledge that AI is plausibly only 2-6 decades away, policy-makers and research funders will begin to respond to the AGI safety challenge, just like they began to respond to CFC damages in the late 70s, to global warming in the late 80s, and to synbio developments in the 2010s. As for society at large, I dunno. They’ll think all kinds of random stuff for random reasons, and in some cases this will seriously impede effective policy, as it does in the USA for science education and immigration reform. Because AGI lends itself to arms races and is harder to handle adequately than global warming or nuclear security are, policy-makers and industry leaders will generally know AGI is coming but be unable to fund the needed efforts and coordinate effectively enough to ensure good outcomes.
At least one clear difference between my projection and Yudkowsky’s is that I expect AI-expert performance on the problem to improve substantially as a greater fraction of elite AI scientists begin to think about the issue in Near mode rather than Far mode.
As a friend of mine suggested recently, current elite awareness of the AGI safety challenge is roughly where elite awareness of the global warming challenge was in the early 80s. Except, I expect elite acknowledgement of the AGI safety challenge to spread more slowly than it did for global warming or nuclear security, because AGI is tougher to forecast in general, and involves trickier philosophical nuances. (Nobody was ever tempted to say, “But as the nuclear chain reaction grows in power, it will necessarily become more moral!”)
Still, there is a worryingly non-negligible chance that AGI explodes “out of nowhere.” Sometimes important theorems are proved suddenly after decades of failed attempts by other mathematicians, and sometimes a computational procedure is sped up by 20 orders of magnitude with a single breakthrough.