- Aaronson, “Could a quantum computer have subjective experience?”
- Against Empathy, a debate between Paul Bloom, Peter Singer, Sam Harris, Jesse Prinz, and others.
- Obstacles to self-driving cars.
- “Physical principles for scalable neural recording” maps the physical limits on brain activity mapping for several different methods. Potentially informative for whole brain emulation forecasting.
- Humor: Thanks, Textbooks is a Tumblr for amusing, weird textbook writing. For example, here’s one textbooks “colloquial” explanation of the 2nd law of thermodynamics: “You can’t shovel manure into the rear end of a horse and expect to get hay out its mouth.”
- Academic urban legends, a study.
- Markov, “Limits on fundamental limits to computation.”
- Funny: An honest political ad for Gil Fulbright.
- Well-produced explanatory video on automation and technological unemployment. Not enough caveats, but of course it’s more engaging that way.
- Molyneux, “How the (hard) problem of consciousness could emerge in robots.”
- And, an overview of reductionist approaches to consciousness, by James Reggia.
- Bensinger, Groundwork for AGI Safety Engineering.
- The Good Country Index.
- The Economist reviews Superintelligence.
- New from Tetlock et al., “Forecasting tournaments: tools for increasing transparency and improving the quality of debate.”
Poor Economics is the best book I’ve read on poverty reduction. The book ends with a summary of its key lessons. Here they are:
As this book has shown, although we have no magic bullets to eradicate poverty, no one-shot cure-all, we do know a number of things about how to improve the lives of the poor. In particular, five key lessons emerge.
First, the poor often lack critical pieces of information and believe things that are not true. They are unsure about the benefits of immunizing children; they think there is little value in what is learned during the first few years of education; they don’t know how much fertilizer they need to use; they don’t know which is the easiest way to get infected with HIV; they don’t know what their politicians do when in office. When their firmly held beliefs turn out to be incorrect, they end up making the wrong decision, sometimes with drastic consequences — think of the girls who have unprotected sex with older men or the farmers who use twice as much fertilizer as they should. Even when they know that they don’t know, the resulting uncertainty can be damaging. For example, the uncertainty about the benefits of immunization combines with the universal tendency to procrastinate, with the result that a lot of children don’t get immunized. Citizens who vote in the dark are more likely to vote for someone of their ethnic group, at the cost of increasing bigotry and corruption.
We saw many instances in which a simple piece of information makes a big difference. However, not every information campaign is effective. It seems that in order to work, an information campaign must have several features: It must say something that people don’t already know (general exhortations like “No sex before marriage” seem to be less effective); it must do so in an attractive and simple way (a film, a play, a TV show, a well-designed report card); and it must come from a credible source (interestingly, the press seems to be viewed as credible). One of the corollaries of this view is that governments pay a huge cost in terms of lost credibility when they say things that are misleading, confusing, or false.
Second, the poor bear responsibility for too many aspects of their lives. The richer you are, the more the “right” decisions are made for you. The poor have no piped water, and therefore do not benefit from the chlorine that the city government puts into the water supply. If they want clean drinking water, they have to purify it themselves. They cannot afford ready-made fortified breakfast cereals and therefore have to make sure that they and their children get enough nutrients. They have no automatic way to save, such as a retirement plan or a contribution to Social Security, so they have to find a way to make sure that they save. These decisions are difficult for everyone because they require some thinking now or some other small cost today, and the benefits are usually reaped in the distant future. As such, procrastination very easily gets in the way. For the poor, this is compounded by the fact that their lives are already much more demanding than ours: Many of them run small businesses in highly competitive industries; most of the rest work as casual laborers and need to constantly worry about where their next job will come from. This means that their lives could be significantly improved by making it as easy as possible to do the right thing — based on everything else we know — using the power of default options and small nudges: Salt fortified with iron and iodine could be made cheap enough that everyone buys it. Savings accounts, the kind that make it easy to put in money and somewhat costlier to take it out, can be made easily available to everyone, if need be, by subsidizing the cost for the bank that offers them. Chlorine could be made available next to every source where piping water is too expensive. There are many similar examples.
Third, there are good reasons that some markets are missing for the poor, or that the poor face unfavorable prices in them. The poor get a negative interest rate from their savings accounts (if they are lucky enough to have an account) and pay exorbitant rates on their loans (if they can get one) because handling even a small quantity of money entails a fixed cost. The market for health insurance for the poor has not developed, despite the devastating effects of serious health problems in their lives because the limited insurance options that can be sustained in the market (catastrophic health insurance, formulaic weather insurance) are not what the poor want.
In some cases, a technological or an institutional innovation may allow a market to develop where it was missing. This happened in the case of microcredit, which made small loans at more affordable rates available to millions of poor people, although perhaps not the poorest. Electronic money transfer systems (using cell phones and the like) and unique identification for individuals may radically cut the cost of providing savings and remittance services to the poor over the next few years. But we also have to recognize that in some cases, the conditions for a market to emerge on its own are simply not there. In such cases, governments should step in to support the market to provide the necessary conditions, or failing that, consider providing the service themselves.
We should recognize that this may entail giving away goods or services (such as bed nets or visits to a preventive care center) for free or even rewarding people, strange as it might sound, for doing things that are good for them. The mistrust of free distribution of goods and services among various experts has probably gone too far, even from a pure cost-benefit point of view. It often ends up being cheaper, per person served, to distribute a service for free than to try to extract a nominal fee. In some cases, it may involve ensuring that the price of a product sold by the market is attractive enough to allow the market to develop. For example, governments could subsidize insurance premiums, or distribute vouchers that parents can take to any school, private or public, or force banks to offer free “no frills” savings accounts to everyone for a nominal fee. It is important to keep in mind that these subsidized markets need to be carefully regulated to ensure they function well. For example, school vouchers work well when all parents have a way of figuring out the right school for their child; otherwise, they can turn into a way of giving even more of an advantage to savvy parents.
Fourth, poor countries are not doomed to failure because they are poor, or because they have had an unfortunate history. It is true that things often do not work in these countries: Programs intended to help the poor end up in the wrong hands, teachers teach desultorily or not at all, roads weakened by theft of materials collapse under the weight of overburdened trucks, and so forth. But many of these failures have less to do with some grand conspiracy of the elites to maintain their hold on the economy and more to do with some avoidable flaw in the detailed design of policies, and the ubiquitous three Is: ignorance, ideology, and inertia. Nurses are expected to carry out jobs that no ordinary human being would be able to complete, and yet no one feels compelled to change their job description. The fad of the moment (be it dams, barefoot doctors, microcredit, or whatever) is turned into a policy without any attention to the reality within which it is supposed to function. We were once told by a senior government official in India that the village education committees always include the parent of the best student in the school and the parent of the worst student in the school. When we asked how they decided who were the best and worst children, given that there are no tests until fourth grade, she quickly changed subjects. And yet even these absurd rules, once in place, keep going out of sheer inertia.
The good news, if that is the right expression, is that it is possible to improve governance and policy without changing the existing social and political structures. There is tremendous scope for improvement even in “good” institutional environments, and some margin for action even in bad ones. A small revolution can be achieved by making sure that everyone is invited to village meetings; by monitoring government workers and holding them accountable for failures in performing their duties; by monitoring politicians at all levels and sharing this information with voters; and by making clear to users of public services what they should expect—what the exact health center hours are, how much money (or how many bags of rice) they are entitled to.
Finally, expectations about what people are able or unable to do all too often end up turning into self-fulfilling prophecies. Children give up on school when their teachers (and sometimes their parents) signal to them that they are not smart enough to master the curriculum; fruit sellers don’t make the effort to repay their debt because they expect that they will fall back into debt very quickly; nurses stop coming to work because nobody expects them to be there; politicians whom no one expects to perform have no incentive to try improving people’s lives. Changing expectations is not easy, but it is not impossible: After seeing a female pradhan in their village, villagers not only lost their prejudice against women politicians but even started thinking that their daughter might become one, too; teachers who are told that their job is simply to make sure that all the children can read can accomplish that task within the duration of a summer camp. Most important, the role of expectations means that success often feeds on itself. When a situation starts to improve, the improvement itself affects beliefs and behavior. This is one more reason one should not necessarily be afraid of handing things out (including cash) when needed to get a virtuous cycle started.
- Resolutions of mathematical conjectures over time.
- Peter McCluskey on Superintelligence and AI takeoff.
- Humor: Wrong Hands (visual puns).
- Michael Barr’s slides on the software defects which caused unintended acceleration in the 2005 Toyota Camry provide a good illustration of how easy it is for well-resourced projects to nevertheless fail to achieve high assurance software in numerous ways.
- Visualizing Africa’s progress.
Several decent, enjoyable books:
- Pennebaker’s The Secret Life of Pronouns
- Levy’s In the Plex
- Ellenberg’s How Not to Be Wrong
- Heller’s Ayn Rand and the World She Made
- Stone’s The Everything Store
- Stross’ The Launch Pad
- Hoffman’s The Dead Hand
Quammen’s Spillover was not particularly “enjoyable” given it’s subject matter, but it was informative and engaging.
Banerjee & Duflo’s Poor Economics is one of the most persuasive books I’ve read on the subject of poverty reduction.
Murray’s Curmudgeon’s Guide to Getting Ahead was short and entertaining but a mixed bag.
Lochbaum et al’s Fukushima was a helpful overview of exactly what happened at Fukushima, in what order, what the policy response was, and what the health and political fallout was.
- “Approval-seeking”: Christiano’s latest post on indirect normativity for AGI.
- “Accuracy of forecasts in strategic intelligence” (in PNAS).
- Some long-term effective altruism advice by Nick Beckstead that I generally agree with: Working in AI or synbio and Finding a job at an organization focused on existential risk.
- Paul Christiano & Katja Grace made some amendments to MIRI’s AI predictions dataset. They explain their amendments and compute the new statistics here.
- Orseau’s two new AGI papers: “Teleporting universal intelligent agents” and “A formal model for multiple, copiable AIs.”
- Bostrom’s new Superintelligence talk, accompanying his book.
- Lilienfeld et al – “Why ineffective psychotherapies appear to work, a taxonomy.”
- A visualization of Superintelligence.
Richard Dawkins and Jon Stewart discussed existential risk on the Sept. 24, 2013 edition of The Daily Show. Here’s how it went down:
STEWART: Here’s my proposal… for the discussion tonight. Do you believe that the end of our civilization will be through religious strife or scientific advancement? What do you think in the long run will be more damaging to our prospects as a human race?
In reply, Dawkins said that Martin Rees (of CSER) thinks humanity has a 50% chance of surviving the 21st century, and one cause for such worry is that powerful technologies could get into the hands of religious fanatics. Stewart replied:
STEWART: … [But] isn’t there a strong probability that we are not necessarily in control of the unintended consequences of our scientific advancement?… Don’t you think it’s even more likely that we will create something [for which] the unintended consequence… is worldwide catastrophe?
DAWKINS: That is possible. It’s something we have to worry about… Science is the most powerful way to do whatever you want to do. If you want to do good, it’s the most powerful way to do good. If you want to do evil, it’s the most powerful way to do evil.
STEWART: … You have nuclear energy and you go this way and you can light the world, but you go this [other] way, and you can blow up the world. It seems like we always try [the blow up the world path] first.
DAWKINS: There is a suggestion that one of the reasons that we don’t detect extraterrestrial civilizations is that when a civilization reaches the point where it could broadcast radio waves that we could pick up, there’s only a brief window before it blows itself up… It takes many billions of years for evolution to reach the point where technology takes off, but once technology takes off, it’s then an eye-blink — by the standards of geological time — before…
STEWART: … It’s very easy to look at the dark side of fundamentalism… [but] sometimes I think we have to look at the dark side of achievement… because I believe the final words that man utters on this Earth will be: “It worked!” It’ll be an experiment that isn’t misused, but will be a rolling catastrophe.
DAWKINS: It’s a possibility, and I can’t deny it. I’m more optimistic than that.
STEWART: … [I think] curiosity killed the cat, and the cat never saw it coming… So how do we put the brakes on our ability to achieve, or our curiosity?
DAWKINS: I don’t think you can ever really stop the march of science in the sense of saying “You’re forbidden to exercise your natural curiosity in science.” You can certainly put the brakes on certain applications. You could stop manufacturing certain weapons. You could have… international agreements not to manufacture certain types of weapons…
- Müller & Bostrom, Future progress in artificial intelligence: a poll among experts.
- Ord, The timing of labour aimed at reducing existential risk.
- In the USA, avionics software must be certified by designated specialists beholden to (e.g.) the FAA. But when it comes to software for self-driving cars, Google is pushing hard for a system of self-certification.
- So apparently there was a hit TV show in the late 90s, with more viewers than Game of Thrones, about a naked guy in a small room who had to survive entirely on sweepstakes winnings (e.g. dog food) for more than a year and who didn’t know he was on TV the whole time. Obviously, this happened in Japan.
Nick Bostrom’s new book — Superintelligence: Paths, Dangers, Strategies — was published today in the UK by Oxford University Press. It opens with a fable about some sparrows and an owl:
It was the nest-building season, but after days of long hard work, the sparrows sat in the evening glow, relaxing and chirping away.
“We are all so small and weak. Imagine how easy life would be if we had an owl who could help us build our nests!”
“Yes!” said another. “And we could use it to look after our elderly and our young.”
“It could give us advice and keep an eye out for the neighborhood cat,” added a third.
Then Pastus, the elder-bird, spoke: “Let us send out scouts in all directions and try to find an abandoned owlet somewhere, or maybe an egg. A crow chick might also do, or a baby weasel. This could be the best thing that ever happened to us, at least since the opening of the Pavilion of Unlimited Grain in yonder backyard.”
The flock was exhilarated, and sparrows everywhere started chirping at the top of their lungs.
Only Scronkfinkle, a one-eyed sparrow with a fretful temperament, was unconvinced of the wisdom of the endeavor. Quoth he: “This will surely be our undoing. Should we not give some thought to the art of owl-domestication and owl-taming first, before we bring such a creature into our midst?”
Replied Pastus: “Taming an owl sounds like an exceedingly difficult thing to do. It will be difficult enough to find an owl egg. So let us start there. After we have succeeded in raising an owl, then we can think about taking on this other challenge.”
“There is a flaw in that plan!” squeaked Scronkfinkle; but his protests were in vain as the flock had already lifted off to start implementing the directives set out by Pastus.
Just two or three sparrows remained behind. Together they began to try to work out how owls might be tamed or domesticated. They soon realized that Pastus had been right: this was an exceedingly difficult challenge, especially in the absence of an actual owl to practice on. Nevertheless they pressed on as best they could, constantly fearing that the flock might return with an owl egg before a solution to the control problem had been found.
It is not known how the story ends, but the author dedicates this book to Scronkfinkle and his followers.
(Last updated Feb. 11, 2015.)
What could an economics graduate student do to improve our strategic picture of superintelligence? What about a computer science professor? A policy analyst at RAND? A program director at IARPA?
In the last chapter of Superintelligence, Nick Bostrom writes:
We find ourselves in a thicket of strategic complexity and surrounded by a dense mist of uncertainty. Though many considerations have been discerned, their details and interrelationships remain unclear and iffy — and there might be other factors we have not thought of yet. How should we act in this predicament?
… Against a backdrop of perplexity and uncertainty, [strategic] analysis stands out as being of particularly high expected value. Illumination of our strategic situation would help us target subsequent interventions more effectively. Strategic analysis is especially needful when we are radically uncertain not just about some detail of some peripheral matter but about the cardinal qualities of the central things. For many key parameters, we are radically uncertain even about their sign…
The hunt for crucial considerations… will often require crisscrossing the boundaries between different academic disciplines and other fields of knowledge.
Bostrom does not, however, provide a list of specific research projects that could illuminate our strategic situation and thereby “help us target subsequent interventions more effectively.”
Below is my personal list of studies which could illuminate our strategic situation with regard to superintelligence. I’m hosting it on my personal site rather than MIRI’s blog to make it clear that this is not “MIRI’s official list of project ideas.” Other researchers at MIRI would, I’m sure, put together a different list. [Read more…]
I read Thiel’s Zero to One (2014) in May, but forgot to mention it in the May books post. I enjoyed it very much. His key argument is that progress comes from monopolies, not from strong competition, so we should encourage certain kinds of monopolies. I generally agree. I also agree with Thiel that technological progress has slowed since the 70s, with the (lone?) exception of IT.
The Info Mesa (2003), by Ed Regis, is fine but less interesting than Great Mambo Chicken (which I’m currently reading) and Nano (which I finished last month).
The Atomic Bazaar (2003), by William Langewiesche, tells the story of nuclear trafficking and the rise of poor countries with nuclear weapons programs, and especially the activities of Abdul Qadeer Khan. It was pretty good, though I wish it had done a better job of explaining the limits, opportunities, and incentives at play in the nuclear arms trade.
Age of Ambition (2014), by Evan Osnos, is a fantastically rich portrait of modern China. Highly recommended.
Human Accomplishment (2003), by Charles Murray, is a fine specimen of quantitative historical analysis. The final chapters are less persuasive than the rest of the book, but despite this terms like magisterial and tour de force come to mind. Murray does an excellent job walking the reader through his methodology, its pros and cons, the reasons for it, and the conclusions that can and can’t be drawn from it. You’ll probably like this if you enjoyed Pinker’s Better Angels of Our Nature.
Superintelligence (2014), by Nick Bostrom, is a fantastic summary of the last ~15 years of strategic thinking about machine superintelligence from (largely) FHI and MIRI, the two institutes focused most directly on the issue. If you want to get a sense of what’s been learned during that time, first read Bostrom’s 1997 paper on superintelligence (and other topics), and then read his new book. It comes out in the UK on July 3rd and in the USA on September 3rd. Highly recommended.
The Honest Truth about Dishonesty (2013), by Dan Ariely, is as fun and practical as the other Ariely books. Recommended.