- Reminder: Quantum computers still aren’t faster than regular old computers.
- In Science: the existence of the American continents may have been deduced by a Medieval scholar from Central Asia.
- Ben Kuhn: A guide to giving away money.
- Interview with Gwern Branwen on intelligence amplification.
- Steven Hawking on Last Week Tonight talking about intelligence explosion and other topics (very funny).
- “Ask audiences what they want, and they’ll tell you vegetables. Watch them quietly, and they’ll mostly eat candy.”
- Huh. So it turns out that Elon Musk is pretty concerned about AGI risk.
- Why Netflix never implemented the algorithm that won the Netflix $1 Million Challenge.
- A small study of which presentations of EA are most effective.
- “The number of new drugs approved per billion US dollars spent on R&D has halved roughly every 9 years since 1950… “
- How Google converted language translation into a problem of vector space mathematics.
- Philosopher Peter Unger has less respect for philosophy than I do.
What does MIRI’s research program study?
The most established term for this was coined by MIRI founder Eliezer Yudkowsky: “Friendly AI.” The term has some advantages, but it might suggest that MIRI is trying to build C-3PO, and it sounds a bit whimsical for a serious research program.
What about safe AGI or AGI safety? These terms are probably easier to interpret than Friendly AI. Also, people like being safe, and governments like saying they’re funding initiatives to keep the public safe.
A friend of mine worries that these terms could provoke a defensive response (in AI researchers) of “Oh, so you think me and everybody else in AI is working on unsafe AI?” But I’ve never actually heard that response to “AGI safety” in the wild, and AI safety researchers regularly discuss “software system safety” and “AI safety” and “agent safety” and more specific topics like “safe reinforcement learning” without provoking negative reactions from people doing regular AI research.
I’m more worried that a term like “safe AGI” could provoke a response of “So you’re trying to make sure that a system which is smarter than humans, and able to operate in arbitrary real-world environments, and able to invent new technologies to achieve its goals, will be safe? Let me save you some time and tell you right now that’s impossible. Your research program is a pipe dream.”
My reply goes something like “Yeah, it’s way beyond our current capabilities, but lots of things that once looked impossible are now feasible because people worked really hard on them for a long time, and we don’t think we can get the whole world to promise never to build AGI just because it’s hard to make safe, so we’re going to give AGI safety a solid try for a few decades and see what can be discovered.” But that’s probably not all that reassuring. [Read more…]
- Karnofsky, Sequence Thinking vs. Cluster Thinking and Modeling Extreme Model Uncertainty and a worked example.
- Robin Hanson argues for variable vigilance, Scott Alexander argues for constant vigilance.
- Pinker, Writing in the 21st Century.
- Timelapse video of a stellar explosion, recorded by Hubble over 4 years.
- Richardson & Robins, “A unification of the counterfactual and graphical approaches to causality.”
- A computer program has passed (an easy version of) the Turing test. It doesn’t mean much.
- My favorite net neutrality rant yet (John Oliver).
- Do rationalists exist?
From Murray’s Human Accomplishment:
The problem with the standard archaeological account of human accomplishment from [the ancient world] is not that the picture is incomplete (which is inevitable), but that the data available to us leave so many puzzles.
The Antikythera Mechanism is a case in point… The Antikythera Mechanism is a bronze device about the size of a brick. It was recovered in 1901 from the wreck of a trading vessel that had sunk near the southern tip of Greece sometime around –65. Upon examination, archaeologists were startled to discover imprints of gears in the corroded metal. So began a half-century of speculation about what purpose the device might have served.
Finally, in 1959, science historian Derek de Solla Price figured it out: the Antikythera Mechanism was a mechanical device for calculating the positions of the sun and moon. A few years later, improvements in archaeological technology led to gamma radiographs of the Mechanism, revealing 22 gears in four layers, capable of simulating several major solar and lunar cycles, including the 19-year Metonic cycle that brings the phases of the moon back to the same calendar date. What made this latter feat especially astonishing was not just that the Mechanism could reproduce the 235 lunations in the Metonic cycle, but that it used a differential gear to do so. Until then, it was thought that the differential gear had been invented in 1575.
See also Wikipedia.
From Stanovich’s popular book on the distinction between rationality and intelligence (p. 196):
In order to illustrate the oddly dysfunctional ways that rationality is devalued in comparison to intelligence… Baron asks us to imagine what would happen if we were able to give everyone an otherwise harmless drug that increased their algorithmic-level cognitive capacities (for example, discrimination speed, working memory capacity, decoupling ability) — in short, that increased their intelligence…
Imagine that everyone in North America took the pill before retiring and then woke up the next morning with more memory capacity and processing speed. Both Baron and I believe that there is little likelihood that much would change the next day in terms of human happiness. It is very unlikely that people would be better able to fulfill their wishes and desires the day after taking the pill. In fact, it is quite likely that people would simply go about their usual business-only more efficiently. If given more memory capacity and processing speed, people would, I believe: carry on using the same ineffective medical treatments because of failure to think of alternative causes (Chapter 10); keep making the same poor financial decisions because of overconfidence (Chapter 8); keep misjudging environmental risks because of vividness (Chapter 6); play host to the contaminated mindware of Ponzi and pyramid schemes (Chapter 11); be wrongly influenced in their jury decisions by incorrect testimony about probabilities (Chapter 10); and continue making many other of the suboptimal decisions described in earlier chapters. The only difference would be that they would be able to do all of these things much more quickly!
Elsewhere (p. 171), Stanovich notes:
Mensa is a club restricted to high-IQ individuals, and one must pass IQ-type tests to be admitted. Yet 44 percent of the members of this club believed in astrology, 51 percent believed in biorhythms, and 56 percent believed in the existence of extraterrestrial visitors-all beliefs for which there is not a shred of evidence.
- Stephen Deangelis at Wired: “Machine Learning: Bane or Blessing for Mankind?”
- Dominic Cummings, policy advisor to the UK education secretary, releases a somewhat amazing 237-page document on education priorities.
- MacKenzie, “Computer-related accidental death: an empirical exploration” (1994).
- Wikipedia’s Outline of Science.
- New Yudkowsky FAI report: “Distributions allowing tiling of staged subjective EU maximizers.”
Re-reading Ross Andersen’s piece on Nick Bostrom and FHI for Aeon magazine, I was struck by several nicely succinct explanations given by FHI researchers — ones which I’ll borrowing for my own conversations with people about these topics:
“There is a concern that civilisations might need a certain amount of easily accessible energy to ramp up,” Bostrom told me. “By racing through Earth’s hydrocarbons, we might be depleting our planet’s civilisation startup-kit. But, even if it took us 100,000 years to bounce back, that would be a brief pause on cosmic time scales.”
“Human brains are really good at the kinds of cognition you need to run around the savannah throwing spears,” Dewey told me. “But we’re terrible at [many other things]… Think about how long it took humans to arrive at the idea of natural selection. The ancient Greeks had everything they needed to figure it out. They had heritability, limited resources, reproduction and death. But it took thousands of years for someone to put it together. If you had a machine that was designed specifically to make inferences about the world, instead of a machine like the human brain, you could make discoveries like that much faster.”
“The difference in intelligence between humans and chimpanzees is tiny,” [Armstrong] said. “But in that difference lies the contrast between 7 billion inhabitants and a permanent place on the endangered species list. That tells us it’s possible for a relatively small intelligence advantage to quickly compound and become decisive.”
“The basic problem is that the strong realisation of most motivations is incompatible with human existence,” Dewey told me. “An AI might want to do certain things with matter in order to achieve a goal, things like building giant computers, or other large-scale engineering projects. Those things might involve intermediary steps, like tearing apart the Earth to make huge solar panels. A superintelligence might not take our interests into consideration in those situations, just like we don’t take root systems or ant colonies into account when we go to construct a building.”
[Bostrom] told me that when he was younger, he was more interested in the traditional philosophical questions… “But then there was this transition, where it gradually dawned on me that not all philosophical questions are equally urgent,” he said. “Some of them have been with us for thousands of years. It’s unlikely that we are going to make serious progress on them in the next ten. That realisation refocused me on research that can make a difference right now. It helped me to understand that philosophy has a time limit.”
- Seán Ó hÉigeartaigh (FHI & CSER) gave a 13-minute introduction to existential risk.
- The World Economic Forum’s Global Risks 2014 report provides a snapshot of what a certain set of thought leaders think is worth worrying about.
- Another snapshot comes from the responses to Edge.org’s 2013 annual question.
- The Onion: World’s Supercomputers Release Study Confirming They Are Not Powerful Enough.
- Frustrated scholar creates new way to fund, publish and peer-review academic work. Dogecoin accepted.
- James Hamblin in The Atlantic: But what would the end of humanity mean for me?
- Nick Beckstead’s suggestions for effective altruism research topics.
- Schwitzgebel, A Theory of Jerks.
- A Type House Divided: inside the split of the Hoefler/Frere-Jones typography team.
- UsefulScience.org: Life-relevant scientific results explained in one sentence, in many categories, with links to the original publication and also news coverage. (This link, like many of my links, is borrowed from Rockstar Research Magazine.)
After Drexler published his 1981 nanotech paper in PNAS, and after it received some positive followups in Nature and in Science in 1983, why did Drexler next write a popular book like Engines of Creation (1986) instead of a technical account like Nanosystems (1992)? Ed Regis writes in Nano (p. 118):
The logical next step for Drexler… was to produce a full-blown account of his molecular-engineering scheme, a technical document that fleshed out the whole story in chapter and verse, with all the technical details. That was the obvious thing to do, anyway, if he wanted to convince the greater science and engineering world that molecular engineering was a real prospect and not just his own private fantasy.
… Drexler instead did something else, spending the next four years, essentially, writing a popular account of the subject in his book, Engines of Creation.
For a dyed-in-the-wool engineer such as himself, this was somewhat puzzling. Why go public with a scheme as wild and woolly as this one before the technical details were even passably well worked out? Why paint vivid word pictures of “the coming era of nanotechnology” before even so much as one paltry designer protein had been coaxed, tricked, or forced into existence? Why not nail down an ironclad scientific case for the whole thing first, and only then proceed to advertise its benefits?
Of course, there were answers. For one thing, Drexler was convinced that he’d already done enough in his PNAS piece to motivate a full course of research-and-development work in academia and industry. After all, he’d described what was possible at the molecular level and by what means, and he’d said what some of the benefits were. How could a bunch of forward-looking researchers, seeing all this, not go ahead and actually do it?…
The other reason for writing a popular book on the subject was to raise some of the economic and social issues involved. Scientists and engineers, it was commonly observed, did not have an especially good track record when it came to assessing the wider impact of what they’d wrought in the lab. Their attitude seemed to be: “We invent it, you figure out what to do with it.”
To Drexler, that was the height of social irresponsibility, particularly where nanotechnology was concerned, because its impacts would be so broad and sweeping…
If anything was clear to Eric Drexler, it was that if the human race was to survive the transition to the nanotech era, it would have to do a bit of thinking beforehand. He’d have to write the book on this because, all too obviously, nobody else was about to.
But there was yet a third reason for writing Engines of Creation, a reason that was, for Drexler, probably the strongest one of all. This was to announce to the world at large that the issue of “limits” [from Limits to Growth] had been addressed head-on…
Laurie Garrett’s Foreign Affairs piece on synbio from a while back exaggerates the state of current progress, but it also contains some good commentary on the difficulty of containing hazardous materials when those hazardous materials — unlike the case of nuclear fissile materials — are essentially information:
Fouchier and Kawaoka drew the wrath of many national security and public health experts, who demanded to know how the deliberate creation of potential pandemic flu strains could possibly be justified… the National Science Advisory Board for Biosecurity… [ordered] that the methods used to create these new mammalian forms of H5N1 never be published. “It’s not clear that these particular [experiments] have created something that would destroy the world; maybe it’ll be the next set of experiments that will be critical,” [Paul] Keim told reporters. “And that’s what the world discussion needs to be about.”
In the end, however, the December 2011 do-not-publish decision… was reversed… [and] both papers were published in their entirety by Science and Nature in 2012, and [the] temporary moratorium on dual-use research on influenza viruses was eventually lifted… Osterholm, Keim, and most of the vocal opponents of the work retreated, allowing the advisory board to step back into obscurity.
… What stymies the very few national security and law enforcement experts closely following this biological revolution is the realization that the key component is simply information. While virtually all current laws in this field, both local and global, restrict and track organisms of concern (such as, say, the Ebola virus), tracking information is all but impossible. Code can be buried anywhere — al Qaeda operatives have hidden attack instructions inside porn videos, and a seemingly innocent tweet could direct readers to an obscure Internet location containing genomic code ready to be downloaded to a 3-D printer. Suddenly, what started as a biology problem has become a matter of information security.
See also Bostrom, “Information Hazards” (2011).
Somehow MIRI’s mission comes in at #10 on this list of 10 responses to the technological unemployment problem.
I suppose technically, Friendly AI is a solution for all the things.
This reminds me of the first draft of MIRI’s environmental policy, which read:
[MIRI] exists to ensure that the creation of smarter than human intelligence benefits society. Because societies depend on their environment to thrive, one implication of our core mission is a drive to ensure that when advanced intelligence technologies become available, they are used to secure the continued viability and resilience of the environment.
Many advanced artificial intelligences (AIs) will have instrumental goals to capture as many resources as possible for their own use, because resources are useful for a broad range of possible AI goals. To ensure that Earth’s resources are used wisely despite the creation of advanced AIs, we must discover how to design these AIs so that they can be given final goals which accord with humane values.
Though poorly designed AIs may pose a risk to the resources and environment on which humanity depends, more carefully designed AIs may be our best solution to long-term environmental concerns. To whatever extent we have goals for environmental sustainability, they are goals that can be accomplished to greater degrees using sufficiently advanced intelligence.
To prevent environmental disasters caused by poorly designed AIs, and to ensure that we one day have the intelligence needed to solve our current environmental dilemmas, [MIRI] is committed to discovering the principles of safe, beneficial AI that will one day allow us all to safeguard our environment as well as our future.
In the end, though, we decided to go with a more conventional (super-boring) environmental policy, available here.
- MIRI wants to fund your independently-organized Friendly AI workshop.
- New DoD report on their plans for autonomous robots, 2013-2038.
- Watch a computer try to learn how to walk (video).
- What a Symposium on Theory of Computing (STOC) might have been like in Medieval times.
- Eliezer Yudkowsky takes another shot (Nov. 2013) at explaining what he was trying to get across in his metaethics sequence.
- The ontological argument for God’s existence, formalized and proved with PVS.
- Ants acting as both a solid and a liquid (video).
- Wikiquote: incorrect predictions.
- Average length of PhD dissertations by major.
My sports coaches always emphasized “the fundamentals.” For example at basketball practice they spent no time whatsoever teaching us “advanced” moves like behind-the-back passes and alley-oops. They knew that even if advanced moves were memorable, and could allow the team to score 5-15 extra points per game, this effect would be dominated by whether we made our free throws, grabbed our rebounds, and kept our turnovers to a minimum.
When I began my internship at what was then called SIAI, I thought, “Wow. SIAI has implemented few business/non-profit fundamentals, and is surviving almost entirely via advanced moves.” So, Louie Helm and I spent much of our first two years at MIRI mastering the (kinda boring) fundamentals, and my impression is that doing so paid off handsomely in organizational robustness and productivity.
On Less Wrong, some kinds of “advanced moves” are sometimes called “Munchkin ideas”:
A Munchkin is the sort of person who, faced with a role-playing game, reads through the rulebooks over and over until he finds a way to combine three innocuous-seeming magical items into a cycle of infinite wish spells. Or who, in real life, composes a surprisingly effective diet out of drinking a quarter-cup of extra-light olive oil at least one hour before and after tasting anything else. Or combines liquid nitrogen and antifreeze and life-insurance policies into a ridiculously cheap method of defeating the invincible specter of unavoidable Death.
Munchkin ideas are more valuable in life than advanced moves are in a basketball game because the upsides in life are much greater. The outcome of a basketball game is binary (win/lose), and advanced moves can’t increase your odds of winning by that much. But in life in general, a good Munchkin idea might find your life partner or make you a billion dollars or maybe even optimize literally everything.
But Munchkin ideas work best when you’ve mastered the fundamentals first. Behind-the-back passes won’t save you if you make lots of turnovers due to poor dribbling skills. Your innovative startup idea won’t do you much good if you sign unusual contracts that make your startup grossly unattractive to investors. And a Munchkin-ish nonprofit can only grow so much without bookkeeping, financial controls, and a donor database.
My guess is that when you’re launching a new startup or organization, the fundamentals can wait. “Do things that don’t scale,” as Paul Graham says. But after you’ve got some momentum then yes, get your shit together, master the fundamentals, and do things in ways that can scale.
This advice is audience-specific. To an audience of Protestant Midwesterners, I would emphasize the importance of Munchkinism. To my actual audience of high-IQ entrepreneurial world-changers, who want to signal their intelligence and Munchkinism to each other, I say “Don’t neglect the fundamentals.” Executing the fundamentals competently doesn’t particularly signal high intelligence, but it’s worth doing anyway.
From The Limits of Safety (p. 1):
On the night of October 25, 1962, an air force sentry was patrolling the perimeter of a military base near Duluth, Minnesota. It was the height of the Cuban missile crisis, and nuclear-armed bombers and interceptor aircraft, parked on air base runways and at commercial airports throughout the United States, were alert and ready for war. The sentry spotted someone climbing the fence, shot at the figure, and sounded the sabotage alarm. At airfields throughout the region, alarms went off, and armed guards rushed into the cold night to prevent Soviet agents from sabotaging U.S. nuclear forces.
At Volk Field in Wisconsin, however, the wrong alarm bell rang: the Klaxon signalling that nuclear had begun went off. Pilots ran to their nuclear-armed interceptors and started the engines. These men had been told that there would be no practice alert drills during the tense crisis, and they fully believed that a nuclear war was starting as they headed down the runway. Fortunately, the base commander contacted Duluth before the planes took off and discovered what had happened. An officer in the command post immediately drove his car onto the runway, flashing his lights and signaling the interceptors. The pilots saw him and stopped their aircraft. The suspected Soviet saboteur that caused the whole incident was, ironically, a bear.
Jon Ronson’s The Psycopath Test (2011) opens with the strange story of Being or Nothingness:
Last July, Deborah received a strange package in the mail… The package contained a book. It was only forty-two pages long, twenty-one of which—every other page—were completely blank, but everything about it—the paper, the illustrations, the typeface—looked very expensively produced. The cover was a delicate, eerie picture of two disembodied hands drawing each other. Deborah recognized it to be a reproduction of M. C. Escher’s Drawing Hands.
The author was a “Joe K” (a reference to Kafka’s Josef K., maybe, or an anagram of “joke”?) and the title was Being or Nothingness, which was some kind of allusion to Sartre’s 1943 essay, Being and Nothingness. Someone had carefully cut out with scissors the page that would have listed the publishing and copyright details, the ISBN, etc., so there were no clues there. A sticker read: Warning! Please study the letter to Professor Hofstadter before you read the book. Good Luck!
Deborah leafed through it. It was obviously some kind of puzzle waiting to be solved, with cryptic verse and pages where words had been cut out, and so on.
Everyone at MIRI was pretty amused when a copy of Being or Nothingness arrived at our offices last year, addressed to Eliezer.
Everyone except Eliezer, anyway. He just rolled his eyes and said, “Do what you want with it; I’ve been getting crazy stuff like that for years.”