Why Engines before Nanosystems?

After Drexler published his 1981 nanotech paper in PNAS, and after it received some positive followups in Nature and in Science in 1983, why did Drexler next write a popular book like Engines of Creation (1986) instead of a technical account like Nanosystems (1992)? Ed Regis writes in Nano (p. 118):

The logical next step for Drexler… was to produce a full-blown account of his molecular-engineering scheme, a technical document that fleshed out the whole story in chapter and verse, with all the technical details. That was the obvious thing to do, anyway, if he wanted to convince the greater science and engineering world that molecular engineering was a real prospect and not just his own private fantasy.

… Drexler instead did something else, spending the next four years, essentially, writing a popular account of the subject in his book, Engines of Creation.

For a dyed-in-the-wool engineer such as himself, this was somewhat puzzling. Why go public with a scheme as wild and woolly as this one before the technical details were even passably well worked out? Why paint vivid word pictures of “the coming era of nanotechnology” before even so much as one paltry designer protein had been coaxed, tricked, or forced into existence? Why not nail down an ironclad scientific case for the whole thing first, and only then proceed to advertise its benefits?

Of course, there were answers. For one thing, Drexler was convinced that he’d already done enough in his PNAS piece to motivate a full course of research-and-development work in academia and industry. After all, he’d described what was possible at the molecular level and by what means, and he’d said what some of the benefits were. How could a bunch of forward-looking researchers, seeing all this, not go ahead and actually do it?…

The other reason for writing a popular book on the subject was to raise some of the economic and social issues involved. Scientists and engineers, it was commonly observed, did not have an especially good track record when it came to assessing the wider impact of what they’d wrought in the lab. Their attitude seemed to be: “We invent it, you figure out what to do with it.”

To Drexler, that was the height of social irresponsibility, particularly where nanotechnology was concerned, because its impacts would be so broad and sweeping…

If anything was clear to Eric Drexler, it was that if the human race was to survive the transition to the nanotech era, it would have to do a bit of thinking beforehand. He’d have to write the book on this because, all too obviously, nobody else was about to.

But there was yet a third reason for writing Engines of Creation, a reason that was, for Drexler, probably the strongest one of all. This was to announce to the world at large that the issue of “limits” [from Limits to Growth] had been addressed head-on…

It’s hard to contain information hazards

Laurie Garrett’s Foreign Affairs piece on synbio from a while back exaggerates the state of current progress, but it also contains some good commentary on the difficulty of containing hazardous materials when those hazardous materials — unlike the case of nuclear fissile materials — are essentially information:

Fouchier and Kawaoka drew the wrath of many national security and public health experts, who demanded to know how the deliberate creation of potential pandemic flu strains could possibly be justified… the National Science Advisory Board for Biosecurity… [ordered] that the methods used to create these new mammalian forms of H5N1 never be published. “It’s not clear that these particular [experiments] have created something that would destroy the world; maybe it’ll be the next set of experiments that will be critical,” [Paul] Keim told reporters. “And that’s what the world discussion needs to be about.”

In the end, however, the December 2011 do-not-publish decision… was reversed… [and] both papers were published in their entirety by Science and Nature in 2012, and [the] temporary moratorium on dual-use research on influenza viruses was eventually lifted… Osterholm, Keim, and most of the vocal opponents of the work retreated, allowing the advisory board to step back into obscurity.

… What stymies the very few national security and law enforcement experts closely following this biological revolution is the realization that the key component is simply information. While virtually all current laws in this field, both local and global, restrict and track organisms of concern (such as, say, the Ebola virus), tracking information is all but impossible. Code can be buried anywhere — al Qaeda operatives have hidden attack instructions inside porn videos, and a seemingly innocent tweet could direct readers to an obscure Internet location containing genomic code ready to be downloaded to a 3-D printer. Suddenly, what started as a biology problem has become a matter of information security.

See also Bostrom, “Information Hazards” (2011).

MIRI’s original environmental policy

Somehow MIRI’s mission comes in at #10 on this list of 10 responses to the technological unemployment problem.

I suppose technically, Friendly AI is a solution for all the things. :)

This reminds me of the first draft of MIRI’s environmental policy, which read:

[MIRI] exists to ensure that the creation of smarter than human intelligence benefits society. Because societies depend on their environment to thrive, one implication of our core mission is a drive to ensure that when advanced intelligence technologies become available, they are used to secure the continued viability and resilience of the environment.

Many advanced artificial intelligences (AIs) will have instrumental goals to capture as many resources as possible for their own use, because resources are useful for a broad range of possible AI goals. To ensure that Earth’s resources are used wisely despite the creation of advanced AIs, we must discover how to design these AIs so that they can be given final goals which accord with humane values.

Though poorly designed AIs may pose a risk to the resources and environment on which humanity depends, more carefully designed AIs may be our best solution to long-term environmental concerns. To whatever extent we have goals for environmental sustainability, they are goals that can be accomplished to greater degrees using sufficiently advanced intelligence.

To prevent environmental disasters caused by poorly designed AIs, and to ensure that we one day have the intelligence needed to solve our current environmental dilemmas, [MIRI] is committed to discovering the principles of safe, beneficial AI that will one day allow us all to safeguard our environment as well as our future.

In the end, though, we decided to go with a more conventional (super-boring) environmental policy, available here.

Assorted links

Don’t neglect the fundamentals

My sports coaches always emphasized “the fundamentals.” For example at basketball practice they spent no time whatsoever teaching us “advanced” moves like behind-the-back passes and alley-oops. They knew that even if advanced moves were memorable, and could allow the team to score 5-15 extra points per game, this effect would be dominated by whether we made our free throws, grabbed our rebounds, and kept our turnovers to a minimum.

When I began my internship at what was then called SIAI, I thought, “Wow. SIAI has implemented few business/non-profit fundamentals, and is surviving almost entirely via advanced moves.” So, Louie Helm and I spent much of our first two years at MIRI mastering the (kinda boring) fundamentals, and my impression is that doing so paid off handsomely in organizational robustness and productivity.

On Less Wrong, some kinds of “advanced moves” are sometimes called “Munchkin ideas”:

A Munchkin is the sort of person who, faced with a role-playing game, reads through the rulebooks over and over until he finds a way to combine three innocuous-seeming magical items into a cycle of infinite wish spells. Or who, in real life, composes a surprisingly effective diet out of drinking a quarter-cup of extra-light olive oil at least one hour before and after tasting anything else. Or combines liquid nitrogen and antifreeze and life-insurance policies into a ridiculously cheap method of defeating the invincible specter of unavoidable Death.

Munchkin ideas are more valuable in life than advanced moves are in a basketball game because the upsides in life are much greater. The outcome of a basketball game is binary (win/lose), and advanced moves can’t increase your odds of winning by that much. But in life in general, a good Munchkin idea might find your life partner or make you a billion dollars or maybe even optimize literally everything.

But Munchkin ideas work best when you’ve mastered the fundamentals first. Behind-the-back passes won’t save you if you make lots of turnovers due to poor dribbling skills. Your innovative startup idea won’t do you much good if you sign unusual contracts that make your startup grossly unattractive to investors. And a Munchkin-ish nonprofit can only grow so much without bookkeeping, financial controls, and a donor database.

My guess is that when you’re launching a new startup or organization, the fundamentals can wait. “Do things that don’t scale,” as Paul Graham says. But after you’ve got some momentum then yes, get your shit together, master the fundamentals, and do things in ways that can scale.

This advice is audience-specific. To an audience of Protestant Midwesterners, I would emphasize the importance of Munchkinism. To my actual audience of high-IQ entrepreneurial world-changers, who want to signal their intelligence and Munchkinism to each other, I say “Don’t neglect the fundamentals.” Executing the fundamentals competently doesn’t particularly signal high intelligence, but it’s worth doing anyway.

Another Cold War close call

From The Limits of Safety (p. 1):

On the night of October 25, 1962, an air force sentry was patrolling the perimeter of a military base near Duluth, Minnesota. It was the height of the Cuban missile crisis, and nuclear-armed bombers and interceptor aircraft, parked on air base runways and at commercial airports throughout the United States, were alert and ready for war. The sentry spotted someone climbing the fence, shot at the figure, and sounded the sabotage alarm. At airfields throughout the region, alarms went off, and armed guards rushed into the cold night to prevent Soviet agents from sabotaging U.S. nuclear forces.

At Volk Field in Wisconsin, however, the wrong alarm bell rang: the Klaxon signalling that nuclear had begun went off. Pilots ran to their nuclear-armed interceptors and started the engines. These men had been told that there would be no practice alert drills during the tense crisis, and they fully believed that a nuclear war was starting as they headed down the runway. Fortunately, the base commander contacted Duluth before the planes took off and discovered what had happened. An officer in the command post immediately drove his car onto the runway, flashing his lights and signaling the interceptors. The pilots saw him and stopped their aircraft. The suspected Soviet saboteur that caused the whole incident was, ironically, a bear.


The Riddle of Being or Nothingness

being or nothingnessJon Ronson’s The Psycopath Test (2011) opens with the strange story of Being or Nothingness:

Last July, Deborah received a strange package in the mail…  The package contained a book. It was only forty-two pages long, twenty-one of which—every other page—were completely blank, but everything about it—the paper, the illustrations, the typeface—looked very expensively produced. The cover was a delicate, eerie picture of two disembodied hands drawing each other. Deborah recognized it to be a reproduction of M. C. Escher’s Drawing Hands.

The author was a “Joe K” (a reference to Kafka’s Josef K., maybe, or an anagram of “joke”?) and the title was Being or Nothingness, which was some kind of allusion to Sartre’s 1943 essay, Being and Nothingness. Someone had carefully cut out with scissors the page that would have listed the publishing and copyright details, the ISBN, etc., so there were no clues there. A sticker read: Warning! Please study the letter to Professor Hofstadter before you read the book. Good Luck!

Deborah leafed through it. It was obviously some kind of puzzle waiting to be solved, with cryptic verse and pages where words had been cut out, and so on.

Everyone at MIRI was pretty amused when a copy of Being or Nothingness arrived at our offices last year, addressed to Eliezer.

Everyone except Eliezer, anyway. He just rolled his eyes and said, “Do what you want with it; I’ve been getting crazy stuff like that for years.”

[Read more…]

Books I finished reading in May 2014

Nano (1996), by Ed Regis, tells the story of nanotechnology up to 1995, and serves as a cautionary tale for others trying to promote the development of a novel science from outside the establishment. Recommended.

The Visioneers (2012), by W. Patrick McCray, tells the story of both Gerard O’Neill — who advocated space colonization in the 1970s — and Eric Drexler, pioneer of nanotechnology. It’s less detailed than Nano, but also recommended.

Soldiers of Reason (2009), by Alex Abella, is a history of RAND Corporation. Most of it is pretty interesting, especially if you happen to run a nonprofit research institute, though I disagree with the author about the purpose and value of rational choice theory.

David and Goliath (2013), by Malcolm Gladwell, includes some great stories as usual but is also Gladwell’s most annoying, disingenuous book yet.

Think Like a Freak (2014), by Levitt & Dubner, also includes some great stories, and is less annoying than David and Goliath, but is basically a repackaging of posts from their blog.

A Troublesome Inheritance (2014), by Nicholas Wade, is about race, genes, and IQ, a touchy subject!

The Knowledge (2014), by Lewis Dartnell, is about what we’d need to know, and what knowledge we’d need to preserve, to reboot human civilization as quickly as possible after some kind of apocalypse. I wish there was more available on this subject. Recommended.

The Up Side of Down (2014), by Megan McArdle, is about how failing well is the key to success. I’ll echo Robin Hanson: “Overall I found it very hard to disagree with anything that McArdle said in her book. If you know me, that is quite some praise.”

Two innovative strategies in sports

From Gladwell’s David and Goliath:

A regulation basketball court is ninety-four feet long. Most of the time, a team would defend only about twenty-four feet of that, conceding the other seventy feet. Occasionally teams played a full-court press—that is, they contested their opponent’s attempt to advance the ball up the court. But they did it for only a few minutes at a time. It was as if there were a kind of conspiracy in the basketball world about the way the game ought to be played, Ranadivé thought, and that conspiracy had the effect of widening the gap between good teams and weak teams. Good teams, after all, had players who were tall and could dribble and shoot well; they could crisply execute their carefully prepared plays in their opponent’s end. Why, then, did weak teams play in a way that made it easy for good teams to do the very things that they were so good at?

Ranadivé looked at his girls. Morgan and Julia were serious basketball players. But Nicky, Angela, Dani, Holly, Annika, and his own daughter, Anjali, had never played the game before. They weren’t all that tall. They couldn’t shoot. They weren’t particularly adept at dribbling. They were not the sort who played pickup games at the playground every evening. Ranadivé lives in Menlo Park, in the heart of California’s Silicon Valley. His team was made up of, as Ranadivé put it, “little blond girls.” These were the daughters of nerds and computer programmers. They worked on science projects and read long and complicated books and dreamed about growing up to be marine biologists. Ranadivé knew that if they played the conventional way—if they let their opponents dribble the ball up the court without opposition—they would almost certainly lose to the girls for whom basketball was a passion. Ranadivé had come to America as a seventeen-year-old with fifty dollars in his pocket. He was not one to accept losing easily. His second principle, then, was that his team would play a real full-court press—every game, all the time. The team ended up at the national championships. “It was really random,” Anjali Ranadivé said. “I mean, my father had never played basketball before.”

From Brafman’s Sway:

The most flattering way to describe the Gator [football] team upon Spurrier’s arrival in 1990 was as a “fixer-upper.” The team had never won a conference title; in fact, it was on probation because of allegations of rule violations by the team’s former coach.

… Spurrier’s most important move was to identify a weak spot in the strategy employed by his opponents. For year the teams in the conference had adhered to a “war of attrition” game strategy: they called conservative plays and held on to the ball for as long as they could, hoping to win a defensive battle…

… Spurrier came to dominate the conference by… introducing what he called the ‘Fun-n-Gun” approach…

Spurrier mixed things up with a generous helping of “big chance plays, where you got to give your players a shot.” In other words, Spurrier’s team passed more often, played more aggressively, and tried to score more touchdowns.

..Spurrier gained an advantage because the other coaches were focused on trying to avoid a potential loss. Think of what it’s like to be a college football coach. As you walk around town, passing fans offer themselves up as instant experts on the game — never afraid to give you a piece of their minds on what you did wrong in yesterday’s match-up. You make one bad move and you get skewered by fans and commentators alike. Meanwhile, ticcket sale revenues, your school’s alumni fundraising, and your job all depend heavily on the football team’s success. All of that pressure adds up… the losses loom large…

You’d have thought that after losing a few games to a team like [the Gators]… the [other] coaches would have reevaluated their war-of-attrition model. But they didn’t. And so Spurrier and his Gators continued to dominate former powerhouses like Alabama, Tennessee, and Auburn. Over the next six years, the coach and his team went on to win four division titles, culminating in the national championship.

Assorted links

An onion strategy for AGI discussion

The stabilization of environments” is a paper about AIs that reshape their environments to make it easier to achieve their goals. This is typically called enforcement, but they prefer the term stabilization because it “sounds less hostile.”

“I’ll open the pod bay doors, Dave, but then I’m going to stabilize the ship… ”

Sparrow (2013) takes the opposite approach to plain vs. dramatic language. Rather than using a modest term like iterated embryo selection, Sparrow prefers the phrase in vitro eugenics. Jeepers.

I suppose that’s more likely to provoke public discussion, but…  will much good will come of that public discussion? The public had a needless freak-out about in vitro fertilization back in the 60s and 70s and then, as soon as the first IVF baby was born in 1978, decided they were in favor of it.

Someone recently suggested I use an “onion strategy” for the discussion of novel technological risks. The outermost layer of the communication onion would be aimed at the general public, and focus on benefits rather than risks, so as not to provoke an unproductive panic. A second layer for a specialist audience could include a more detailed elaboration of the risks. The most complete discussion of risks and mitigation options would be reserved for technical publications that are read only by professionals.

Eric Drexler seems to wish he had more successfully used an onion strategy when writing about nanotechnology. Engines of Creation included frank discussions of both the benefits and risks of nanotechnology, including the “grey goo” scenario that was discussed widely in the media and used as the premise for the bestselling novel Prey.

Ray Kurzweil may be using an onion strategy, or at least keeping his writing in the outermost layer. If you look carefully, chapter 8 of The Singularity is Near takes technological risks pretty seriously, and yet it’s written in such a way that most people who read the book seem to come away with an overwhelmingly optimistic perspective on technological change.

George Church may be following an onion strategy. Regenesis also contains a chapter on the risks of advanced bioengineering, but it’s presented as an “epilogue” that many readers will skip.

Perhaps those of us writing about AGI for the general public should try to discuss:

  • astronomical stakes rather than existential risk
  • Friendly AI rather than AGI risk or the superintelligence control problem
  • the orthogonality thesis and convergent instrumental values and complexity of values rather than “doom by default”
  • etc.

MIRI doesn’t have any official recommendations on the matter, but these days I find myself leaning toward an onion strategy.

Older assorted links

Immediately before launching this new site I was posting regular assorted links to Facebook. I’ve collected those links below.


May 31st links

May 29th links

May 27th links

[Read more…]

Feynman on dealing with nanotechnology risks

Nano (p. 113) quotes Eric Drexler describing the time he first met Richard Feynman at a party:

We talked about the PNAS article [on nanotechnology], and generally he indicated that, yeah, this was a sensible thing… at one point I was talking about the need for institutions to handle some of the problems [nanotechnology] raised, and [Feynman] remarked [that] institutions were made up of people and therefore of fools.

Feynman sounds downright Yudkowskian on this point, if you ask me. :)