MarginNote: the only iPhone app that lets you annotate both PDFs and epub files

As far as I can tell, MarginNote is the only iPhone app that lets you annotate & highlight both PDFs and epub files, and sync those annotations to your computer. And by “PDFs and epub files” I basically mean “all text files,” since Calibre and other apps can convert any text file into an epub, except for PDFs with tables and images. (The Kindle iPhone app can annotate text files, but can’t sync those annotations anywhere unless you bought the text directly from Amazon.)

This is important for people who like to read nonfiction “on the go,” like me — and plausibly some of my readers, so I figured I’d share my discovery.

The most-acclaimed new classical music of 2015

Barney Sherman’s 2015 classical music mega-meta-list is now up (part 1part 2), pulling together the results of 64 different “best of the 2015” lists from classical music critics.

Most of the selections are new performances of older works. Here, I want to highlight the contemporary classical pieces that were recorded for the first time in 2015 (and usually composed in the last few years), and that were included in 6 or more of the lists in Sherman’s analysis:

  1. Anna Thorvaldsdottir: In the Light of Air
  2. Andrew Norman: Play / Try
  3. Julia Wolfe: Anthracite Fields
  4. Various composers: Render
  5. Various composers: Clockworking
  6. John Luther Adams: The Wind in High Places
  7. John Adams: Absolute Jest

The links go to Spotify. If you’re relatively new to contemporary classical music, my guess is that Absolute Jest is the most widely accessible selection here, followed by Render.

When will videogame writing improve?

The best plays and films have had great writing for a long time. The best TV shows have had great writing for about a decade now. But the writing in the best videogames is still cringe-inducingly awful. This is despite the fact that videogame blockbusters regularly have production budgets of $50M or more. When will videogames hit their “golden age” (at least, for writing)?

My favorite kind of music

I think I’ve finally realized that I have a favorite kind of music, though unfortunately it doesn’t have a genre name, and it cuts across many major musical traditions — Western classical, jazz, rock, electronica, and possibly others.

I tend to love music that:

  1. Is primarily tonal but uses dissonance for effective contrast. (The Beatles are too tonal; Arnold Schoenberg and Cecil Taylor are too atonal; Igor Stravinsky and Charles Mingus are just right.)
  2. Obsessively composed, though potentially with substantial improvisation within the obsessively composed structure. (Coleman’s Free Jazz is too free. Amogh Symphony’s Vectorscan is innovative and complex but doesn’t sound like they tried very hard to get the compositional details right. The Rite of Spring and Chiastic Slide and even Karma are great.)
  3. Tries to be as emotionally affecting as possible, though this may include passages of contrastingly less-emotional music. (Anthony Braxton and Brian Ferneyhough are too cold and anti-emotional. Rich Woodson shifts around too quickly to ever build up much emotional “momentum.” Master of Puppets and Escalator Over the Hill and Tabula Rasa are great.)
  4. Is boredom-resistant by being fairly complex or by being long and subtly-evolving enough that I don’t get bored of it quickly. (The Beatles are too short and simple — yes, including their later work. The Soft Machine is satisfyingly complex and varied. The minimalists and Godspeed! You Black Emperor are often simple and repetitive, but their pieces are long enough and subtly-evolving enough that I don’t get bored of them.)

Property #2, I should mention, is pretty similar to Holden Karnofsky’s notion of “awe-inspiring” music. Via email, he explained:

One of the emotions I would like to experience is awe … A piece of music might be great because the artists got lucky and captured a moment, or because it’s just so insane that I can’t find anything else like it, or because I have an understanding that it was the first thing ever to do X, or because it just has that one weird sound that is so cool, but none of those make me go “Wow, this artist is awesome. I am in awe of them. I feel like the best parts of this are things they did on purpose, by thinking of them, by a combination of intelligence and sweat that makes me want to give them a high five. I really respect them for their achievement. I feel like if I had done this I would feel true pride that I had used the full extent of my abilities to do something that really required them.”

It’s no accident that most of the things that do this for me are “epic” in some way and usually took at least a solid year of someone’s life, if not 20 years, to create.

To illustrate further what I mean by each property, here’s how I would rate several musical works on each property:

Tonal w/ dissonance? Obsessively composed? Highly emotional? Boredom-resistant?
Mingus, The Black Saint and the Sinner Lady Yes Yes Yes Yes, complex
Stravinsky, The Rite of Spring Yes Yes Yes Yes, complex
The Soft Machine, Third Yes Yes Yes Yes, complex
Schulze, Irrlicht Yes I think so? Yes Yes, slowly-evolving
Adams, Harmonielehre Yes Yes Yes Yes, complex
The Beatles, Sgt. Pepper Not enough dissonance Yes Yes No
Coleman, Free Jazz Yes Not really Sometimes Yes, complex
Amogh Symphony, Vectorscan Yes Not really Yes Yes, complex
Stockhausen, Licht cycle Too dissonant Yes Not often Yes, complex
Autechre, Chiastic Slide Yes Yes Yes Yes, complex
Anthony Braxton, For Four Orchestras Too dissonant Yes No Yes, complex


Applying economics to the law for the first time

Teles (2010) quotes Douglas Baird, a Stanford Law student in the 70s and later dean of the University of Chicago Law School:

In the early seventies, people like Posner would come in and spend six weeks studying family law, and they’d write a couple of articles explaining why everything everyone was saying in family law was 100 percent wrong [because they’d ignored economics]. And then the replies would be, “No, we were only 80 percent wrong.” And Posner never got things exactly right, but he always turned everything upside down, and people talked about law differently… By the time I came along, and I wasn’t trained as economist, it was clear that… doing great work was easy… I used to say that this was just like knocking over Coke bottles with a baseball bat… You could just go in and write something revolutionary and go in tomorrow and write another article. I remember writing articles where the time between getting the idea and getting it accepted from a major law review was four days. I’m not Richard Posner, and few of us are. I got out of law school, and I was interested in bankruptcy law, which was inhabited by intellectual midgets… It was a complete intellectual wasteland. I got tenure by saying, “Jeez, a dollar today is worth more than a dollar tomorrow.” You got tenure for that! The reality is that there was just an open field begging for people to do great work.


Musical shiver moments, 2015 edition

Back in 2004, I wrote a list of (what I now call) “musical shiver moments.” A musical shiver moment is a moment in a musical track that hits you with special emotional force (perhaps sending a shiver down your spine). It can be the climax of a pop song, or the beginning of a catchy riff, or a particularly well-conceived mood shift, etc.

A classic example is the moment the drums finally enter in Phil Collins’ “In the Air Tonight.” Another is the chord shift for the final performance of the chorus in Whitney Houston’s “I Will Always Love You.”

(Note that for most of these shiver moments to have their impact, you need to listen to all or most of the track up to that point, first. You can’t just jump right to the shiver moment.)

It’s been over a decade since I made my original list. Here are a few more I’ve discovered since then:

  • “Solo begins” – Carla Bley – Escalator Over the Hill: Hotel Overture – 7:45
  • “The world crumbles” – Arvo Pärt – Tabula Rasa: Ludus – 7:20
  • “I knew nothing of the horses” – Scott Walker – Tilt: Farmer in the City – 5:22
  • “The riff enters” – Justice – Cross: Genesis – 0:38
  • “Desperate cry” – Osvaldo Golijov – The Dreams and Prayers of Isaac the Blind: Agitato – 7:00
  • “Sudden slices” – Klaus Schulze – Irrlicht: Satz Ebene – 9:30
  • “The theme enters” – John Adams – Grand Pianola Music: On the Great Divide – 2:20
  • “Swelling” – M83 – Hurry Up We’re Dreaming: My Tears Are Becoming a Sea – 1:11
  • “The sweet” – Anna von Hausswolff – Ceremony: Red Sun – 2:10
  • “Drums enter” – The Shining – In the Kingdom of Kitsch You Will Be a Monster: Goretex Weather Report – 1:05
  • “Entrance” – Ryan Power – Identity Picks: Sweetheart – 0:05
  • “Tone added” – Jon Hopkins – Immunity: We Disappear – 2:20
  • “Verse 2 begins” – The Fiery Furnaces – EP: Here Comes the Summer – 1:30
  • “Tonight” – Frank Ocean – channel ORANGE: Pyramids – 5:22
  • “Electronic instruments solo” – James Blake – James Blake: I Never Learnt to Share – 3:40
  • “Guitar solo peaks” – Janelle Monae – The ArchAndroid: Cold War – 2:11
  • “Surprising transition” – Kanye West – My Beautiful Dark Twisted Fantasy: Lost in the World – 0:59
  • “Soprano rising” – Henryk Górecki – Symphony No. 3: 1st movement – 15:57
  • “New instrument enters” – Fuck Buttons – Tarot Sport: Surf Solar – 5:18
  • “Into the final stretch” – Lindstrøm – Where You Go I Go Too: Where You Go I Go Too – 22:46
  • “New instrument” – Modeselektor – Happy Birthday!: Sucker Pin – 3:10
  • “Rising” – Glasvegas – Glasvegas: Ice Cream Van – 3:30
  • “Quiet after the storm” – Howard Shore – The Fellowship of the Ring: The Bridge of Khazad Dum – 4:57
  • “Finale” – John Adams – Harmonielehre: Part I – 17:01
  • “Chorus” – Phantom Planet – Phantom Planet: Knowitall – 1:06
  • “Suddenly, a groove” – Herbie Hancock – Crossings: Sleeping Giant – 11:09
  • “You thought this track couldn’t get any more epic. You were wrong.” – Godspeed You! Black Emperor – Allelujah! Don’t Bend! Ascend!: We Drift Like Worried Fire – 18:48
  • “One of my favorite melodies, 2nd time” – Jean Sibelius – Symphony No. 5: 3rd movement – 1:55

(The time markings for the classical pieces will be off for some performances/recordings, naturally.)

What are some of your musical shiver moments?

added after initial publication of this post:

  • “From percussion to melody” – Nils Frahm – Spaces: For / Peter / Toilet Brushes / More – 13:49
  • “Final atmospheric passage” – Dave Douglas – Dark Territory: Loom Large – 4:57
  • “One last time” – John Murphy – Adagio in D Minor: Adagio in D Minor (2012 Remaster) – 3:04
  • “Building groove” – Tonbruket – Forevergreens: First Flight of a Newbird – 3:19
  • “Is this the climax yet?” – Blanck Mass – World Eater: Rhesus Negative – 7:44
  • “Panic” & “Double time” – The Algorithm – Polymorphic Code: Panic – 3:32 & 6:51
  • [more to come]

Discovering CRISPR

Eric Lander tells his version of the story. Here is his take — which might or might not be reasonable — on lessons learned from the story:

The most important [lesson] is that medical breakthroughs often emerge from completely unpredictable origins. The early heroes of CRISPR were not on a quest to edit the human genome—or even to study human disease. Their motivations were a mix of personal curiosity (to understand bizarre repeat sequences in salt-tolerant microbes), military exigency (to defend against biological warfare), and industrial application (to improve yogurt production).

The history also illustrates the growing role in biology of “hypothesis-free” discovery based on big data. The discovery of the CRISPR loci, their biological function, and the tracrRNA all emerged not from wet-bench experiments but from open-ended bioinformatic exploration of large-scale, often public, genomic datasets. “Hypothesis-driven” science of course remains essential, but the 21st century will see an increasing partnership between these two approaches.

It is instructive that so many of the Heroes of CRISPR did their seminal work near the very start of their scientific careers (including Mojica, Horvath, Marraffini, Charpentier, Vogel, and Zhang)—in several cases, before the age of 30. With youth often comes a willingness to take risks—on uncharted directions and seemingly obscure questions—and a drive to succeed. It’s an important reminder at a time that the median age for first grants from the NIH has crept up to 42.

Notably, too, many did their landmark work in places that some might regard as off the beaten path of science (Alicante, Spain; France’s Ministry of Defense; Danisco’s corporate labs; and Vilnius, Lithuania). And, their seminal papers were often rejected by leading journals—appearing only after considerable delay and in less prominent venues. These observations may not be a coincidence: the settings may have afforded greater freedom to pursue less trendy topics but less support about how to overcome skepticism by journals and reviewers.

Finally, the narrative underscores that scientific breakthroughs are rarely eureka moments. They are typically ensemble acts, played out over a decade or more, in which the cast becomes part of something greater than what any one of them could do alone.

Warning: some people on Twitter are saying this article is basically PR for Lander’s Broad Institute, where Feng Zhang did his CRISPR work. Zhang is currently in a patent dispute over CRISPR with Jennifer Doudna.

ETA: Doudna comments on the article. And here is a “Landergate” link list.

Industry funding defeats transitivity

From The Philosophy of Evidence-Based Medicine:

[One problem] is that industry-sponsored trials are more likely to show a beneficial effect than non-industry funded trials [261,536–540]… This bias can have paradoxical consequences. For example, Heres et al. [541] examined randomized trials that compared different antipsychotic medications. They found that olanzapine beat risperidone, risperidone beat quetiapine, and quetiapine beat olanzapine! The relative success of the drugs was directly related to who sponsored the trial. For example, if the manufacturers of risperidone sponsored the trial, then risperidone was more likely to appear more effective than the others.

The reference is Heres et al. (2006).

Cochrane’s trick

From The Philosophy of Evidence-Based Medicine:

Archie Cochrane (who inspired the creation of the Cochrane Collaboration) explained what happened when he reported the preliminary results of a trial that compared home versus hospital treatment for varicose veins. The Medical Research Council gave its ethical approval, but cardiologists in the planned location of the trial (Cardiff) refused to take part because they were certain, based on their expertise, that hospital treatment was far superior…

Eventually Cochrane succeeded at beginning the trial in Bristol. Six months into the trial, the ethics committee called on Cochrane to compile and report on the preliminary results. At that stage, home care showed a slight but not statistically significant benefit. Cochrane, however, decided to play a trick on his colleagues: he prepared two reports, one with the actual number of deaths, and one with the number reversed. The rest of the story is best told from Cochrane’s perspective:

“As we were going into the committee, in the anteroom, I showed some cardiologists the results. They were vociferous in their abuse: ‘Archie,’ they said, ‘we always thought you were unethical. You must stop the trial at once.’ I let them have their way for some time and then apologised and gave them the true results, challenging them to say, as vehemently, that coronary care units should be stopped immediately. There was dead silence…”

Three types of nonfiction books I read

I realized recently that when I want to learn about a subject, I mentally group the available books into three categories.

I’ll call the first category “convincing.” This is the most useful kind of book for me to read on a topic, but for most topics, no such book exists. Many basic textbooks on the “hard” sciences (e.g. “settled” physics and chemistry) and the “formal” sciences (e.g. “settled” math, statistics, and computer science) count. In the softer sciences (including e.g. history), I know of very few books with the intellectual honesty and epistemic rigor to be convincing (to me) on their own. David Roodman’s book on microfinance, Due Diligence, is the only example that comes to mind as I write this.

Don’t get me wrong: I think we can learn a lot from studying softer sciences, but rarely is a single book on the softer sciences written in such a way as to be convincing to me, unless I know the topic well already.

I think of my 2nd category as “raw data.” These books make a good case that the data they present were collected and presented in a fairly reasonable way, and I find it useful to know what the raw data are, but if and when the book attempts to persuade me of non-obvious causal hypotheses, I find the book illuminating but unconvincing (on its own). Some examples:

Finally, my 3rd category for nonfiction is “food for thought.” Besides being unconvincing about non-obvious causal inferences, these books also fail to make a good case that the data supporting their arguments were collected and presented in a reasonable way. So what I get from them is just some basic terminology, and some hypotheses and arguments and stories I didn’t know about before. This category includes the vast majority of all non-fiction, e.g.:

My guess is that I’m more skeptical than most heavy readers of non-fiction, including most scientists. I’m sure I’ll blog more in the future about why.

Some 2016 movies I’m looking forward to

I’m only counting films to be first released in 2016 according to IMDB. In descending order of how confident I am that I’ll rate it as “really liked” or “loved”:

  1. Coen brothers, Hail, Caesar!
  2. Stanton, Finding Dory
  3. Linklater, Everybody Wants Some
  4. Nichols, Midnight Special
  5. Villeneuve, Story of Your Life
  6. Dardenne brothers, The Unknown Girl
  7. Farhadi, Seller
  8. Scorsese, Silence

For all other movies coming out in 2016 that I’ve seen mentioned, I’m <70% confident I’ll rate them as “really liked” or “loved.”

Some books I’m looking forward to, January 2016 edition

* = added this round

Books, music, etc. from December 2015



Music I most enjoyed discovering this month:


Ones I really liked, or loved:

  • Phoenix (2014, Christian Petzold)

One of my favorite melodies, rediscovered

There are about a dozen melodies that I find myself humming and whistling without realizing it. One is “Yellow Submarine.” Another begins at about 2:20 in Adams’ “On the Dominant Divide.”

Another is one that I’ve been humming for years but couldn’t remember where I had heard it.

Well, today, I finally stumbled into that melody once again! It turns out it’s the melody that begins at about 1:09 into the 3rd movement of Sibelius’ 5th symphony.

Ahhhhhhhh. So good.

If you’re an “AI safety lurker,” now would be a good time to de-lurk

Recently, the study of potential risks from advanced artificial intelligence has attracted substantial new funding, prompting new job openings at e.g. Oxford University and (in the near future) at Cambridge University, Imperial College London, and UC Berkeley.

This is the dawn of a new field. It’s important to fill these roles with strong candidates. The trouble is, it’s hard to find strong candidates at the dawn of a new field, because universities haven’t yet begun to train a steady flow of new experts on the topic. There is no “long-term AI safety” program for graduate students anywhere in the world.

Right now the field is pretty small, and the people I’ve spoken to (including e.g. at Oxford) seem to agree that it will be a challenge to fill these roles with candidates they already know about. Oxford has already re-posted one position, because no suitable candidates were found via the original posting.

So if you’ve developed some informal expertise on the topic — e.g. by reading books, papers, and online discussions — but you are not already known to the folks at Oxford, Cambridge, FLI, or MIRI, now would be an especially good time to de-lurk and say “I don’t know whether I’m qualified to help, and I’m not sure there’s a package of salary, benefits, and reasons that would tempt me away from what I’m doing now, but I want to at least let you know that I exist, I care about this issue, and I have at least some relevant skills and knowledge.”

Maybe you’ll turn out not to be a good candidate for any of these roles. Maybe you’ll learn the details and decide you’re not interested. But if you don’t let us know you exist, neither of those things can even begin to happen, and these important roles at the dawn of a new field will be less likely to be filled with strong candidates.

I’m especially passionate about de-lurking of this sort because when I first learned about MIRI, I just assumed I wasn’t qualified to help out, and wouldn’t want to, anyway. But after speaking to some folks at MIRI, it turned out I really could help out, and I’m glad I did. (I was MIRI’s Executive Director for ~3.5 years.)

So if you’ve been reading and thinking about long-term AI safety issues for a while now, and you have some expertise in computer science, AI, analytic/formal philosophy, mathematics, statistics, policy, risk analysis, forecasting, or economics, and you’re not already in contact with the people at the organizations I named above, please step forward and tell us you exist.

UPDATE Jan. 2, 2016: At this point in the original post, I recommended that people de-lurk by emailing me or by commenting below. However, I was contacted by far more people than I expected (100+), so I had to reply to everyone (on Dec. 19th) with a form email instead. In that email I thanked everyone for contacting me as I had requested, apologized for not being able to respond individually, and made the following request:

If you think you might be interested in a job related to long-term AI safety either now or in the next couple years, please fill out this 3-question Google form, which is a lot easier than filling out any of the posted job applications. This will make it much easier for the groups that are hiring to skim through your information and decide which people they want to contact and learn more about.

Everyone who contacted/contacts me after Dec. 19th will instead receive a link to this section of this blog post. If I’ve linked you here, please consider filling out the 3-question Google form above.

(Note that although I work as a GiveWell research analyst, my focus at GiveWell is not AI risks, and my views on this topic are not necessarily GiveWell’s views.)

Some books I’m looking forward to, December 2015 edition

* = added this round

Coyne et al. are kinda pissed about Authentic Happiness

In the following passage, Coyne et al. (2010) repeatedly cite Authentic Happiness, by positive psychology co-founder and past APA president Martin Seligman, as an example of what they’re saying is wrong with positive psychology:

Critical discussions of the potential contributions of a positive psychology have been hampered by the sloganeering of the leaders of the movement and their labeling of the alternative as a “negative psychology”…

The ridiculing of pessimists as losers in positive psycholgy self-help books, money back guarantees on websites offering personal coaches and self-help techniques claiming to promote happiness, and the presentation of pseudoscientific happiness regression equations [Happines = Set range + Circumstances + Factors under voluntary control] all… suggest that, while the leaders of positive psychology claim it to be science based, they feel free to deliver platonic noble lies to the unwashed masses…

…support for such victim blaming can come not only from the fringe, but from mainstream positive psychology. Anyone who doubts this need only to Google “positive psychology” and “coaching” and experiment by adding some names of proponents of mainstream positive psychology. They will soon be brought to websites with claims that retaining a personal coach or engaging in web-based exercises for a substantial fee is guaranteed to instill happines that lasts and that happiness is related to health. More efficiently, the skeptical reader can reach websites with similar claims by simply joining the American Psychological Association listserv Friends of Positive Psychology… and by double clicking on the web links provided in the signatures of posters there.

Data collection for the Global Burden of Disease project

From Epic Measures:

Of the 2 billion deaths since 1970 the new Global Burden would ultimately cover, only about 25 percent had been recorded in a vital registration system accessible to researchers… [Christopher Murray’s] proposal to the Gates Foundation had said the entire project would take three years to complete, giving a deadline of July 2010. Three years to gather and analyze all available details about the health of every person on Earth…

Different countries brought a varied set of challenges. In China, regulations forbade almost all core health data from leaving the country, so Chinese partners had to do analyses and share the results with Seattle. U.S. states, by contrast, sold annual databases of their in-patient hospital users to anyone in the world, for prices ranging from $35 to $2,000. In Ghana, almost the exact equivalent records were available free.

In Nigeria, Africa’s largest country by population, the data indexers surveyed hospitals, police stations, health clinics, libraries, colonial archives, and even cemetery plot records. In Libya, the latest census and civil registries turned out to be available online, but only after clicking through seven Web pages written in Arabic. In Iraq, during the end of the American-led occupation, months of spadework revealed the existence of two recent government household surveys. These would help estimate how many Iraqis were being killed or injured by war, as opposed to other causes, a hugely disputed topic. Trying e-mail, Skype, and phone, Speyer finally managed to reach the Iraqi official in charge of statistics and information technology. “She said they’d be happy to share the survey microdata with us, and I said, ‘Can you e-mail it or upload it to a website?’” he recalls. “She said no. She burned it onto a CD and told me I had to pick that up in the Baghdad Green Zone.”

…Another completely separate stream of information, and a big one, came from others’ published scientific studies. About what? About “health.” There were ten thousand articles a month published with a reference to epidemiology. To the maximum degree possible, Murray wanted all of those results pulled, digitized, and entered into Global Burden, too. Put another way, a fraction of a fraction of the data supplied to the study’s scientists was to be everything everyone else had ever discovered.

Books, music, etc. from November 2015



Music I most enjoyed discovering this month:


Ones I really liked, or loved:

Glass invented meta-analysis to prove someone wrong

Apparently Gene Glass invented meta-analysis because he wanted to prove someone wrong:

…in the summer of 1974, I set about to do battle with Hans Eysenck and prove that psychotherapy – my psychotherapy – was an effective treatment. I joined the battle with Eysenck’s (1965) review of the psychotherapy outcome literature. Eysenck began his famous reviews by eliminating from consideration all theses, dissertations, project reports or other contemptible items not published in peer-reviewed journals. This arbitrary exclusion of literally hundreds of evaluations of therapy outcomes was indefensible to my mind. It’s one thing to believe that peer review guarantees truth; it is quite another to believe that all truth appears in peer-reviewed journals.

Next, Eysenck eliminated any experiment that did not include an untreated control group. This makes no sense whatever, because head-to-head comparisons of two different types of psychotherapy contribute a great deal to our knowledge of psychotherapy effects…

Having winnowed a huge literature down to 11 studies (!) by whim and prejudice, Eysenck proceeded to describe their findings solely in terms of whether or not statistical significance was reached at the .05 level…

Finally, Eysenck did something truly staggering in its illogic. If a study showed significant differences favoring therapy over control on what he regarded as a ‘subjective’ measure of outcome (e.g., the Rorschach or the Thematic Apperception Test), he discounted the findings entirely. So be it; he may be a tough judge, but that’s his right. But then, when encountering a study that showed differences on an ‘objective’ outcome measure (e.g., grade-point average) but no differences on a subjective measure (such as the Thematic Apperception Test), Eysenck discounted the entire study because the outcome differences were ‘inconsistent’.

Looking back on it, I can almost credit Eysenck with the invention of meta-analysis by anti-thesis. By doing everything in the opposite way that he did, one would have been led straight to meta-analysis. Adopt an a posteriori attitude toward including studies in a synthesis, replace statistical significance by measures of strength of relationship or effect, and view the entire task of integration as a problem in data analysis where ‘studies’ are quantified and the resulting database subjected to statistical analysis, and meta-analysis assumes its first formulation. Thank you, Professor Eysenck.

…[Our] first meta-analysis of the psychotherapy outcome research finished in 1974-1975 found that the typical therapy trial raised the treatment group to a level about two-thirds of a standard deviation on average above the average of untreated controls…

…[Researchers’] reactions [to the meta-analysis] foreshadowed the eventual reception of the work among psychologists. Some said that the work was revolutionary and proved what they had known all along; others said it was wrongheaded and meaningless. The widest publication of the work came in 1977, in an article by Mary Lee Smith and myself in the American Psychologist. Eysenck responded to the article by calling it ‘mega-silliness’…