Industry funding defeats transitivity

From The Philosophy of Evidence-Based Medicine:

[One problem] is that industry-sponsored trials are more likely to show a beneficial effect than non-industry funded trials [261,536–540]… This bias can have paradoxical consequences. For example, Heres et al. [541] examined randomized trials that compared different antipsychotic medications. They found that olanzapine beat risperidone, risperidone beat quetiapine, and quetiapine beat olanzapine! The relative success of the drugs was directly related to who sponsored the trial. For example, if the manufacturers of risperidone sponsored the trial, then risperidone was more likely to appear more effective than the others.

The reference is Heres et al. (2006).

Cochrane’s trick

From The Philosophy of Evidence-Based Medicine:

Archie Cochrane (who inspired the creation of the Cochrane Collaboration) explained what happened when he reported the preliminary results of a trial that compared home versus hospital treatment for varicose veins. The Medical Research Council gave its ethical approval, but cardiologists in the planned location of the trial (Cardiff) refused to take part because they were certain, based on their expertise, that hospital treatment was far superior…

Eventually Cochrane succeeded at beginning the trial in Bristol. Six months into the trial, the ethics committee called on Cochrane to compile and report on the preliminary results. At that stage, home care showed a slight but not statistically significant benefit. Cochrane, however, decided to play a trick on his colleagues: he prepared two reports, one with the actual number of deaths, and one with the number reversed. The rest of the story is best told from Cochrane’s perspective:

“As we were going into the committee, in the anteroom, I showed some cardiologists the results. They were vociferous in their abuse: ‘Archie,’ they said, ‘we always thought you were unethical. You must stop the trial at once.’ I let them have their way for some time and then apologised and gave them the true results, challenging them to say, as vehemently, that coronary care units should be stopped immediately. There was dead silence…”

Three types of nonfiction books I read

I realized recently that when I want to learn about a subject, I mentally group the available books into three categories.

I’ll call the first category “convincing.” This is the most useful kind of book for me to read on a topic, but for most topics, no such book exists. Many basic textbooks on the “hard” sciences (e.g. “settled” physics and chemistry) and the “formal” sciences (e.g. “settled” math, statistics, and computer science) count. In the softer sciences (including e.g. history), I know of very few books with the intellectual honesty and epistemic rigor to be convincing (to me) on their own. David Roodman’s book on microfinance, Due Diligence, is the only example that comes to mind as I write this.

Don’t get me wrong: I think we can learn a lot from studying softer sciences, but rarely is a single book on the softer sciences written in such a way as to be convincing to me, unless I know the topic well already.

I think of my 2nd category as “raw data.” These books make a good case that the data they present were collected and presented in a fairly reasonable way, and I find it useful to know what the raw data are, but if and when the book attempts to persuade me of non-obvious causal hypotheses, I find the book illuminating but unconvincing (on its own). Some examples:

Finally, my 3rd category for nonfiction is “food for thought.” Besides being unconvincing about non-obvious causal inferences, these books also fail to make a good case that the data supporting their arguments were collected and presented in a reasonable way. So what I get from them is just some basic terminology, and some hypotheses and arguments and stories I didn’t know about before. This category includes the vast majority of all non-fiction, e.g.:

My guess is that I’m more skeptical than most heavy readers of non-fiction, including most scientists. I’m sure I’ll blog more in the future about why.

Some 2016 movies I’m looking forward to

I’m only counting films to be first released in 2016 according to IMDB. In descending order of how confident I am that I’ll rate it as “really liked” or “loved”:

  1. Coen brothers, Hail, Caesar!
  2. Stanton, Finding Dory
  3. Linklater, Everybody Wants Some
  4. Nichols, Midnight Special
  5. Villeneuve, Story of Your Life
  6. Dardenne brothers, The Unknown Girl
  7. Farhadi, Seller
  8. Scorsese, Silence

For all other movies coming out in 2016 that I’ve seen mentioned, I’m <70% confident I’ll rate them as “really liked” or “loved.”

Some books I’m looking forward to, January 2016 edition

* = added this round

Books, music, etc. from December 2015

Books

Music

Music I most enjoyed discovering this month:

Movies/TV

Ones I really liked, or loved:

  • Phoenix (2014, Christian Petzold)

One of my favorite melodies, rediscovered

There are about a dozen melodies that I find myself humming and whistling without realizing it. One is “Yellow Submarine.” Another begins at about 2:20 in Adams’ “On the Dominant Divide.”

Another is one that I’ve been humming for years but couldn’t remember where I had heard it.

Well, today, I finally stumbled into that melody once again! It turns out it’s the melody that begins at about 1:09 into the 3rd movement of Sibelius’ 5th symphony.

Ahhhhhhhh. So good.

If you’re an “AI safety lurker,” now would be a good time to de-lurk

Recently, the study of potential risks from advanced artificial intelligence has attracted substantial new funding, prompting new job openings at e.g. Oxford University and (in the near future) at Cambridge University, Imperial College London, and UC Berkeley.

This is the dawn of a new field. It’s important to fill these roles with strong candidates. The trouble is, it’s hard to find strong candidates at the dawn of a new field, because universities haven’t yet begun to train a steady flow of new experts on the topic. There is no “long-term AI safety” program for graduate students anywhere in the world.

Right now the field is pretty small, and the people I’ve spoken to (including e.g. at Oxford) seem to agree that it will be a challenge to fill these roles with candidates they already know about. Oxford has already re-posted one position, because no suitable candidates were found via the original posting.

So if you’ve developed some informal expertise on the topic — e.g. by reading books, papers, and online discussions — but you are not already known to the folks at Oxford, Cambridge, FLI, or MIRI, now would be an especially good time to de-lurk and say “I don’t know whether I’m qualified to help, and I’m not sure there’s a package of salary, benefits, and reasons that would tempt me away from what I’m doing now, but I want to at least let you know that I exist, I care about this issue, and I have at least some relevant skills and knowledge.”

Maybe you’ll turn out not to be a good candidate for any of these roles. Maybe you’ll learn the details and decide you’re not interested. But if you don’t let us know you exist, neither of those things can even begin to happen, and these important roles at the dawn of a new field will be less likely to be filled with strong candidates.

I’m especially passionate about de-lurking of this sort because when I first learned about MIRI, I just assumed I wasn’t qualified to help out, and wouldn’t want to, anyway. But after speaking to some folks at MIRI, it turned out I really could help out, and I’m glad I did. (I was MIRI’s Executive Director for ~3.5 years.)

So if you’ve been reading and thinking about long-term AI safety issues for a while now, and you have some expertise in computer science, AI, analytic/formal philosophy, mathematics, statistics, policy, risk analysis, forecasting, or economics, and you’re not already in contact with the people at the organizations I named above, please step forward and tell us you exist.

UPDATE Jan. 2, 2016: At this point in the original post, I recommended that people de-lurk by emailing me or by commenting below. However, I was contacted by far more people than I expected (100+), so I had to reply to everyone (on Dec. 19th) with a form email instead. In that email I thanked everyone for contacting me as I had requested, apologized for not being able to respond individually, and made the following request:

If you think you might be interested in a job related to long-term AI safety either now or in the next couple years, please fill out this 3-question Google form, which is a lot easier than filling out any of the posted job applications. This will make it much easier for the groups that are hiring to skim through your information and decide which people they want to contact and learn more about.

Everyone who contacted/contacts me after Dec. 19th will instead receive a link to this section of this blog post. If I’ve linked you here, please consider filling out the 3-question Google form above.

(Note that although I work as a GiveWell research analyst, my focus at GiveWell is not AI risks, and my views on this topic are not necessarily GiveWell’s views.)

Some books I’m looking forward to, December 2015 edition

* = added this round

Coyne et al. are kinda pissed about Authentic Happiness

In the following passage, Coyne et al. (2010) repeatedly cite Authentic Happiness, by positive psychology co-founder and past APA president Martin Seligman, as an example of what they’re saying is wrong with positive psychology:

Critical discussions of the potential contributions of a positive psychology have been hampered by the sloganeering of the leaders of the movement and their labeling of the alternative as a “negative psychology”…

The ridiculing of pessimists as losers in positive psycholgy self-help books, money back guarantees on websites offering personal coaches and self-help techniques claiming to promote happiness, and the presentation of pseudoscientific happiness regression equations [Happines = Set range + Circumstances + Factors under voluntary control] all… suggest that, while the leaders of positive psychology claim it to be science based, they feel free to deliver platonic noble lies to the unwashed masses…

…support for such victim blaming can come not only from the fringe, but from mainstream positive psychology. Anyone who doubts this need only to Google “positive psychology” and “coaching” and experiment by adding some names of proponents of mainstream positive psychology. They will soon be brought to websites with claims that retaining a personal coach or engaging in web-based exercises for a substantial fee is guaranteed to instill happines that lasts and that happiness is related to health. More efficiently, the skeptical reader can reach websites with similar claims by simply joining the American Psychological Association listserv Friends of Positive Psychology… and by double clicking on the web links provided in the signatures of posters there.

Data collection for the Global Burden of Disease project

From Epic Measures:

Of the 2 billion deaths since 1970 the new Global Burden would ultimately cover, only about 25 percent had been recorded in a vital registration system accessible to researchers… [Christopher Murray’s] proposal to the Gates Foundation had said the entire project would take three years to complete, giving a deadline of July 2010. Three years to gather and analyze all available details about the health of every person on Earth…

Different countries brought a varied set of challenges. In China, regulations forbade almost all core health data from leaving the country, so Chinese partners had to do analyses and share the results with Seattle. U.S. states, by contrast, sold annual databases of their in-patient hospital users to anyone in the world, for prices ranging from $35 to $2,000. In Ghana, almost the exact equivalent records were available free.

In Nigeria, Africa’s largest country by population, the data indexers surveyed hospitals, police stations, health clinics, libraries, colonial archives, and even cemetery plot records. In Libya, the latest census and civil registries turned out to be available online, but only after clicking through seven Web pages written in Arabic. In Iraq, during the end of the American-led occupation, months of spadework revealed the existence of two recent government household surveys. These would help estimate how many Iraqis were being killed or injured by war, as opposed to other causes, a hugely disputed topic. Trying e-mail, Skype, and phone, Speyer finally managed to reach the Iraqi official in charge of statistics and information technology. “She said they’d be happy to share the survey microdata with us, and I said, ‘Can you e-mail it or upload it to a website?’” he recalls. “She said no. She burned it onto a CD and told me I had to pick that up in the Baghdad Green Zone.”

…Another completely separate stream of information, and a big one, came from others’ published scientific studies. About what? About “health.” There were ten thousand articles a month published with a reference to epidemiology. To the maximum degree possible, Murray wanted all of those results pulled, digitized, and entered into Global Burden, too. Put another way, a fraction of a fraction of the data supplied to the study’s scientists was to be everything everyone else had ever discovered.

Books, music, etc. from November 2015

Books

Music

Music I most enjoyed discovering this month:

Movies/TV

Ones I really liked, or loved:

Glass invented meta-analysis to prove someone wrong

Apparently Gene Glass invented meta-analysis because he wanted to prove someone wrong:

…in the summer of 1974, I set about to do battle with Hans Eysenck and prove that psychotherapy – my psychotherapy – was an effective treatment. I joined the battle with Eysenck’s (1965) review of the psychotherapy outcome literature. Eysenck began his famous reviews by eliminating from consideration all theses, dissertations, project reports or other contemptible items not published in peer-reviewed journals. This arbitrary exclusion of literally hundreds of evaluations of therapy outcomes was indefensible to my mind. It’s one thing to believe that peer review guarantees truth; it is quite another to believe that all truth appears in peer-reviewed journals.

Next, Eysenck eliminated any experiment that did not include an untreated control group. This makes no sense whatever, because head-to-head comparisons of two different types of psychotherapy contribute a great deal to our knowledge of psychotherapy effects…

Having winnowed a huge literature down to 11 studies (!) by whim and prejudice, Eysenck proceeded to describe their findings solely in terms of whether or not statistical significance was reached at the .05 level…

Finally, Eysenck did something truly staggering in its illogic. If a study showed significant differences favoring therapy over control on what he regarded as a ‘subjective’ measure of outcome (e.g., the Rorschach or the Thematic Apperception Test), he discounted the findings entirely. So be it; he may be a tough judge, but that’s his right. But then, when encountering a study that showed differences on an ‘objective’ outcome measure (e.g., grade-point average) but no differences on a subjective measure (such as the Thematic Apperception Test), Eysenck discounted the entire study because the outcome differences were ‘inconsistent’.

Looking back on it, I can almost credit Eysenck with the invention of meta-analysis by anti-thesis. By doing everything in the opposite way that he did, one would have been led straight to meta-analysis. Adopt an a posteriori attitude toward including studies in a synthesis, replace statistical significance by measures of strength of relationship or effect, and view the entire task of integration as a problem in data analysis where ‘studies’ are quantified and the resulting database subjected to statistical analysis, and meta-analysis assumes its first formulation. Thank you, Professor Eysenck.

…[Our] first meta-analysis of the psychotherapy outcome research finished in 1974-1975 found that the typical therapy trial raised the treatment group to a level about two-thirds of a standard deviation on average above the average of untreated controls…

…[Researchers’] reactions [to the meta-analysis] foreshadowed the eventual reception of the work among psychologists. Some said that the work was revolutionary and proved what they had known all along; others said it was wrongheaded and meaningless. The widest publication of the work came in 1977, in an article by Mary Lee Smith and myself in the American Psychologist. Eysenck responded to the article by calling it ‘mega-silliness’…

My biggest complaint about Last.fm

I use last.fm to track what music I listen to. Unfortunately, it’s not very accurate.

The first problem is that it doesn’t track music listened to on most online services (e.g. Youtube, Bandcamp). But I can’t really complain about that, since I just discovered there’s an app for that. Though, its Youtube support is shaky, I assume because it’s hard to tell what’s a non-musical video and what is a music track.

A bigger problem for me is that last.fm counts up what I listen to by counting tracks played rather than by counting time played. So if I listen to a punk band for one hour, and then I listen to Miles Davis for one hour, last.fm will make it look as though I like the punk band 10x more than I like Miles Davis, because the punk band writes 3 minute tracks and Miles Davis records 30 minute tracks.

A comparison of Mac and cloud programs for PDF rich text extraction

I like reading things via the Kindle app on my phone, because then I can read from anywhere. Unfortunately, most of what I want to read is in PDF format, so the text can’t “reflow” on my phone’s small screen like a normal ebook does. PDF text extraction programs aim to solve this problem by extracting the text (and in some cases, other elements) from a PDF and exporting it to a format that allows text reflow, for example .docx or .epub.

Which PDF text extraction program is best? I couldn’t find any credible comparisons, so I decided to do my own.

My criteria for were:

  1. The program must run on Mac OS X or run in the cloud.
  2. It must be free or have a free trial available, so I can run this test without spending hundreds of dollars.
  3. It must be easy to use. If I have to install special packages or tweak environment variables to run the program, it doesn’t qualify.
  4. It must preserve images, tables, and equations amidst the text, since the documents I want to read often include important charts, tables, and equations. (It’s fine if equations and tables are simply handled as images.)
  5. It must be able to handle multi-column pages.
  6. It must work with English, but I don’t care about other languages because I can’t read them anyway.
  7. I don’t care that much about final file size or how long the conversion takes, so long as the program doesn’t crash on 1 out of every 10 attempts and doesn’t create crazy 200mb files or something like that.

To run my test, I assembled a gauntlet of 16 PDFs of the sort I often read, including several PDFs from journal websites, a paper from arXiv, and multiple scanned-and-OCRed academic book chapters.

A quick search turned up way too many Mac or cloud-based programs to test, so I decided to focus on a few that were from major companies or were particularly easy to use.

[Read more…]

Jazz and other classical musics

From Other Classical Musics: Fifteen Great Traditions, on what the book means by “classical musics” and why jazz is one of them:

The term ‘art music’ is too broad… ‘Court music’ would have worked for some traditions, but not for all; ‘classical’ is the adjective best capable of covering what every society regards as its own Great Tradition…

According to our rule-of-thumb, a classical music will have evolved in a political-economic environment with built-in continuity… where a wealthy class of connoisseurs has stimulated its creation by a quasi-priesthood of professionals; it will have enjoyed high social esteem. It will also have had the time and space to develop rules of composition and performance, and to allow the evolution of a canon of works, or forms… almost all classical music has vernacular roots, and periodically renews itself from them;

…As a newish nation whose dominant culture is essentially European, America has – like Australasia – imported Europe’s classical music, but in jazz it has its own indigenous classical form. Those in doubt as to whether jazz belongs in this book should bear in mind that its controlled-improvisatory nature aligns it with almost all other classical musics. Doubters might also consider how closely jazz’s historical trajectory mirrors that of European music, if telescoped into a much shorter time. It too has vernacular roots, and was raised by a series of master-musicians to the status of an art-music; it too has evolved via a ‘classical’ period through a succession of modernist phases, and has become every bit as esoteric as European classical modernism. Since the 1950s jazz has had its own early-music revivalists (from trad bands to Wynton Marsalis) and, again like Western classical music, it too seems unsure where to go next. And now that it’s gone native on every continent, jazz is as global as Beethoven.

Some books I’m looking forward to, November 2015 edition

* = added this round

Some practical obstacles to becoming a fan of modern classical music

I think there is a ton of modern classical music (MCM) that listeners would enjoy if they had a way to more cheaply discover it.

Why can’t people cheaply discover MCM they like, and what can be done about it? Below are some guesses.

 

Obstacle 1: There are almost no full-time critics of classical/MCM music.

According to this post, there may be about a dozen full-time critics of classical/MCM music in the USA. This is probably more a symptom of the difficulty of exploring MCM than a cause, but it helps perpetuate the problem. The best solution to this is probably to increase demand for MCM critics by fixing the problems below (and perhaps others I’ve missed).

Obstacle 2: MCM critics do not rate works/albums or give them genre labels.

This one really bugs me. Honestly, how is someone with limited time supposed to navigate the world of MCM if nobody will tell them which works and albums are the best ones, and roughly what they sound like? Sure, this information is sometimes (but not always!) buried somewhere in reviews of new works or albums, but it needs to be at the top of the review, right under the artist and work/album title. Yeah, yeah, musical quality can’t be reduced to a single number and a list of genre tags, blah blah blah GET OVER IT and BE MORE HELPFUL.

Obstacle 3: Because of obstacle #2, there’s no way to aggregate expert opinion on MCM works.

Rock/pop fans have MetacriticAOTY, etc. This is impossible for MCM because MCM critics do not rate albums.

Obstacle 4: MCM critics don’t even make lists.

Even if critics didn’t rate every album they reviewed, they could at least make year-end “best of” lists and genre-specific “best of” lists. But MCM critics almost never do this. Seriously, how is anyone with limited time supposed to navigate MCM without listicles?

Obstacle 5: Many MCM works aren’t recorded for years after their debut.

Suppose you read a review of a new rock/pop album, and you want to hear it. What do you do? You stream it on Spotify or buy it on iTunes.

But suppose you read a review of a new MCM work, and you want to hear it. What do you do? The answer is probably “buy a plane ticket to another city on a specific date and pay $80 to hear it once in a concert hall, otherwise wait 1-15 years for the work to be recorded and released and hope you remember to listen to it then.” To someone used to the rock/pop world, this is utter madness.

To become more consumable by a mass-ish market, MCM works need to be recorded and released first, and performed for the public later, after people have had a chance to stream/buy it and decide whether they want to endure the cost and inconvenience of seeing it live.

Unfortunately, I don’t know enough about the MCM business to know how to fix this.

 

Not on this list

Conspicuously missing from my list of obstacles is “most MCM composers write unlistenable random noise.” They do, of course, but I don’t see that as a problem.

“Unlistenable random noise” is hyperbole, I know, except for some pieces like John Cage’s HPSCHD. What I mean is that most MCM composers tend to write music that sounds to most people like “unlistenable random noise.” As Joe Queenan put it,

During a radio interview between acts at the Metropolitan Opera in New York, a famous singer recently said she could not understand why audiences were so reluctant to listen to new music, given that they were more than ready to attend sporting events whose outcome was uncertain. It was a daft analogy… when Spain plays Germany, everyone knows that the game will be played with one ball, not eight; and that the final score will be 1-0 or 3-2 or even 8-1 – but definitely not 1,600,758 to Arf-Arf the Chalet Ate My Banana. The public may not know in advance what the score will be, but it at least understands the rules of the game.

… It is not the composers’ fault that they wrote uncompromising music… but it is not my fault that I would rather listen to Bach.

If the obstacles listed above were fixed (plus some I probably missed), then MCM would be in the same place rock/pop music is: composers can write whatever they want, including unlistenable random noise, and some of it will find a big audience, most of it won’t, and that’s fine.

Chomsky on the War on Drugs

More Chomsky, from Understanding Power (footnotes also reproduced):

So take a significant question you never hear asked despite this supposed “Drug War” which has been going on for years and years: how many bankers and chemical corporation executives are in prison in the United States for drug-related offenses? Well, there was recently an O.E.C.D. [Organization for Economic Cooperation and Development] study of the international drug racket, and they estimated that about a half-trillion dollars of drug money gets laundered internationally every year—more than half of it through American banks. I mean, everybody talks about Colombia as the center of drug-money laundering, but they’re a small player: they have about $10 billion going through, U.S. banks have about $260 billion.

Okay, that’s serious crime—it’s not like robbing a grocery store. So American bankers are laundering huge amounts of drug money, everybody knows it: how many bankers are in jail? None. But if a black kid gets caught with a joint, he goes to jail.

…Or why not ask another question — how many U.S. chemical corporation executives are in jail? Well, in the 1980s, the C.I.A. was asked to do a study on chemical exports to Latin America, and what they estimated was that more than 90 percent of them are not being used for industrial production at all — and if you look at the kinds of chemicals they are, it’s obvious that what they’re really being used for is drug production. Okay, how many chemical corporation executives are in jail in the United States? Again, none — because social policy is not directed against the rich, it’s directed against the poor.

Books, music, etc. from October 2015

Books

Music

Music I most enjoyed discovering this month:

Movies/TV

Ones I really liked, or loved: