Book review: Randomistas

It seemed appropriate to follow up reading Banerjee and Duflo's Poor Economics (which I reviewed here last month) with Andrew Leigh's Randomistas. Where Banerjee and Duflo focused their book on experiments in development, Leigh ranges much wider. The subtitle is "How radical researchers are changing our world". I'm not sure I would go as far as 'radical', but the book does provide a useful overview of the use of randomised controlled trials in a range of applications, from medicine to education, crime and justice, poverty and development, agriculture, and even politics and philanthropy. This is a good place to start if you want to see how widely randomised trials are being used.

I like the way that Leigh writes, and especially this bit of motivation for experiments:

In the film Sliding Doors, we follow the life of Gwyneth Paltrow's character, Helen, according to whether or not she manages to catch a train... What makes Sliding Doors a fun movie is that we get to see both pathways - like rereading a Choose Your Own Adventure book. We get to see what economists call the 'counterfactual' - the road not taken...

Researchers have spent years thinking about how best to come up with credible comparison groups, but the benchmark to which they keep returning is the randomised trial. There's simply no better way to determine the counterfactual than to randomly allocate participants into two groups: one that gets the treatment, and another that does not.

It is surprising to see the variety of randomised trials that have been employed. Like me, I bet you didn't realise the link between Sesame Street and randomised trials:

In its first year, Sesame Street was evaluated in a randomised trial, which compared a treatment group (children who were encouraged to watch the program) with a regular control group. Unfortunately, the researchers hadn't reckoned on the show's popularity. With more than one-third of American children tuning in to each episode, there wasn't much difference in viewing rates between the two groups...

So the next year, researchers took a different approach - focusing on cities where Sesame Street was only available on cable and then randomly providing cable television to a subset of low-income households... Children who watched Sesame Street had the same cognitive skills as non-viewers who were a year older.

Leigh writes in an engaging style, and while not all of the content is new (especially if you have read Poor Economics), there is plenty to retain the reader's interest. The subtitle of the book even resulted from an experiment!

Leigh also doesn't shy away from criticisms of randomised trials, and takes note of the replication crisis in social science, to which randomised trials have not been immune. I felt like this particular part of the book could have been expanded more. However, I was interested to note that Leigh makes explicit mention of one experiment that has been (and continues to be) widely reported - Sheena Iyengar and Mark Lepper's 2000 study of the paradox of choice (ungated), in relation to jam purchases (see also this article by Barry Schwartz, who wrote the book The Paradox of Choice):

A decade after the initial study appeared, a team of psychologists collated as many of these replication studies as they could find... Among the fifty replication studies, a majority went in the opposite direction from the original jam choice experiment. Averaging all the results, the psychologists concluded that the number of available options had 'virtually zero' impact on customer satisfaction or purchases.

I really enjoyed reading this book, especially as a companion to Poor Economics. Recommended for your late holiday reading list!