All graduate students and econometrics students should read this book

Every now and again I read a book that has a very simple point which is heavily overstated and makes me wonder if I am somehow missing something important. This is what I found when reading "The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives" by Stephen Ziliak and Dierdre McCloskey. Anyway, I'll relate what I took away from the book, and hopefully someone will enlighten me if I somehow missed the point entirely.

Ziliak and McCloskey's overall premise is that generations of economists (and psychologists, medical researchers, and others) have been misusing statistics by relying solely on statistical significance, while ignoring practical significance (what they refer to as "oomph"). They dismiss the sole focus on statistical significance as "sizeless science" (and sometimes as "sign econometrics", where the researcher is only concerned about whether an effect is positive or negative, and not how big the effect is). That is certainly an important point, and one I am sure I am as guilty of as many others (in my blog as well as my writing, I don't doubt).

The remedy appears to be a re-focus on meaningful interpretation of the size of the effects, which is hard to argue against. It's all very well for us to note when coefficients are statistically significant, but are they economically significant is a more important question. Oftentimes we find variables that are statistically significance make no difference at all (for one example from my own work, see the (lack of) effect of social media on election results here, or ungated here).

On the flip side, variables that are not statistically significant might be important, with the statistical test simply being under-powered to identify it. Some of my most recent work, looking at the effect of a pastoral care intervention on pass rates in ECON100, illustrates this (I'll blog on that work at a later time).

However, I don't think it takes 250+ pages to make these points. I remember well my graduate studies really struggling through some McCloskey papers, and this book wasn't a whole lot easier to read. I also found the book rather repetitive. As one extreme example, the same quote from Karl Pearson appears on page 198-199, and again one paragraph later.

Researchers who are fans of Bayesian analysis will no doubt find a lot of support for their approach in the book, although Ziliak and McCloskey state up front that this was not their intent. The authors could also have pointed us more directly to meta-analysis, which is growing in importance in economics as it already has in other fields such as public health. Indeed my colleague Jacques Poot has done a great amount of work on meta-analysis in economics (see here for example).

I was a little confused by the authors' preference for confidence intervals. To my mind, this suffers from the same problems as what they refer to as asterisk economics. Or worse, since you set only one level for the confidence interval, whereas with asterisks at least you are showing multiple levels of significance (though, admittedly, not the bounds). I'm unconvinced that confidence intervals will make authors interpret effects sizes more carefully (which is what Ziliak and McCloskey argue).

Anyway, despite the flaws this is definitely a book that all graduate students in economics should read, as well as undergraduate students of econometrics. See here and here (pdf) for some other reviews of the book. Finally, I give the last word on p-values here to xkcd: