Back in 2021, I wrote a post about the past and future of statistical significance, based on three papers published in the Journal of Economic Perspectives (see here). The third paper I referred to in that post was by Edward Miguel (University of California, Berkeley) and focused on open science and research transparency.
Miguel has a much longer 2018 article on the same topic, co-authored with Garret Christensen (also University of California, Berkeley), published in the Journal of Economic Literature (open access). The 2018 article covers much of the same ground, but in more detail. Christensen and Miguel begin by outlining the problems with the empirical literature, which they see as:
...publication bias, specification searching, and an inability to replicate results.
They survey the literature on each issue. First, publication bias occurs where the statistically significant results are more likely to be published than statistically insignificant results. This is well known already, and there is plenty of evidence of this bias across multiple fields, including economics. Christensen and Miguel summarise this literature, but pointedly note that:
Of course, and not to be facetious, one cannot completely rule out publication bias even among this body of publication bias studies.
Indeed. Second, specification searching occurs when researchers selectively report only some of the analyses that they have conducted (specifically, the analyses that lead to statistically significant results), while ignoring any other analyses that were conducted. Since some analyses will lead to statistically significant results simply by chance, specification searching means that we cannot necessarily believe the results of published studies. Christensen and Miguel single out the analysis of sub-groups as a particularly problematic type of specification searching (and one that is a pet hate of mine). The more sub-groups that researchers analyse, the more likely they are to turn up something that is statistically significant.
Third, the replication crisis is well-known in psychology, but economics has been facing a replication crisis of its own. Christensen and Miguel outline many of the challenges that prevent sensible replication in the first place (e.g. data unavailability), and note the different types of replication that are available to researchers. However, the problem continues to be that replications carry very little weight in the research literature. They are barely cited, and few credible researchers are willing to spend their scarce time and resources on replicating the work of others rather than doing their own new research.
Finally, Christensen and Miguel present some potential solutions to ensure greater transparency and reproducibility in economics, as well as improving the credibility of reported results. These include: (1) improved research design and greater use of meta-analyses (with improved tests for publication bias); (2) making appropriate corrections for multiple testing when many outcome variables, or many specifications, are used; (3) adopting pre-analysis plans that specify variables and model specifications in advance (which was a focus of the more recent JEP papers in my previous post); (4) improving disclosure and reporting standards; and (5) more open data and materials, to enable replication of published results. I think that they could have gone a step further by advocating for more open access to published research, as well as open peer review (both pre-publication and post-publication). Open peer review is scary (both for authors and for reviewers), but does enable more effective critique of (as well as support for) published research.
It is difficult to argue with any of the recommendations that Christensen and Miguel make. The sad thing is that, several years on, we don't seem all that much closer to achieving the goal of transparent, reproducible, and credible economic research.
[HT: Jacques Poot]
Read more:
No comments:
Post a Comment