Back in 2018, I wrote a comment on an article that I was the reviewer for. The article (open access) described four types of 'economic citizen'. My comment (open access) argued that one of the four types was not distinct from the other three, but instead captured a different dimension of economic citizenship that might apply to any of the other three types. The comment was published alongside the article, along with a reply (open access) by the original article's authors, in the journal Education Sciences. What has happened since is that, according to Google Scholar, the original article has been cited 23 times. The comment has been cited just one, by the authors' reply, which has itself never been cited.
Now it may be that my critique of the original article is not valid, and so subsequent authors citing the original article don't feel the need to cite the comment. Or, it could be that the critique is valid (I still think it is), but is being ignored for some reason. If this is a common experience for research, then it calls into question whether research is self-correcting or not. If research findings are not easily overturned or challenged, then future researchers may be wasting a lot of time and effort on research that will not advance the field, being based on questionable earlier research.
How big a problem is this? That is the question addressed in this new article by Jörg Ankel‐Peters, Nathan Fiala, and Florian Neubauer (all Leibniz Institute for Economic Research), published in the journal Economic Inquiry (open access). Ankel‐Peters et al. look at all 56 replications published as comments between 2010 and 2020 in the American Economic Review, arguably the overall top journal in economics (and certainly in the top five journals). As they explain:
For the self‐correction claim to hold, we hypothesize that a comment should lead to a strong reaction of the literature, especially for a comment raising substantive concerns about an OP. If it does not respond strongly, we argue, the prior in the literature sustains. We look at two facets of a strong response: (1) Citations of the comment relative to citations to the OP after comment publication (henceforth: citation ratio), and (2) Whether the comment affects the OP's annual citations.
Ankel‐Peters et al. don't conduct formal statistical tests, and instead rely on a descriptive analysis. Nevertheless, the results are compelling:
We find that AER comments do not affect the OP's [Original Paper's] citations and hence their influence on the literature. We observe an average citation ratio of 14%. Comments are cited on average seven times per year since their publication—compared to an average of 74 citations per year for the OP since publication of the comment. Comments are, hence, not cited much in absolute terms, and a lot less than the OP. The latter implies that most OP citations ignore the comment.
They also find very similar results when they focus on citations in the top five economics journals, and similar results when they account (subjectively) for whether the comment 'must be cited' alongside the original paper because of how substantive the concerns raised are. Ankel-Peters et al. conclude that:
We interpret this as evidence for the absence of self-correction mechanisms in economics.
Obviously, that is disappointing. Replications are seen as important to ensuring the integrity and credibility of research (for example, see here). If papers that have failed to replicate, or that have substantive problems with them, are being cited uncritically in the following literature, then economics research could easily be led down some dead-end paths. What can be done to mitigate this problem? Ankel-Peters et al. don't offer much in the way of solutions. However, the bare minimum would be that comments and replications should receive more prominence so that readers of the original research are aware of them. To this end, Ankel-Peters et al. report a very small victory:
In response to a previous version of our paper, the current AER editor has let us know that the journal has changed its policy and now, for new comments, will provide a link on the OP's website. This is a small but perhaps important first step to giving replication work in economics the attention it needs and deserves.
Indeed. Now all journals need to follow AER's lead (which isn't much of a lead - many journals already do that). A better solution would be if top journals required citation of relevant high-quality comments or replications whenever an original paper is cited. And it would be even better if top journals published more high-quality replications and comments, as well as more high-quality systematic reviews and meta-analyses. Then we might advance economics research in a more informed way.
No comments:
Post a Comment