The marginal benefit of peer review

I engage in a lot of peer review. Last year, I reviewed 24 papers submitted to 18 different journals. The year before was similar. And some of those article went through multiple rounds of peer review. I typically avoid asking for anything new from authors in the second or subsequent rounds of peer review, but I have heard horror stories from colleagues of papers sitting in peer review for years, while reviewers ask for revision after revision, all the time with new analyses or robustness checks being appended to the paper. A legitimate question is, does all of that extra peer review time add value?

A recent IZA Discussion Paper by Aboozar Hadavand (Johns Hopkins University), Daniel Hamermesh (Columbia University), and Wesley Wilson (University of Oregon), provides us with an initial answer. They collated data from the journal Economic Inquiry over the period from 2009 to 2018. Interesting, Economic Inquiry offers submitting authors two tracks:

Submitting authors could choose between a fast track, in which the article receives a simple yes or no; or a regular track, which might lead to acceptance with minor revisions, to a revise/re-submit response with subsequent additional refereeing, or to rejection...

Hadavand et al. then compare articles that were published through the fast track (which didn't suffer multiple rounds of peer review) with articles that were published through the regular track (which could have). Of course, authors self-select into the track they submit to, so the authors use a two-stage selection model to deal with that. When looking at the subsequent citation performance of articles (which is an indicator of article quality), they find that:

...no matter what vectors of covariates are included in this second stage, those regular-track articles that go through multiple rounds of refereeing have no greater scholarly impact than those that obtain only one set of referee reports... Compared to regular-track articles that go through only one round of refereeing, multiple rounds of refereeing never enhance an article’s subsequent scholarly impact in any of the econometric formulations that we have constructed.

In other words, the additional rounds of peer review seem to add no value (in terms of the article quality). To some extent, this makes sense. Economists recognise that marginal benefit is diminishing. So, each additional round of peer review should add less benefit than earlier rounds. However, I think many people would be surprised to know that the marginal benefit basically immediately falls to zero, and may even be zero for the first round of peer review.

This calls into question the whole cost-benefit calculation of peer review overall. Hadavand provide a rough estimate that the time cost of peer review in economics is US$50 million per year, and US$1 billion in total across all disciplines. How much of that cost is essentially a waste? It would be good to get a sense of whether this study is an outlier, or whether it can be replicated for other journals, and perhaps more importantly, in other disciplines.

[HT: Marginal Revolution and Development Impact]