...researchers are taught to be open to new ideas, research is all about finding new things and being aware of flaws in existing paradigms—but researchers can be sooooo reluctant to abandon their own pet ideas...
My story goes like this. As scientists, we put a lot of effort into writing articles, typically with collaborators: we work hard on each article, try to get everything right, then we submit to a journal.
What happens next? Sometimes the article is rejected outright, but, if not, we’ll get back some review reports which can have some sharp criticisms: What about X? Have you considered Y? Could Z be biasing your results? Did you consider papers U, V, and W?
The next step is to respond to the review reports, and typically this takes the form of, We considered X, and the result remained significant. Or, We added Y to the model, and the result was in the same direction, marginally significant, so the claim still holds. Or, We adjusted for Z and everything changed . . . hmmmm . . . we then also though about factors P, Q, and R. After including these, as well as Z, our finding still holds. And so on.
The point is: each of the remarks from the reviewers is potentially a sign that our paper is completely wrong, that everything we thought we found is just an artifact of the analysis, that maybe the effect even goes in the opposite direction! But that’s typically not how we take these remarks. Instead, almost invariably, we think of the reviewers’ comments as a set of hoops to jump through: We need to address all the criticisms in order to get the paper published.
Gelman argues that there is a problem with the way that researchers are trained to deal with peer review. Researchers deal with peer review by making the minimal number of changes necessary in order to ensure publication, Gelman argues that that needs to change. I don't disagree. However, I think he is missing something important about researchers, as real people.
People are loss averse. We value losses much more than equivalent gains (in other words, we like to avoid losses much more than we like to capture equivalent gains). Loss aversion makes people subject to the endowment effect - we are unwilling to give up something that we already have, because then we would face a loss (and we are loss averse). Or at least, there would have to be a big offsetting gain in order to convince us to give something up that we already have. The endowment effect applies to objects (the original Richard Thaler experiment that demonstrated endowment effects gave people coffee mugs), but it also applies to ideas.
I've thought for a long time that ideology was simply an extreme example of the endowment effect and loss aversion in practice. Haven't you ever wondered why it's so difficult to convince some people of the rightness of your way of thinking? It's because, in order for them to agree with you, that other person would have to give up their own way of thinking, and that would be a loss (and they are loss averse). It seems unlikely that the benefits of agreeing with you are enough to offset the loss they feel from giving up their prior beliefs, at least for some people. Once you consider loss aversion, it's easy to see how ideologies can become entrenched. An ideology is simply lots of people suffering from loss aversion and the endowment effect.
Now, back to peer review. Researchers are also loss averse, and when they submit an article for publication, the article includes their own ideas, analysis, and conclusions. They don't want to give those up, since that would be a loss (and researchers, like everyone else, want to avoid losses). So, it is natural for researchers to deal with reviewers' comments in a way that minimises the sense of loss that they feel from making the necessary changes (while preserving the gains from getting the article finally published). Making small, incremental changes to a research paper minimises the loss the researchers feel, and so that is the way that researchers deal with peer review.
How to solve the problem? Gelman's solution appears to be better training for researchers. However, I can't see how you can train someone out of being loss averse, which means that we are stuck with researchers who will, as Gelman says, "respond to legitimate scientific criticism in an angry, defensive, closed, non-scientific way". Maybe instead it's time to reconsider peer review? Maybe peer review should exist only to screen out the most egregious errors and not to quibble over minor details (most of which can easily be quibbled over later in online commentary on the paper)?
[HT: Marginal Revolution]
