This week in research #74

Here's what caught my eye in research over the past week:

• Nguyen (with ungated earlier version here) finds that working hours per week are reduced by 0.38% for a 1% increase in the minimum wage in Vietnam
• Nasser and Deutscher find a 15 percentage point decrease in attendance at women's football matches in Germany when there are overlapping match days between women’s and men’s matches, with a slightly larger effect for matches of the same team for women’s and men’s matches (for spectators, women's football and men's football are substitutes)

Also new from the Waikato working papers series:

• Luengo et al. investigates the effect of continuous, free-form communication among traders on mispricing in experimental asset markets, and find that, contrary to expectations, communication has limited effectiveness, with only a slight reduction in mispricing observed in the most complex asset scenario

The disturbing lack of impact of comments and replications of economics research

Back in 2018, I wrote a comment on an article that I was the reviewer for. The article (open access) described four types of 'economic citizen'. My comment (open access) argued that one of the four types was not distinct from the other three, but instead captured a different dimension of economic citizenship that might apply to any of the other three types. The comment was published alongside the article, along with a reply (open access) by the original article's authors, in the journal Education Sciences. What has happened since is that, according to Google Scholar, the original article has been cited 23 times. The comment has been cited just one, by the authors' reply, which has itself never been cited.

Now it may be that my critique of the original article is not valid, and so subsequent authors citing the original article don't feel the need to cite the comment. Or, it could be that the critique is valid (I still think it is), but is being ignored for some reason. If this is a common experience for research, then it calls into question whether research is self-correcting or not. If research findings are not easily overturned or challenged, then future researchers may be wasting a lot of time and effort on research that will not advance the field, being based on questionable earlier research.

How big a problem is this? That is the question addressed in this new article by Jörg Ankel‐Peters, Nathan Fiala, and Florian Neubauer (all Leibniz Institute for Economic Research), published in the journal Economic Inquiry (open access). Ankel‐Peters et al. look at all 56 replications published as comments between 2010 and 2020 in the American Economic Review, arguably the overall top journal in economics (and certainly in the top five journals). As they explain:

For the self‐correction claim to hold, we hypothesize that a comment should lead to a strong reaction of the literature, especially for a comment raising substantive concerns about an OP. If it does not respond strongly, we argue, the prior in the literature sustains. We look at two facets of a strong response: (1) Citations of the comment relative to citations to the OP after comment publication (henceforth: citation ratio), and (2) Whether the comment affects the OP's annual citations.

Ankel‐Peters et al. don't conduct formal statistical tests, and instead rely on a descriptive analysis. Nevertheless, the results are compelling:

We find that AER comments do not affect the OP's [Original Paper's] citations and hence their influence on the literature. We observe an average citation ratio of 14%. Comments are cited on average seven times per year since their publication—compared to an average of 74 citations per year for the OP since publication of the comment. Comments are, hence, not cited much in absolute terms, and a lot less than the OP. The latter implies that most OP citations ignore the comment.

They also find very similar results when they focus on citations in the top five economics journals, and similar results when they account (subjectively) for whether the comment 'must be cited' alongside the original paper because of how substantive the concerns raised are. Ankel-Peters et al. conclude that:

We interpret this as evidence for the absence of self-correction mechanisms in economics.

Obviously, that is disappointing. Replications are seen as important to ensuring the integrity and credibility of research (for example, see here). If papers that have failed to replicate, or that have substantive problems with them, are being cited uncritically in the following literature, then economics research could easily be led down some dead-end paths. What can be done to mitigate this problem? Ankel-Peters et al. don't offer much in the way of solutions. However, the bare minimum would be that comments and replications should receive more prominence so that readers of the original research are aware of them. To this end, Ankel-Peters et al. report a very small victory:

In response to a previous version of our paper, the current AER editor has let us know that the journal has changed its policy and now, for new comments, will provide a link on the OP's website. This is a small but perhaps important first step to giving replication work in economics the attention it needs and deserves.

Indeed. Now all journals need to follow AER's lead (which isn't much of a lead - many journals already do that). A better solution would be if top journals required citation of relevant high-quality comments or replications whenever an original paper is cited. And it would be even better if top journals published more high-quality replications and comments, as well as more high-quality systematic reviews and meta-analyses. Then we might advance economics research in a more informed way.

The effect of the Dobbs US Supreme Court decision on medical school applications

In 2022, the US Supreme Court ruling in Dobbs v. Jackson Women's Health Organization effectively overturned the famous 1973 Roe v Wade decision, returning the decision about abortion policy to individual states. Many states had laws that would ban abortion if Roe v Wade was ever overturned, and those laws immediately went into effect. Other states responded by enacting legislation to protect the availability of abortion. Overall, the landscape for medical care in the US changed overnight, with different effects in different parts of the country.

How did that change affect the incentives to become a doctor? That is effectively the question addressed in this 2024 article by Joshua Hess (University of South Carolina), published in the journal Economics Letters (ungated earlier version here). Hess looks at the effect of the decision on medical school applications, focusing on the change between years before 2022 (before the Dobbs decision) and 2022 and 2023 (after the Dobbs decision), for states with and without abortion restrictions after the Dobbs decision. In other words, Hess applies a difference-in-differences analysis (evaluating the difference between states with and without abortion restrictions on their books, between the time before and the time after the Dobbs decision, when those restrictions would become effective).

Looking at the share of women applying to medical schools, Hess finds that:

Women’s share of applications to treated schools increased by 0.65% (p-value = 4.8×10−06) relative to control schools in 2022 and by 1.17% (p-value = 3.2 × 10−06) in 2023.

So, Hess finds that women make up a larger share of medical school applications in treated states. I find that a little surprising. But then things get a little weirder, because:

...there is little change in total enrollments over the time frame. It increased from 21,507 in 2018 to 22,845 in 2023. Consequently, changes in women’s (men’s) share strongly imply a similar change in the number of women (men) medical students.

In other words, taking those two results together, states that have abortion bans must have gained more medical school applications from women and fewer applications from men, after the Dobbs decision (and relative to states without abortion bans).

It is difficult for me to understand the decision-making of medical school applicants here. However, I can offer some idle speculation. In states with abortion bans, the status of health care as a profession declined after the Dobbs decision. Moreover, the status of women's health care declined even more, making women's health less attractive as a specialty, compared with other specialties. There is some weak supporting evidence for this, as Hess finds that:

...the share of women applicants to top OB/GYN schools in states with abortion bans increased by nearly 1% more than not top OB/GYN schools in states with abortion bans...

OB/GYN is a medical specialty that attracts more women than men, and the larger effect there than overall is consistent with reducing the status of women's health care. However, if the decrease in the status of health care as a profession was explaining these results, then the total number of medical school applications should decline, when Hess notes that it hasn't changed. Instead, more women are being attracted to medical school, and fewer men.

Are the outside options (the next best alternative to medical school) better for men than for women? I don't know the answer to that question, but even if that were true it would only explain the decrease in male medical school applicants, and not the corresponding increase in female applicants.

That leaves me with one last possible explanation: that women responded to the threat to women's health in states with abortion bans by deciding to become doctors. That would be an effective way of fighting to protect women's health, a form of resistance. So, even as men become less likely to apply to medical school due to the reduction in status of medicine (especially in women's health care), women respond by applying to medical school in greater numbers.

Hess isn't able to extricate the reasons underlying the results he finds. So, my takeaway from this paper is that it raises more questions than it answers. Clearly, this is not the final word on this topic, and hopefully there is other research going on that will help us to better understand why there has been such a gender shift in medical school applications.

The mental health of economics PhD students and staff in Europe

Back in 2021, I wrote a post about the mental health of PhD students in economics. It was based on two studies and this Substack post by Scott Cunningham. The conclusion was that economics PhD students were suffering, but perhaps no more so than PhD students in other disciplines. However, patting ourselves on the back for being no worse than any other discipline seems like a failure to me, especially when many students are genuinely in mental health crises.

The study in Cunningham's post that was focused exclusively on economics PhD students was US-based, so it is worth wondering if the results apply elsewhere. This new article by Elisa Macchi (Brown University) and co-authors, forthcoming in the American Journal of Health Economics (ungated earlier version here), provides an answer to that, being based on data from 14 top European economics departments. The study uses a similar methodology to the US study that Cunningham discussed, and two of the authors (Valentin Bolotnyy and Paul Barreira) are the same. So, these studies are about as comparable as they can get. However, this new study also looks beyond PhD students, also considering the mental health of staff in economics departments.

Specifically, Macchi et al. got survey responses from 556 students and 255 staff, from 14 universities across Europe:

...Bocconi University, Bonn Graduate School of Economics, Central European University, European University Institute, London School of Economics, Mannheim Graduate School of Economics, Paris School of Economics, Sciences Po, Stockholm School of Economics and Social Sciences, University College London, Universitat Pompeu Fabra, University of Warwick, University of Zurich, and Uppsala Universitet.

It's worth noting that most of the top-ranked European economics departments are included in that list. Notable exceptions are Toulouse School of Economics, Oxford University, and Barcelona School of Economics (all ranked in the top ten in Europe, according to RePEc). Macchi et al. explain that the "restricted our interest to Economics departments that offer a cohort-style PhD program, where graduate students are admitted in cohorts to a graduate school, rather than following a chair-style model. That might explain the exclusion of other top universities from the sample.

The surveys were quite detailed, and in terms of mental health they included commonly used measures of depression, anxiety, suicidality, loneliness, and 'imposter phenomenon'. The last of these deserves a bit more explanation, and Macchi et al. note that imposter phenomenon:

...is a condition in which one feels like a fraud and worries about being found out. Individuals experiencing imposter phenomenon do not believe that their success is due to their competence, but rather ascribe success to external factors such as luck. Those experiencing imposter phenomenon often experience fear, stress, self-doubt, and discomfort with their achievements. Imposter fears interfere with a persons ability to accept and enjoy their abilities and achievements, and have a negative impact on emotional well-being...

Many PhD students (and indeed, many academic staff) can probably relate to that. Given the range of measures employed, the two samples (students and staff), and the comparisons with the US sample (where enabled by the use of the same questions), the paper has a huge amount of detail, and so it's difficult to excerpt from. The relevance of the comparisons with the US are somewhat limited because Macchi et al. conducted their survey starting in November 2021, when many people were still feeling the mental health impacts of the COVID-19 pandemic. However, Macchi et al. note attempt to establish how much of the difference in results (for depression and anxiety) relate to the pandemic.

The headline results are that there are:

...high rates of depression and anxiety symptoms, as well as suicidal or self-harm ideation, loneliness, and imposter phenomenon among graduate students in European Economics departments. 34.7% of graduate students experience moderate to severe symptoms of depression or anxiety and 17.3% report suicidal or self-harm ideation in a two-week period. 59% of students experience frequent or intense imposter phenomenon.

And in comparison with the US sample:

The prevalence of severe and moderate depression and anxiety symptoms in our sample of European Economics graduate students is notably higher than in the 2017-2018 sample of graduate students from top Economics departments in the U.S. (Bolotnyy, Basilico, and Barreira 2022) and higher than in a meta-analysis of depression, anxiety, and suicidal ideation among PhD students prior to the COVID-19 pandemic (Satinsky et al. 2021).

The Satinsky et al. paper is the other research that Cunningham referred to in his Substack post that I mentioned earlier. So, European PhD students have worse mental health that US PhD students. However, how much of that is due to the pandemic? Macchi et al. use data on the trends in mental health among Harvard University students, and note that:

...we can attribute approximately 74% of the difference in the prevalence of moderate-severe depression and 30% of the difference in the prevalence of moderate-severe anxiety between our European sample and the 2017-2018 U.S. sample to the impact of the COVID-19 pandemic.

So, the differences in mental health were not entirely driven by the pandemic. European PhD students do indeed appear to suffer more from depression and anxiety than US PhD students. What about staff though? Macchi et al. find that:

In our faculty sample, the prevalence of severe and moderate anxiety is on average lower than graduate students as well as than comparable statistics for the post COVID-19 European population. This average, however, hides a substantial heterogeneity by seniority level. Untenured tenure-track faculty in Europe are as likely to experience depression and anxiety symptoms as graduate students in our sample, and non-tenure track faculty show even higher prevalence of depression or anxiety symptoms. In contrast, the prevalence of depression and anxiety symptoms among European tenured faculty in our sample is about 70% lower than among their graduate students and is well below the comparable rates in the post-pandemic European population.

That makes a lot of sense. Un-tenured junior academics face many of the same workload and other pressures that PhD students do. Senior and tenured academics do not. So, it shouldn't be a surprise that there is a demonstrable difference in mental health measures between junior and senior academic staff.

Macchi et al. then turn to other results from their survey, showing that:

...25.9% of students in the European sample report having experienced at least one form of sexual harassment. Excluding a form of harassment not included in the U.S. study, the sexual harassment prevalence rate in our European graduate student sample (19.5%) is comparable to the U.S. sample (19.4%).

Again, that is not good. And worryingly:

...European Economics PhD students with moderate-severe symptoms of depression or anxiety are less likely to be in treatment (19.2%) than Economics PhD students in U.S. top departments (25.2%).

That difference in access to treatment may explain some of the differences in mental health between European and US PhD students. That also leads to the first of several recommendations that Macchi et al. make (which I think should be read alongside the recommendations that Bolotnyy et al. made for the US study, which I outlined in this post). Macchi et al. recommend that: (1) the usage of mental health services by students and staff be normalised and enabled; (2) that sexual harassment be addressed; (3) that relationships between students and their advisors be improved; and (4) more structure be offered in PhD programmes to avoid students getting into ruts. I think we can and should support all of those recommendations, and they're certainly something that would help PhD students, not just in Europe and not just in economics, but more generally.

[HT: Marginal Revolution, back in 2023]

Read more:

• Scott Cunningham on the mental health of PhD students in economics