Sex, Drugs and Economics: What machine learning is telling us about the gender dynamics in economics seminars

Tuesday, 7 February 2023

What machine learning is telling us about the gender dynamics in economics seminars

Economics seminars have come in for a lot of criticism for apparent gender bias (see here, or for a broader view on gender in economics see the links at the end of this post). To some extent, past research on gender dynamics in economics seminars has been limited by the hand-coding of data, so limited detail on the interactions within the seminar are available to analyse. Not any more, thanks to machine learning, as explained in two new papers.

The first paper, by Amy Handlan and Haoyu Sheng (both Brown University), uses machine learning for audio analysis of presentations at the 2022 NBER Summer Institute. The audio classification algorithm that they use allows them to automatically classify speakers by gender, age, and the tone of their comments. They find that:

...women in the audience are more likely to ask female presenters questions than male presenters, and female presenters are more likely to be assigned female discussants.

This suggests that there is significant gender sorting (although Handlan and Sheng can't say anything about the wider audience in each presentation, because they only have data on audience members who spoke). Interestingly, next they find that:

Regarding interruptions, we find that there are similar number of interruptions for both male and female presenters.

That is somewhat at odds with earlier research (see here, for example). However, more consistent with the earlier research, Handlan and Sheng also find that:

...interruptions for female presenters last longer than those for male presenters...

But what about tone? In that respect, they find:

...gender differences in tone within speakers. On average, female speakers are more likely to sound positive or happy, while male speakers are more likely to sound negative and serious or angry. This holds whether we consider only presenters or only speaking audience members. Furthermore, when we consider how speakers may change their tone over time, we find that tone is highly persistent for both men and women. That is, if you sound happy and are uninterrupted, you are likely to continue sounding happy.

So far, so good. But what happens when speakers are interrupted, or responding to others? In that case:

...we find that speakers sound more negative when responding to women, whether the person is an audience member asking a question to a female speaker or a presenter responding to a female audience member. When we look at the interaction between presenter gender and audience gender, we find that female speakers respond more negatively to other women compared to how they respond to men. The gap in tone responding to men versus women is larger for female speakers than for male speakers.

This finding that women are harsher towards other women than men are towards women, is unfortunately looking increasingly common in economics. Handlan and Sheng suggest that this may be due to societal norms that lead to higher expectations of women, as well as because:

Women may speak more positively to men in an attempt to offset a larger negative bias from men compared to women.

I'd label those reasons speculative for now. Handlan and Sheng aren't able to test them, of course. However, the paper has some interesting insights, and not just about gender. For instance, people (presenters and audience members) in macroeconomics seminars are more likely to sound sad, but less likely to sound angry, than people in microeconomics seminars. Given that this was 2022, historically high inflation might make macroeconomists sad, and microeconomists angry? We need some causal analysis of tone!

Anyway, moving on. The second paper I want to discuss is this job market paper by Mateo Seré (University of Antwerp). His dataset is much broader than Handlan and Sheng's, covering all web-streamed (on YouTube) seminars that were part of a seminar series of a university economics department in the US or Europe, over the period from 2020 to 2022 (as well as the National Bureau of Economic Research (NBER), the American Economic Association (AEA), and the Centre for Economic Policy Research (CEPR)). Seré focuses specifically on interruptions to the seminar, and first notes that:

...the distribution of interruptions made during seminars presented by women is slightly shifted to the right compared to the distribution for males.

Specifically, female presenters receive between 0.9 and 1.5 more interruptions than male presenters, controlling for presenter characteristics and seminar characteristics. That is a large effect, given that the mean number of interruptions is 11. So, female presenters receive about 10 percent more interruptions than male presenters. That is inconsistent with Handlan and Sheng's results, but remember that their results come from the NBER Summer Institute only, whereas Seré is looking at a much broader set of seminar series.

When looking at who does the interrupting, Seré finds that:

...the proportion of female interruptions is significant to explain the overall number of interruptions in a seminar only when the presenter is a woman.

In other words, the additional interruptions that female speakers experience are mostly driven by female audience members. Seré notes that male audience members ask female presenters 0.2 more questions than they ask male presenters, but female audience members ask female presenters 1.0 more questions than they ask male presenters. Given than most audience members are male (and so they ask more questions overall), this is a substantial difference.

Seré then finds that there is a difference in the nature of interruptions between male and female audience members:

Being a female presenter is related with an increase in the number of questions from females in the audience and with a decrease in the ones made by males. Furthermore, while men in the audience make more comments on average when the presenter is female, the gender of the presenter has no effect on the number of comments made by women.

This is interesting, as it is closest to the results on tone from Handlan and Sheng. Questions require a response, whereas comments do not. Looking across both studies, if female audience members ask more questions of female presenters rather than making comments, perhaps those questions come across as more negative in tone?

Together, these two papers paint an interesting picture of some of the dynamics within economics seminars. Clearly, this is just the beginning of research in this area. In particular, Handlan and Sheng have made their algorithm available for others to use for coding and analysing other seminar series. Expect more research in this space in the future.

[HT: Marginal Revolution for the Handlan and Sheng paper, and Development Impact for the Seré paper]