Discrimination on #EconTwitter

I've written a number of times about correspondence experiments designed to identify discrimination in labour markets. In such an experiment, the researcher applies for a bunch of jobs, using fake job 'applicants' that differ only on the basis of some known characteristics (gender, for example). The difference in callback rates (or some other similar measure) between applicants with different characteristics provides a measure of discrimination on the basis of those characteristics.

A new application of this approach instead looks at discrimination on #EconTwitter (on X). The research is reported in this 2025 article by Nicolás Ajzenman (McGill University), Bruno Ferman (Sao Paulo School of Economics), and Pedro Sant’Anna (MIT), published in the journal American Economic Review: Insights (ungated earlier version here). In their experiment, Ajzenman et al. created 80 'bot' accounts on X between May and August 2022 that mimicked real PhD student accounts, but differed in terms of gender (male or female), race (Black or White), and university affiliation (top-ranked [top ten in the 2017 US News ranking of economics graduate programmes] or lower-ranked [ranked 79-100] university). Each bot account was active for twelve days, during which time it initially randomly retweeted posts from economics journals to establish its credibility, then followed a random selection of 100 members of the #EconTwitter community (from a pool of over 10,000 X users who had tweeted or retweeted content using the #EconTwitter hashtag between January and February of 2022).

Ajzenman et al. then measure the number of times each account is followed back by the user it followed, and look at the difference in follow-back rates for bot accounts with different characteristics (gender, race, and university affiliation). The raw follow-back rates are shown in Figure 1 in the paper:

Bot accounts that presented as Black males from lower-ranked universities received the lowest follow-back rate of 14.4 percent, while bots presenting as White females from top-ranked universities were followed back 23.9 percent of the time. In their regression models, Ajzenman et al. find that:

Users in the #EconTwitter community are 2.1 percentage points (12 percent) more likely to follow White than Black PhD students, 3.5 percentage points (21 percent) more likely to follow students from top-ranked universities than those from lower-ranked ones, and 4.3 percentage points (25 percent) more likely to follow female than male students. We also find that the racial gap in follow-backs remains among students claiming to be from top-ranked universities. This suggests that racial discrimination persists even in the presence of a signal indicative of higher academic potential.

Turning to how the results vary based on the characteristics of the X users that were followed, Ajzenman et al. find no differences by gender or race, or by the reach of the X user (measured by the number of followers they have). They also find no difference between X users who have demonstrated concern about the lack of diversity in economics (by following one or more X accounts devoted to the topic) and those that haven't. However, they also find that:

...subjects who signal concern about the lack of diversity in economics discriminate the most in terms of university affiliation. While both groups of subjects favor students affiliated with top-ranked institutions, the difference in follow-back rates between top- and lower-ranked students is considerably larger among concerned subjects (8.6 percentage points, against 2.5 percentage points for the remaining subjects, p-value of difference < 0.05)...

So, concern about demographic diversity clearly doesn't imply an absence of other forms of discrimination, as these results seem to show that those members of the #EconTwitter community who are most concerned about diversity discriminate more against students from lower-ranked institutions than users who are less concerned.

Given the longstanding gender bias in economics, the higher follow-back rates for bot accounts presenting as female are perhaps the most surprising result. Ajzenman et al. suggest several possible explanations. First, some users may be conscious of the barriers women face in the profession and therefore make a deliberate effort to engage with them. Second, some users may be using X partly to establish social rather than professional relationships, which would suggest a more negative interpretation of the higher follow-back rate. Ajzenman et al. also suggest two more strategic explanations. Researchers may see advantages in collaborating with women because female economists tend to receive less credit for joint work. Alternatively, a woman gaining admission to a highly ranked PhD programme may be providing a stronger signal of ability, given the additional barriers women face in reaching that point. The experiment cannot distinguish among these explanations, so teasing out which mechanisms are more important would require further, more detailed, research.

And no doubt Ajzenman et al. had hoped to conduct some more detailed research than what they reported, but their data collection had to be cut short. As they explain:

We planned to run 30 experimental waves between May and December 2022, which would have given us more than enough power to identify reasonable effects. This was to account for potential problems, such as X blocking some accounts. We stopped earlier because an X user saw some of the accounts and posted about the experiment during the eleventh wave, which compromised the continuity of the experiment.

Sometimes research just doesn't go as planned, and that might explain why this research got published in the more modest AER: Insights journal, rather than the top-five journal the authors probably were initially hoping for.

Sometimes research just doesn't go as planned. However, Ajzenman et al. had already collected enough data to reveal a substantial amount of discrimination on #EconTwitter. Perhaps the most important result is that signalling concern about diversity doesn't necessarily make people immune to other forms of bias, especially in terms of academic prestige.

[HT: Marginal Revolution]

Computer gaming and binge drinking may be complements, not substitutes

In economics, two goods are substitutes if consumers tend to consume more of one if the price of the other increases. One way of thinking about that is that if the price of Good X increases, consumers switch to purchasing Good Y instead, and the quantity of Good Y demanded increases. Two goods are complements if consumers tend to consume less of one if the price of the other increases. In this case, if the price of Good X increases, consumers buy less of Good X (due to the Law of Demand), but also buy less of Good Y, and the quantity of Good Y demanded decreases.

Whether a pair of goods are substitutes or complements is determined by the cross-price elasticity of demand: the responsiveness of the quantity demanded of one good to a change in the price of the other good. If the cross-price elasticity is positive, the two goods are substitutes. If the cross-price elasticity is negative, the two goods are complements. Another way of thinking about this is that, following a change in the price of one good, ceteris paribus (holding all else constant), we would expect the quantities demanded of substitutes to move in opposite directions, while the quantities demanded of complements would move in the same direction.

There are obvious examples of substitutes and complements. Coke and Pepsi are the iconic example of substitute goods used in almost every introductory economics class. An example of complements that I use in my classes is video game consoles and games. However, it isn't always straightforward to determine whether a pair of goods are substitutes or complements. Sometimes they may be substitutes in one context, but complements in another. So, whether goods are substitutes or complements is an empirical question.

Take the example of computer gaming and binge drinking. When I was growing up, those two 'goods' certainly seemed like complements. My friends and I spent many nights drinking beer or RTDs and playing hotseat turn-based strategy games like Robosport, Warlords II, or Heroes of Might and Magic.[*] That experience made me a little surprised to see the hypothesis in this 2021 article by Torleif Halkjelsvik, Geir Brunborg, and Elin Bye (all Norwegian Institute of Public Health), published in the journal Drug and Alcohol Review (open access), which was that binge drinking and computer gaming are substitutes. Now, modern computer gaming differs in meaningful ways from how it looked when I was young. Nevertheless, I was surprised that Halkjelsvik et al. hypothesised in the direction they did.

Their hypothesis rested on several ideas, and was motivated by the observed increase in gaming and decrease in alcohol consumption by young people over time. First, alcohol and gaming are both outlets for thrill seeking, and are both responses to boredom, so increasing computer gaming might reduce the need for drinking. Second, both drinking and computer gaming are sources of social bonding, so again more computer gaming reduces the need for drinking.

Halkjelsvik et al. test their hypothesis with data from the European School Survey Project on Alcohol and Other Drugs (ESPAD), which surveys 15 and 16-year-old students every four years. They use data from 23 countries over the period from 1995 to 2015 (although noting that not all countries are part of the survey in every year), and look at the correlation between frequency of binge drinking (drinking five or more drinks on an occasion) and frequency of computer gaming, using a multi-level linear probability model. If their hypothesis that gaming displaces drinking is correct, the relationship should be negative. However, Halkjelsvik et al. find that:

...the association between country-level changes in computer gaming and binge drinking was estimated as positive...

So, increases in the average frequency of computer gaming at the country level tended to be associated with increases in the frequency of binge drinking. And, at the individual level:

The between individual-effect was positive, suggesting a four percentage point (±2 percentage points) higher binge drinking prevalence among students who report playing computer games daily.

Of course, the analysis that Halkjelsvik et al. conducted doesn't establish a causal relationship, it only shows correlations. And, importantly, they aren't directly testing whether computer gaming and binge drinking are complements in the economic sense, as that would require looking at how consumption of one responds to changes in the price of the other. However, their results are at least consistent with computer gaming and binge drinking being complements. Rather than moving in opposite directions, as we might expect if gaming displaced drinking (as Halkjelsvik et al. hypothesised), gaming and binge drinking tend to move in the same direction. Which, admittedly on the basis of a rather smaller and less representative sample, my friends and I could have told them.

*****

[*] My kids are bemused at the very idea that there was ever such a thing as hotseat multiplayer games. Sadly, they gradually died out as online games became more widely available in the early 2000s. However, they were really good for multi-tasking with some tabletop gaming at the same time, since only one player played the hotseat game at a time.

Taking advantage of loss aversion in education

Many years ago (I forget exactly when), I introduced extra credit into my ECON110 class (which is what is now ECONS102). The idea was to provide an incentive for students to attend class, since they could earn extra credit for completing various in-class exercises. A couple of years later, I briefly changed the way that I framed the extra credit, from being "extra marks that would be gained from attending", to "extra marks that would be lost by not attending".

If students were purely rational, the change from 'gain framing' to 'loss framing' the extra credit should have had no impact on student attendance. However, I was looking to exploit the fact that most people are quasi-rational, rather than purely rational. Quasi-rational decision-makers are loss averse, meaning that they value losses more than equivalent gains. For a loss averse person, losing $20 makes them unhappy to a greater extent than winning $20 makes them happy.

Does a change from 'gain framing' to 'loss framing' work? That is the question that this new article by Antal Ertl, Éva Holb (both Eötvös Lóránd Science University), and Barna Bakó (Corvinus University of Budapest), published in the Journal of Economic Behavior and Organization (open access), tries to answer. They use data from a field experiment at Corvinus University of Budapest, where students enrolled in a compulsory macroeconomics course for business students were randomised into one of three conditions: (1) Gain group, which earned points in each of four tests and the final examination as usual; (2) Loss group, which started each test and the final exam with full points, but had points deducted for each incorrect answer; and (3) Hybrid group, which was the same as the Gain group for the tests, but switched to the loss framing for the final examination.

Ertl et al. have a sample of 321 students who consented to be part of the research, completed an initial questionnaire at the start of the term, and earned a non-zero grade. Randomisation was conducted at the level of the tutorial group (so all students in a tutorial were in the same treatment), in such a way that each teacher had groups across more than one treatment. One wrinkle in their analysis is that the best three out of the four tests would count towards a student's grade, meaning that students may end up putting differential effort into each test, depending on how they have performed in the other tests already completed. So, in addition to looking at the effect of treatment on each test mark individually, Ertl et al. look at the effect on the 'best three' tests collectively, as well as the exam mark.

If randomisation were perfect and the treatment groups were balanced, the comparison between the Loss group and the Gain group would demonstrate the overall effect of loss framing on student performance. The comparison between the Loss group and the Hybrid group for the final exam, compared with the same comparison for the best three tests, would demonstrate whether students adjust in such a way that the loss framing has less impact over time (because the Hybrid group would be in their first loss-framed assessment, while the Loss group would be in their fifth such assessment). The treatment groups weren't perfectly balanced, with students sorting into tutorial groups in part based on whether they worked part-time. So, Ertl et al. control for working part-time, the tutorial day and time, and the tutorial group teacher, as well as other demographic and background variables.

In their main analysis, they find support for the positive effects of loss framing:

For the Loss treatment, the effect on the average of the Best 3 Tests is 3.2 percentage points, although the difference is not statistically significant. The treatment effect on the Final Test score, however, shows a large difference of 9.6 percentage points when not controlling for Best 3 Tests’ scores, i.e., how well students did throughout the semester before the Final Test.

After controlling for performance in the best three tests, the effect of the loss framing on performance in the final examination is a statistically significant 7.8 percentage points. Turning to the comparison of the Loss and Hybrid groups, Ertl et al. find that:

...the estimated effect sizes for Loss and Hybrid are essentially the same for the Final Test, once we take into account how well students did perform throughout the semester...

These results are consistent with loss framing leading to better student performance, and there being no novelty effect - the effect of loss framing doesn't appear to decline over time. Ertl et al. go on to show that the effects are similar for both male and female students, but larger for students who did not take advanced mathematics in high school than for those that did. They also show that the treatment did not seem to negatively affect students' perceptions of the course, because the teaching evaluations were similar for the different treatment groups.

Finally, Ertl et al. do provide a note of caution in their conclusion:

previous studies have highlighted possible psychological and motivational costs associated with loss framing... These findings suggest that the mechanism by which loss framing improves performance may, at least in part, operate through heightened tension and concern about avoiding mistakes rather than through enhanced intrinsic motivation. Moreover, in extreme cases, loss-framed grading may even produce adverse effects — for example, low-performing students might become discouraged early in the semester after ‘‘losing’’ too many points. Once it becomes apparent that only a passing grade is attainable at best, the loss-framed structure may make this limitation increasingly salient, potentially exacerbating anxiety and disengagement. Over time, this could have broader implications for students’ well-being and their willingness to enroll in courses or programs that employ such systems.

Ertl et al. don't directly test for these effects, but they should be a concern. We may be able to improve student performance through loss-framing assessments, but that might come at a cost to student mental health and wellbeing.

And that brings me back to the example I started with, from my ECON110 class. When I switched extra credit from gain-framed to loss-framed, student attendance in class did improve slightly. However, the bigger impact seemed to be the number of students who would contact me by email, seeking special consideration for missing the extra credit, offering to provide medical certificates or other evidence to explain their absence, and asking for extra chances to complete the in-class exercises. It turned out to be administratively much more costly for me, and so the change was short-lived (to the extent that I cannot even remember which year I tried this in). Those reactions could suggest a negative psychological effect of the switch from gain framing to loss framing.

So, not all interventions that are effective for promoting student performance should be adopted. We need to carefully consider both the benefits and the costs of the intervention first. Taking advantage of student loss aversion might be worth exploring further, but I would want to see a wider evaluation that included student wellbeing outcomes before adopting it.

This week in research #139

Here's what caught my eye in research over the past week (a quiet week, it seems):

• List (open access) comments on how to address the generalisability of research
• Lehner et al. (open access) find that the opening of a Walmart Supercenter is associated with a 2.2 percentage point (18%) increase in poverty, and that the increase is largest for younger and less-educated adults

And the latest paper from my own research (led by my former PhD student Muhammad Irfan, along with Ushan Goonawardane, and Craig Robertson), which was also covered in the New Zealand Herald:

• Our new article (open access if you register for free) in the New Zealand Medical Journal performs a comparison of methamphetamine contamination of 423 properties across New Zealand, before and after the requirement to test for methamphetamine was eased in May 2018, and finds a significant increase in methamphetamine contamination