John List on critical thinking and teaching economics

When it comes to buzzwords in education, critical thinking probably ranks at or near the very top. The problem is that not everyone agrees what critical thinking is, how students can be taught to think critically, or even whether critical thinking is something that can be explicitly taught at all (as opposed to being tacit knowledge). So, it can be instructive to see how other people, particularly top thinkers in your own discipline, conceptualise critical thinking, and how they propose training students to think critically.

So, I was glad I finally round time this week to read this 2022 article by John List (University of Chicago), published in the Journal of Economic Education (ungated earlier version here). He develops a 'Critical Thinking Hierarchy', and then describes how this can be applied to student learning.

List defines critical thinking as "individual skills that facilitate logical and informed decisions", and notes that:

In my own experiences, these skills can naturally be divided into two complementary pillars:

• Connecting the dots with empiricism: developing and assimilating empirical evidence and updating of one’s beliefs

• Connecting the dots with abstract thought: putting the puzzle together with conceptual reasoning; thought experiments

The hierarchy that List develops is then described in Figure 1 from the paper:

The hierarachy starts with modal thinking, characterised by various biases, preconceptions, and prejudices. It then moves to neophyte thinking, where the importance of thinking and empiricism begin to be recognised, but the execution isn't yet there. Adept thinking progresses to critical questioning, recognising blind spots, and understanding causation. Finally, great thinkers understand and correct for their own biases, and constantly re-examine their assumptions. An obvious question, though, is how a teacher moves their students up through the hierarchy. List advocates that teachers should encourage their students to 'slow think' (drawing inspiration from Daniel Kahneman's research, as outlined in his book Thinking, Fast and Slow). Specifically, List recommends that we teach students to apply six basic tenets:

1. state, explain, and clarify the question(s)

2. think through the question(s) from multiple points of view, expressing their own priors using logical thinking

3. gather, organize, assimilate information and data

4. identify assumptions, shortcomings, and implications of the data generation process

5. update priors, both their own priors and consider how other’s views might change

6. explain and apply what they learn, connecting what they just learned to other economic concepts, learnings from another course, and/or their everyday life

He then offers a set of necessary conditions through which those six basic tenets can be embedded to develop critical thinking (CT):

A. Early discussion of simple empirical tools; distinction between correlation and causation; and, provide examples throughout the course that distinguish causation and correlation

B. Theory of mind should be reinforced (I cannot think of a better place than game theory); briefly introduce psychological biases that prevent the student from becoming Adept

C. Because CT development is a social activity, both lab and field experiments should be used as pedagogical devices to promote CT...

D. Connect theory to empiricism and highlight potential shortcomings of both without undoing the major insights

Finally, the punchline is that this approach is part of the textbook that List co-authored with Daron Acemoglu and David Laibson. So, was the whole article an elaborate advertisement for the textbook? Perhaps, but I think there is a greater value in what List has shared. Right now, my colleagues and I are re-designing the economics curriculum at the University of Waikato from the ground up. We haven't explicitly embedded critical thinking in that design and yet, when I look at the four necessary conditions that List has proposed, and the two critical thinking skills that the article starts with, I can definitely see those in what we are developing. In particular, linking economic theory to empirical research and insights is already a particular strength of economics at Waikato.

List wrote his article in 2021, and it was published in 2022, before the release of ChatGPT and the widespread adoption of large language models. It would be reasonable to ask whether generative AI would make a difference to what List proposes. I don't believe that it change the core of List's framework. In fact, it makes the framework more important and changes how we should implement it, because the thoughtful integration of generative AI into higher education may lead to greater opportunities for students to engage in intentional development of the critical thinking skills that List proposes. Think about the application of simple empirical skills. These are skills that students can learn to use alongside generative AI, if their interactions are intentionally designed to promote engagement and learning, rather than cognitive offloading. For example, students might use generative AI to collate data and test an empirical relationship, while thinking about how they might distinguish causation from correlation. A custom GPT would allow for limitation in the range of techniques that could be employed so that novice students don't get overwhelmed. Lab and field experiments can be integrated into classroom learning, and I know that both Steve Tucker and I do that in our classes already. Experiments can be incredibly powerful tools for promoting student learning.

Critical thinking may be an education buzzword, but that doesn't mean that it isn't important. I like List's metaphor of 'connecting the dots'. As generative AI continues to develop and becomes a more useful tool across a wider range of analytical and other activities, connecting the dots alone will not be enough.  Students will need to know which dots matter, whether they genuinely connect, and what conclusions the connected dots allow them to draw.

The impact of ransomware attacks on hospitals and patients

In May 2021, the Waikato District Health Board (DHB) was hit with a ransomware attack. It took some four weeks for clinical services to be restored, and in the meantime, surgeries were postponed, and patients and health staff were negatively impacted. Health services have been a common target of these ransomware attacks, and the consequences could be tragic. Fortunately, in the case of the Waikato DHB attack, there is no evidence of patients dying as a result.

That isn't always the case though. This recent article by Hannah Neprash, Claire McGlave, and Sayeh Nikpay (all University of Minnesota), published in the American Economic Journal: Economic Policy (ungated earlier version here) looks at the impact of ransomware attacks on hospitals in the US. They use Medicare administrative claims data, along with data from HackNotice and the Office for Civil Rights Breach Portal on ransomware attacks on hospitals. They find 74 attacks over the period from 2016 to 2021, affecting 160 hospitals.

Neprash et al. then look at the effect of the ransomware attack on hospital volume (separating emergency room, inpatient, and outpatient volume), and hospital revenue from Medicare, as well as patient mortality. They use a difference-in-differences analysis, which involves looking at the difference in each outcome variable between the time before and the time after the ransomware attack, for hospitals that were attacked, and those that were not. They identify control group hospitals (that weren't attacked) as those most similar to the affected hospitals in terms of non-profit status, health system membership, and quartile of Medicare admissions in the year prior to the attack. They also conduct an event study, which allows them to look at how the impact changes over time.

In their main analysis, Neprash et al. find that:

During the initial week of a ransomware attack, hospital volume falls by 17–24 percent in the ER, inpatient, and outpatient settings. Medicare revenue declines by 19–39 percent at ransomware-attacked hospitals. A full recovery to pre-attack volume and revenue occurs within two to three weeks on average. A back-of-the-envelope calculation suggests that the average ransomware attack reduces annual hospital revenue by roughly 1 percent.

These are quite substantial effects (and notice the recovery time is not dissimilar to the case of the Waikato DHB attack). What happens to the patients who are affected by their hospital being attacked? It turns out that many patients were redirected to nearby hospitals, although the extent of redirection differs by type of care, as when Neprash et al. look at the local hospital market rather than the individual hospital, they find that:

...nearby hospitals absorb displaced emergency department patient volume from attacked facilities, such that market-level emergency department volume does not change during attacks. Inpatient and outpatient hospital volume is partially absorbed by nearby hospitals, though not fully, resulting in a market-level volume decrease during the first week of a ransomware attack.

That also means the costs of a ransomware attack spill over to neighbouring hospitals, which have to absorb some of the displaced patients. What about patient outcomes? Here are the most serious impacts, as Neprash et al. find that:

...ransomware attacks increase in-hospital mortality for patients already admitted to ransomware-attacked hospitals when the attack begins, compared to patients whose admissions concluded in the five weeks prior. Our estimates suggest that ransomware attacks resulted in the deaths of between 69 and 76 Medicare patients—representing roughly 1 Medicare death per month due to ransomware over the course of our study period.

Notice the effects fall on patients who were already in hospital care at the time the attack started. Many of those patients may not be easily redirected to other hospitals, and so have little choice but to ‘ride out’ the attack in the affected hospital. Neprash et al. also find larger mortality impacts at smaller or independent hospitals, during particularly severe attacks, and among patients with complex care needs (such as patients in intensive care, or those with multiple chronic conditions). Neprash et al. don't offer much of a policy prescription in their discussion of their results, limiting themselves to recommending:

...a combination of policies designed to reduce the likelihood of any successful ransomware attacks (e.g., minimum cybersecurity standards for hospitals) and policies designed to reduce the severity of ransomware attacks when they do happen (e.g., incident planning requirements).

For me, the key takeaway from this research is that ransomware attacks are not just a cybersecurity issue, they are a health security issue. We can be thankful that the Waikato DHB attack avoided the much worse outcomes that US hospitals have experienced. The response nevertheless required serious efforts by IT professionals and imposed a heavy workload on health and administrative staff. We should treat these results as a warning that hospitals need to treat resilience to cyberattacks as part of their core patient-safety planning, rather than simply as an IT problem.

Discrimination on #EconTwitter

I've written a number of times about correspondence experiments designed to identify discrimination in labour markets. In such an experiment, the researcher applies for a bunch of jobs, using fake job 'applicants' that differ only on the basis of some known characteristics (gender, for example). The difference in callback rates (or some other similar measure) between applicants with different characteristics provides a measure of discrimination on the basis of those characteristics.

A new application of this approach instead looks at discrimination on #EconTwitter (on X). The research is reported in this 2025 article by Nicolás Ajzenman (McGill University), Bruno Ferman (Sao Paulo School of Economics), and Pedro Sant’Anna (MIT), published in the journal American Economic Review: Insights (ungated earlier version here). In their experiment, Ajzenman et al. created 80 'bot' accounts on X between May and August 2022 that mimicked real PhD student accounts, but differed in terms of gender (male or female), race (Black or White), and university affiliation (top-ranked [top ten in the 2017 US News ranking of economics graduate programmes] or lower-ranked [ranked 79-100] university). Each bot account was active for twelve days, during which time it initially randomly retweeted posts from economics journals to establish its credibility, then followed a random selection of 100 members of the #EconTwitter community (from a pool of over 10,000 X users who had tweeted or retweeted content using the #EconTwitter hashtag between January and February of 2022).

Ajzenman et al. then measure the number of times each account is followed back by the user it followed, and look at the difference in follow-back rates for bot accounts with different characteristics (gender, race, and university affiliation). The raw follow-back rates are shown in Figure 1 in the paper:

Bot accounts that presented as Black males from lower-ranked universities received the lowest follow-back rate of 14.4 percent, while bots presenting as White females from top-ranked universities were followed back 23.9 percent of the time. In their regression models, Ajzenman et al. find that:

Users in the #EconTwitter community are 2.1 percentage points (12 percent) more likely to follow White than Black PhD students, 3.5 percentage points (21 percent) more likely to follow students from top-ranked universities than those from lower-ranked ones, and 4.3 percentage points (25 percent) more likely to follow female than male students. We also find that the racial gap in follow-backs remains among students claiming to be from top-ranked universities. This suggests that racial discrimination persists even in the presence of a signal indicative of higher academic potential.

Turning to how the results vary based on the characteristics of the X users that were followed, Ajzenman et al. find no differences by gender or race, or by the reach of the X user (measured by the number of followers they have). They also find no difference between X users who have demonstrated concern about the lack of diversity in economics (by following one or more X accounts devoted to the topic) and those that haven't. However, they also find that:

...subjects who signal concern about the lack of diversity in economics discriminate the most in terms of university affiliation. While both groups of subjects favor students affiliated with top-ranked institutions, the difference in follow-back rates between top- and lower-ranked students is considerably larger among concerned subjects (8.6 percentage points, against 2.5 percentage points for the remaining subjects, p-value of difference < 0.05)...

So, concern about demographic diversity clearly doesn't imply an absence of other forms of discrimination, as these results seem to show that those members of the #EconTwitter community who are most concerned about diversity discriminate more against students from lower-ranked institutions than users who are less concerned.

Given the longstanding gender bias in economics, the higher follow-back rates for bot accounts presenting as female are perhaps the most surprising result. Ajzenman et al. suggest several possible explanations. First, some users may be conscious of the barriers women face in the profession and therefore make a deliberate effort to engage with them. Second, some users may be using X partly to establish social rather than professional relationships, which would suggest a more negative interpretation of the higher follow-back rate. Ajzenman et al. also suggest two more strategic explanations. Researchers may see advantages in collaborating with women because female economists tend to receive less credit for joint work. Alternatively, a woman gaining admission to a highly ranked PhD programme may be providing a stronger signal of ability, given the additional barriers women face in reaching that point. The experiment cannot distinguish among these explanations, so teasing out which mechanisms are more important would require further, more detailed, research.

And no doubt Ajzenman et al. had hoped to conduct some more detailed research than what they reported, but their data collection had to be cut short. As they explain:

We planned to run 30 experimental waves between May and December 2022, which would have given us more than enough power to identify reasonable effects. This was to account for potential problems, such as X blocking some accounts. We stopped earlier because an X user saw some of the accounts and posted about the experiment during the eleventh wave, which compromised the continuity of the experiment.

Sometimes research just doesn't go as planned, and that might explain why this research got published in the more modest AER: Insights journal, rather than the top-five journal the authors probably were initially hoping for.

Sometimes research just doesn't go as planned. However, Ajzenman et al. had already collected enough data to reveal a substantial amount of discrimination on #EconTwitter. Perhaps the most important result is that signalling concern about diversity doesn't necessarily make people immune to other forms of bias, especially in terms of academic prestige.

[HT: Marginal Revolution]

Computer gaming and binge drinking may be complements, not substitutes

In economics, two goods are substitutes if consumers tend to consume more of one if the price of the other increases. One way of thinking about that is that if the price of Good X increases, consumers switch to purchasing Good Y instead, and the quantity of Good Y demanded increases. Two goods are complements if consumers tend to consume less of one if the price of the other increases. In this case, if the price of Good X increases, consumers buy less of Good X (due to the Law of Demand), but also buy less of Good Y, and the quantity of Good Y demanded decreases.

Whether a pair of goods are substitutes or complements is determined by the cross-price elasticity of demand: the responsiveness of the quantity demanded of one good to a change in the price of the other good. If the cross-price elasticity is positive, the two goods are substitutes. If the cross-price elasticity is negative, the two goods are complements. Another way of thinking about this is that, following a change in the price of one good, ceteris paribus (holding all else constant), we would expect the quantities demanded of substitutes to move in opposite directions, while the quantities demanded of complements would move in the same direction.

There are obvious examples of substitutes and complements. Coke and Pepsi are the iconic example of substitute goods used in almost every introductory economics class. An example of complements that I use in my classes is video game consoles and games. However, it isn't always straightforward to determine whether a pair of goods are substitutes or complements. Sometimes they may be substitutes in one context, but complements in another. So, whether goods are substitutes or complements is an empirical question.

Take the example of computer gaming and binge drinking. When I was growing up, those two 'goods' certainly seemed like complements. My friends and I spent many nights drinking beer or RTDs and playing hotseat turn-based strategy games like Robosport, Warlords II, or Heroes of Might and Magic.[*] That experience made me a little surprised to see the hypothesis in this 2021 article by Torleif Halkjelsvik, Geir Brunborg, and Elin Bye (all Norwegian Institute of Public Health), published in the journal Drug and Alcohol Review (open access), which was that binge drinking and computer gaming are substitutes. Now, modern computer gaming differs in meaningful ways from how it looked when I was young. Nevertheless, I was surprised that Halkjelsvik et al. hypothesised in the direction they did.

Their hypothesis rested on several ideas, and was motivated by the observed increase in gaming and decrease in alcohol consumption by young people over time. First, alcohol and gaming are both outlets for thrill seeking, and are both responses to boredom, so increasing computer gaming might reduce the need for drinking. Second, both drinking and computer gaming are sources of social bonding, so again more computer gaming reduces the need for drinking.

Halkjelsvik et al. test their hypothesis with data from the European School Survey Project on Alcohol and Other Drugs (ESPAD), which surveys 15 and 16-year-old students every four years. They use data from 23 countries over the period from 1995 to 2015 (although noting that not all countries are part of the survey in every year), and look at the correlation between frequency of binge drinking (drinking five or more drinks on an occasion) and frequency of computer gaming, using a multi-level linear probability model. If their hypothesis that gaming displaces drinking is correct, the relationship should be negative. However, Halkjelsvik et al. find that:

...the association between country-level changes in computer gaming and binge drinking was estimated as positive...

So, increases in the average frequency of computer gaming at the country level tended to be associated with increases in the frequency of binge drinking. And, at the individual level:

The between individual-effect was positive, suggesting a four percentage point (±2 percentage points) higher binge drinking prevalence among students who report playing computer games daily.

Of course, the analysis that Halkjelsvik et al. conducted doesn't establish a causal relationship, it only shows correlations. And, importantly, they aren't directly testing whether computer gaming and binge drinking are complements in the economic sense, as that would require looking at how consumption of one responds to changes in the price of the other. However, their results are at least consistent with computer gaming and binge drinking being complements. Rather than moving in opposite directions, as we might expect if gaming displaced drinking (as Halkjelsvik et al. hypothesised), gaming and binge drinking tend to move in the same direction. Which, admittedly on the basis of a rather smaller and less representative sample, my friends and I could have told them.

*****

[*] My kids are bemused at the very idea that there was ever such a thing as hotseat multiplayer games. Sadly, they gradually died out as online games became more widely available in the early 2000s. However, they were really good for multi-tasking with some tabletop gaming at the same time, since only one player played the hotseat game at a time.