Lockdowns, alcohol, and why all natural experiments are not created equal

A natural experiment occurs when there is some unexpected policy or other shock that affects some, but usually not all, of the units of observation (often people, but could be firms, states, or countries). Because by definition the shock is not anticipated (that's why it's called a 'shock'!), it is often as good as randomly assigned, and can be used to extract the causal impact of one variable on another. So, for example, if a school classroom is destroyed in a fire, and the students are distributed to other classes, that might be able tell us something about the effect of class sizes on student outcomes.

However, perhaps more than other experiments, a natural experiment relies crucially on how you define the counterfactual - what would have happened in the absence of the shock. Sometimes, you can assume that there are other similar people (or firms, states, or countries) that were not affected by the shock. In the case of the torched classroom, students in another nearby (and sufficiently similar) school might provide a good counterfactual for how students would have performed if their classroom hadn't burned down. However, that doesn't work so well if everyone was affected by the shock. In that case, you might assume that whatever was happening before the shock would continue afterwards. That's sometimes a heroic assumption that simply fails on the face of it.

The coronavirus pandemic is a good example of this. While it might be attractive to think of the pandemic (or lockdowns) as a natural experiment, it is very difficult to tell what would have happened if there hadn't been a pandemic. Comparing outcomes before and after the pandemic might be ok, but comparing the period before the pandemic with the period during the pandemic is fraught, because the fundamental relationships may be different.

However, that hasn't stopped Christopher Snowdon, in this discussion paper on the effects of lockdowns on alcohol-related mortality in the UK. Snowdon uses the lockdown to take aim at the 'single distribution theory' and its implications, which he outlines as:

Starting in the 1960s, a view began to take hold in the alcohol research community that the amount of heavy drinking and alcohol-related harm... in a society were directly linked to per capita alcohol consumption... It was argued that this was not because heavy drinkers consumed a large quantity of alcohol (thereby raising the mean), but because the entire distribution of consumption was determined by the mean...

This is known as the single distribution theory and it was given support by the influential theories of the British epidemiologist Geoffrey Rose in the early 1990s...

Adhering to the belief that reducing per capita alcohol consumption is necessary and sufficient to reduce alcohol-related harm, proponents of the whole population approach endorse population-wide, supply-side interventions in the market - targeting ‘the Three A’s’: affordability, availability and advertising. Policies include alcohol duty rises, minimum pricing, tougher licensing laws and advertising bans.

Single distribution theory is mathematically dubious. We don't need research to tell us this. A simple example will suffice. If greater availability of alcohol increases alcohol consumption among heavy drinkers, regardless of any effect on the mean alcohol consumption of the population, you would expect greater harm (among those heavy drinkers). So, Snowdon has created a bit of a straw man argument in order to support a call for reductions in alcohol control (ok, maybe some public health researchers use single distribution theory to support restrictions on alcohol availability, but I had never heard of it before reading this discussion paper, so it can't be that widely employed).

Anyway, back to the evidence that Snowdon provides. He first assets that alcohol availability was curtailed during the pandemic:

The number of places in which alcohol could be bought collapsed on 20 March when pubs, clubs, bars and restaurants were closed by law... Hotels were closed for leisure and tourism, although some remained open for limited business purposes. Nightclubs were closed from 20 March 2020 until 19 July 2021 in England and until January 2022 in Scotland and Wales. At a stroke, most premises with a commercial alcohol licence were shut...

The number of places selling alcohol therefore fell by approximately two-thirds. Its availability, as defined in modern public health, was greatly reduced.

That seems sensible at first glance (although see my comments below). Snowdon then presents some evidence on the how alcohol consumption changed, finishing with

In summary, per capita alcohol consumption fell in the UK but most people did not change the amount they drank and those who did went in opposite directions, with heavy drinkers tending to drink more. There was no single distribution.

Finally, Snowdon looks at the effect on alcohol-specific mortality (noting that "‘alcohol-specific’ deaths are wholly due to alcohol use, as distinct from the broader category of ‘alcohol-related’ deaths which include diseases in which alcohol is one risk factor, such as cancer"). He shows that:

...there is clear evidence of a dramatic increase in alcohol-specific mortality in the UK with the death rate rising by 18.7 per cent in 2020...

So, to summarise, according to Snowdon: the availability of alcohol declined dramatically; alcohol consumption didn't change on average (but there was a change in distribution, with more drinking among heavy drinkers, and less drinking among occasional drinkers); and alcohol-specific mortality increased. Snowdon then concludes that:

A more reasonable conclusion to draw, which was amply illustrated by the extraordinary natural experiment of 2020, is that harmful drinking is not driven primarily, if at all, by ‘commercial determinants’ but by personal circumstances, hardship and stress. From this we might conclude that tackling harmful drinking requires focusing on harmful drinkers rather than on the whole population.

I disagree. This natural experiment offers minimal guidance on how alcohol availability affects consumption or harm. First, in spite of the facts that bars and hospitality venues were forced to close, it's not clear that alcohol availability even reduced. People could still buy alcohol, as Snowdon himself notes:

During the lockdowns, off-licences - including home delivery companies and most supermarkets - were almost the only places from which alcohol could be bought.

So, people could still access alcohol. They probably bought more alcohol using home delivery, and less from bars. Snowdon doesn't provide any evidence on this. Alcohol consumption on average didn't change, so clearly people were getting it from somewhere. The relationship between alcohol outlets and alcohol availability fundamentally changed during the lockdowns. Comparing the time before lockdown with the time during lockdown isn't going to tell us anything meaningful about how availability and consumption are related. The counterfactual is not appropriate.

All we can really conclude from this paper is that heavy drinkers drank more, and alcohol-specific mortality increased. However, even that (seemingly obvious) relationship is tainted by the lack of a valid counterfactual. Medical services were overwhelmed during the pandemic, and so mortality might be higher from all causes included alcohol-specific mortality, but not because of increased drinking among heavy drinkers.

In both cases, it would be better to compare the time before lockdowns with the time after lockdowns, ignoring any data during the lockdown period. It is likely that some alcohol outlets closed down during lockdown and did not reopen. The lockdown therefore provides an exogenous change in the number of outlets, and that comparison may be a valid natural experiment, with a sensible counterfactual. Perhaps we'll see some research along those lines in the future. Instead, what Snowdon is done is largely unhelpful.

This is the second paper of Snowdon's that I've taken issue with this year (see here), although admittedly the earlier one was written in 2018. However, I had been planning to read his 2010 book The Spirit Level Delusion, which offers a critique of Wilkinson and Pickett's famous book The Spirit Level (which I discussed here, with some additional critique of my own here). If the standard of evidence in Snowdon's book is the same as in these two papers, I don't think I will bother.

[HT: Eric Crampton at Offsetting Behaviour]

Summarising topics for students using movie trailers

Following on from yesterday's post about communicating research results to the general public, I read this 2018 article I found by Lara Goudsouzian (DeSales University), published in the Journal of Microbiology and Biology Education (open access). Goudsouzian discusses the use of movie trailers as a way to quickly introduce new topics for students (in biology, presumably). The paper doesn't include a lot of detail, but apparently the Apple iMovie app is able to easily create movie trailers based on photographs or video clips using one of several templates (and the process is described here). Goudsouzian created five trailers over a semester long course, and then her students:

...were asked to self-report their attitudes toward the movie trailer introductions using a five-point Likert scale...

The results are summarised in Figure 2 from the article:

It's pretty clear that students found the trailers to be interesting, unique and unexpected, and that they increased students' interest in the topics. However:

While some students did report using the movie trailers to study for exams, most disagreed with the assertion that the movie trailers could be used as a study tool for summative assessments...

That much is obvious from the bottom bar in the figure. Goudsouzian tries to argue that an increase in student interest would lead to higher student engagement, which in turn leads to greater learning, but I'd like to see some actual evidence of that before I believe it. However, Goudsouzian does offer this in the conclusion:

Rather than creating movie trailer introductions for students, instructors might choose to assign this work to students, as a way of familiarizing them with the topic before entering the class or as a means to review the material after it has already been covered.

That's very similar to this example I linked to in yesterday's post. And it makes me wish that I had read this article a couple of months ago, when planning how to approach flexible assessment in my ECONS101 and ECONS102 classes. We have adopted weekly video reflections for the trimester starting shortly (as one of two options for students, the other being in-class tests). However, an alternative might have been getting students to create trailers for each topic (having said that, the brief for a video reflection is open-ended enough that a trailer could satisfy the requirements, and would be pretty novel and interesting for the tutors to grade). We definitely need assessment items that make it more difficult for students to cheat (see here for more on that point). 

Reducing the opportunities for cheating is not the only reason to consider alternative types of assessment. Finding novel ways for teachers to keep students engaged is also important, and one way to keep students engaged is to have engaging assessments. In ECON100, we used to have a group video assignment, until it got to the stage where the novelty had clearly worn off and the students weren't taking it seriously (I linked to the last winning videos here). At that point, we ditched the video assignment and we haven't brought it back since. The takeaway is that every so often we need to refresh our assessment styles (if not our teaching styles). That not only keeps the students engaged, but gives us as teachers a chance to try something a little different. And even if only for that purpose, movie trailers may be a good option.

Cartoons as a means of communicating research results

The latest IRANZ Connections newsletter (from the Independent Research Association of New Zealand) pointed me to this post from MRINZ, which in turn pointed to this new article by Cilein Kearns (MRINZ) and co-authors, published in the journal Journal of Visual Communication in Medicine (sorry, I don't see an ungated version online, but there is a non-technical summary here). They compared study participants' preferences for receiving summaries of research results, across three different types: (1) a summary using a comic format; (2) a traditional non-technical lay summary; and (3) a published scientific abstract. Their research participants were 1236 research participants from a previous study, receiving a summary of earlier research (see the MRINZ post linked above), who received all three types of summaries in a random order, and rated each of them. The results are interesting:

The most preferred summaries were the comic N=716 (57.9%), lay summary N=321 (26.0%), and scientific abstract N=199 (16.1%). The least preferred summaries were the scientific abstract N=614 (49.7%), lay summary N=380 (30.7%) and comic N=242 (19.6%)... The trend of a majority most preferring the comic, and a majority least preferring the scientific abstract, was observed in all groupings except 18-24 year olds, where although the comic was still most preferred, the preferences for scientific abstract and lay summary were reversed.

When scored separately on understanding, satisfaction, and enjoyment, they get the following picture (although we are mostly left to guess at whether the differences are statistically significant):

What did the research participants like (and dislike) about comics? Kearns et al. report that:

The comic was reported to be easier to read and understand, more interesting, engaging, enjoyable, better at communicating data, to be relatable, provide more useful information through context, and to be more memorable. Many felt it was a faster format to read, despite being the longest in terms of both scrolling required, and text content. Many commented specifically on enjoyment of the art style and use of humour, and felt that the comic medium was non-threatening compared to the other media...

For the minority of respondents who least preferred the comic, reasons included finding the visuals distracting, the format too long or too short, that it had not enough or too much detail, could undermine the seriousness of the topic with the style or use of humour, and could be patronising or childish for some people.

This is some interesting research, because there is definitely a push from funders and research organisations for researchers to find better ways of communicating their results and conclusions (and methods) to the general public. The Conversation is one way of achieving this. Blogging, Twitter, and other social media are also available.

However, the format of the message is also important, and it is here than more novel ways of communicating may be important. Comics were used by Siouxsie Wiles and artist Toby Morris during the coronavirus pandemic to communicate important public health messages. Motu summarises each of its working papers in the form of a haiku. I've also seen some examples of using a movie-trailer-style video to summarise research (I can't find the original examples I recall, but here's a student assignment example, and research that finds using movie trailers to summarise course topics increases student interest - perhaps more on that in a future post).

The NZAE Conference last week had a poster session, with posters that were mostly uninspiring. Perhaps some cartoon posters are in order? Of course, the problem with these ideas is that time spent on communicating research through cartoons or other novel means takes a lot of time to do well, and the opportunity cost of that time spent is lost time on other research. Most researchers aren't cartoonists, graphic designers, professional type setters, or haiku poets (except for Motu, it seems). This is where research organisations (universities or other employers of researchers) could step up and make an important contribution. The Wiles and Morris collaboration could provide a model for collaboration in science communication that reduces the time cost to the researcher, and likely results in a higher quality output for the interested general public.

Who knows? Maybe in future I might summarise some of my research in comic form. Of course, in that case I would definitely like to collaborate with my preferred artist.

Book review: Good Economics for Hard Times

In late 2019, I reviewed Abhijit Banerjee and Esther Duflo's 2011 book Poor Economics. At the time, they were the latest Nobel Prize winners in economics (along with Michael Kremer). They also had a new book out at that time, Good Economics for Hard Times, which I passed over in favour of reading their earlier book, which was more closely based on their own research.

I've now read the more recent book, and like their earlier work, Good Economics for Hard Times is clearly written, well researched, and full of genuinely interesting insights. This is important because, as Banerjee and Duflo write in the introduction:

...it seems a large part of the general public has entirely stopped listening to economists about economics.

One can only hope that the general public would pay attention to this book, but sadly I fear that would not be the case. Banerjee and Duflo's solution, though, is as follows:

Our sense is that the best economics is frequently the least strident. The world is a sufficiently complicated and uncertain place that the most valuable thing economists have to share is often not their conclusion, but the path they took to reach it - the facts they knew, the way they interpreted those facts, the deductive steps they took, the remaining sources of their uncertainly.

That is the formula they follow in this book, including consideration of the costs (and not just the benefits) of international trade, the puzzle of why migration is so low, the sources of economic growth, the economics of climate change, and the rise of automation. These are the big issues of our times, along with inequality, about which they write:

Once again, this underscores the urgent need to set ideology aside and advocate for the things most economists agree on, based on the recent research. In a policy world that has mostly abandoned reason, if we do not intervene we risk becoming irrelevant, so let's be clear. Tax cuts for the wealthy do not produce economic growth.

That's the sort of clear statement (and advocacy for a particular normative position) that is missing from a lot of academic writing. However, while Banerjee and Duflo are excellent on laying out the research on particular issues, the book is noticeably light on solutions. I think this book would best be read in concert with a book that has solutions, but is much lighter on evidence, such as Mariana Mazzucato's Mission Economy (which I reviewed here). The combination of those two books, along with a bit of thoughtful filling in of the gaps, would provide an excellent outline of the problems, the evidence, and a prescription for the future.

We need some new inspiration to tackle the challenges that we face, and the solutions will involve some uncomfortable tradeoffs. We are vulnerable to politicians and policy advocates who try to sell us on ideas of change that come without associated costs. They should be resisted. As Banerjee and Duflo note in their conclusion:

Ideas are powerful. Ideas drive change. Good economics alone cannot save us. But without it, we are doomed to repeat the mistakes of yesterday. Ignorance, intuitions, ideology, and inertia combine to give us answers that look plausible, promise much, and predictably betray us.

I interpret that as saying that we don't need a new economics. We need to properly apply the tools of the economics we already have, and listen to the lessons it provides. And that in itself is a lesson worth listening to. Recommended!