Lockdowns, alcohol, and why all natural experiments are not created equal

A natural experiment occurs when there is some unexpected policy or other shock that affects some, but usually not all, of the units of observation (often people, but could be firms, states, or countries). Because by definition the shock is not anticipated (that's why it's called a 'shock'!), it is often as good as randomly assigned, and can be used to extract the causal impact of one variable on another. So, for example, if a school classroom is destroyed in a fire, and the students are distributed to other classes, that might be able tell us something about the effect of class sizes on student outcomes.

However, perhaps more than other experiments, a natural experiment relies crucially on how you define the counterfactual - what would have happened in the absence of the shock. Sometimes, you can assume that there are other similar people (or firms, states, or countries) that were not affected by the shock. In the case of the torched classroom, students in another nearby (and sufficiently similar) school might provide a good counterfactual for how students would have performed if their classroom hadn't burned down. However, that doesn't work so well if everyone was affected by the shock. In that case, you might assume that whatever was happening before the shock would continue afterwards. That's sometimes a heroic assumption that simply fails on the face of it.

The coronavirus pandemic is a good example of this. While it might be attractive to think of the pandemic (or lockdowns) as a natural experiment, it is very difficult to tell what would have happened if there hadn't been a pandemic. Comparing outcomes before and after the pandemic might be ok, but comparing the period before the pandemic with the period during the pandemic is fraught, because the fundamental relationships may be different.

However, that hasn't stopped Christopher Snowdon, in this discussion paper on the effects of lockdowns on alcohol-related mortality in the UK. Snowdon uses the lockdown to take aim at the 'single distribution theory' and its implications, which he outlines as:

Starting in the 1960s, a view began to take hold in the alcohol research community that the amount of heavy drinking and alcohol-related harm... in a society were directly linked to per capita alcohol consumption... It was argued that this was not because heavy drinkers consumed a large quantity of alcohol (thereby raising the mean), but because the entire distribution of consumption was determined by the mean...

This is known as the single distribution theory and it was given support by the influential theories of the British epidemiologist Geoffrey Rose in the early 1990s...

Adhering to the belief that reducing per capita alcohol consumption is necessary and sufficient to reduce alcohol-related harm, proponents of the whole population approach endorse population-wide, supply-side interventions in the market - targeting ‘the Three A’s’: affordability, availability and advertising. Policies include alcohol duty rises, minimum pricing, tougher licensing laws and advertising bans.

Single distribution theory is mathematically dubious. We don't need research to tell us this. A simple example will suffice. If greater availability of alcohol increases alcohol consumption among heavy drinkers, regardless of any effect on the mean alcohol consumption of the population, you would expect greater harm (among those heavy drinkers). So, Snowdon has created a bit of a straw man argument in order to support a call for reductions in alcohol control (ok, maybe some public health researchers use single distribution theory to support restrictions on alcohol availability, but I had never heard of it before reading this discussion paper, so it can't be that widely employed).

Anyway, back to the evidence that Snowdon provides. He first assets that alcohol availability was curtailed during the pandemic:

The number of places in which alcohol could be bought collapsed on 20 March when pubs, clubs, bars and restaurants were closed by law... Hotels were closed for leisure and tourism, although some remained open for limited business purposes. Nightclubs were closed from 20 March 2020 until 19 July 2021 in England and until January 2022 in Scotland and Wales. At a stroke, most premises with a commercial alcohol licence were shut...

The number of places selling alcohol therefore fell by approximately two-thirds. Its availability, as defined in modern public health, was greatly reduced.

That seems sensible at first glance (although see my comments below). Snowdon then presents some evidence on the how alcohol consumption changed, finishing with

In summary, per capita alcohol consumption fell in the UK but most people did not change the amount they drank and those who did went in opposite directions, with heavy drinkers tending to drink more. There was no single distribution.

Finally, Snowdon looks at the effect on alcohol-specific mortality (noting that "‘alcohol-specific’ deaths are wholly due to alcohol use, as distinct from the broader category of ‘alcohol-related’ deaths which include diseases in which alcohol is one risk factor, such as cancer"). He shows that:

...there is clear evidence of a dramatic increase in alcohol-specific mortality in the UK with the death rate rising by 18.7 per cent in 2020...

So, to summarise, according to Snowdon: the availability of alcohol declined dramatically; alcohol consumption didn't change on average (but there was a change in distribution, with more drinking among heavy drinkers, and less drinking among occasional drinkers); and alcohol-specific mortality increased. Snowdon then concludes that:

A more reasonable conclusion to draw, which was amply illustrated by the extraordinary natural experiment of 2020, is that harmful drinking is not driven primarily, if at all, by ‘commercial determinants’ but by personal circumstances, hardship and stress. From this we might conclude that tackling harmful drinking requires focusing on harmful drinkers rather than on the whole population.

I disagree. This natural experiment offers minimal guidance on how alcohol availability affects consumption or harm. First, in spite of the facts that bars and hospitality venues were forced to close, it's not clear that alcohol availability even reduced. People could still buy alcohol, as Snowdon himself notes:

During the lockdowns, off-licences - including home delivery companies and most supermarkets - were almost the only places from which alcohol could be bought.

So, people could still access alcohol. They probably bought more alcohol using home delivery, and less from bars. Snowdon doesn't provide any evidence on this. Alcohol consumption on average didn't change, so clearly people were getting it from somewhere. The relationship between alcohol outlets and alcohol availability fundamentally changed during the lockdowns. Comparing the time before lockdown with the time during lockdown isn't going to tell us anything meaningful about how availability and consumption are related. The counterfactual is not appropriate.

All we can really conclude from this paper is that heavy drinkers drank more, and alcohol-specific mortality increased. However, even that (seemingly obvious) relationship is tainted by the lack of a valid counterfactual. Medical services were overwhelmed during the pandemic, and so mortality might be higher from all causes included alcohol-specific mortality, but not because of increased drinking among heavy drinkers.

In both cases, it would be better to compare the time before lockdowns with the time after lockdowns, ignoring any data during the lockdown period. It is likely that some alcohol outlets closed down during lockdown and did not reopen. The lockdown therefore provides an exogenous change in the number of outlets, and that comparison may be a valid natural experiment, with a sensible counterfactual. Perhaps we'll see some research along those lines in the future. Instead, what Snowdon is done is largely unhelpful.

This is the second paper of Snowdon's that I've taken issue with this year (see here), although admittedly the earlier one was written in 2018. However, I had been planning to read his 2010 book The Spirit Level Delusion, which offers a critique of Wilkinson and Pickett's famous book The Spirit Level (which I discussed here, with some additional critique of my own here). If the standard of evidence in Snowdon's book is the same as in these two papers, I don't think I will bother.

[HT: Eric Crampton at Offsetting Behaviour]