When I introduce the difference between causation and correlation in my ECONS101 class, I talk about how, even when there is a good story to tell about why a change in one variable causes a change in the other, that doesn't necessarily mean that an observed relationship is causal. It appears that I am just a susceptible to a good story as anyone else. When a research paper has a good story, and the data and methods seem credible, I'm willing to update my priors by a lot (unless the results also contradict a lot of the prior research). I guess that's a form of confirmation bias.
So, I was willing to accept at face value the results of the article on tea drinking and mortality in England that I blogged about earlier this week. To recap, that research found that the increase in tea drinking in 18th Century England, by promoting the boiling of water, reduced mortality. However, now I'm not so sure. What has caused me to re-evaluate my position is this other paper by Francisca Antman and James Flynn (both University of Colorado, Boulder), on the effect of beer drinking on mortality in pre-industrial England.
Antman is the author of the tea-drinking article, so it should be no surprise to expect that the methods and data sources are similar, given the similarity of the two papers in terms of research question and setting. However, there are some key differences between the two papers (which I will come to in a minute). First, why study beer? Antman and Flynn explain that:
Although beer in the present day is regarded primarily as a beverage that would be worse for health than water, several features of both beer and water available during this historical period suggest the opposite was likely to be true. First, brewing beer would have required boiling the water, which would kill many of the dangerous pathogens that could be found in contaminated drinking water. As Bamforth (2004) puts it, ‘the boiling and the hopping were inadvertently water purification techniques’ which made beer safer than water in 17th century Great Britain. Second, the fermentation process which resulted in alcohol may have added antiseptic qualities to the beverage as well...
Notice that the first mechanism here is basically the same as for tea. Boiling water makes water safer to drink, even when it is being used in brewing. Also:
...beer in this period, which sometimes referred to as ”small beer,” was generally much weaker than it is today, and thus would have been closer to purified water. Accum (1820) found that small beer in late 18th and early 19th century England averaged just 0.75% alcohol by volume, a tiny fraction of the content of even the ‘light’ beers of today.
The data sources are very similar to those used for the tea drinking paper, and the methods are substantially similar as well. Antman and Flynn compare parish-level summer deaths (which are more likely to be associated with water-borne disease than summer deaths) between areas with high water quality and low water quality, before and after a substantial increase in the malt tax in 1780. Using this difference-in-differences approach, they find that:
...the summer death rate in low water quality parishes increases by 22.2% relative to high water quality parishes, with a p-value on the equality of the two coefficients of .001.
Antman and Flynn then use a second identification strategy, which is to compare summer deaths between parishes that have gley soil (suitable for growing barley, which is then malted and used to make beer) and parishes without gley soil, before and after the change in the malt tax. In this analysis, they find that:
...parishes with gley soil had summer death rates which increased by approximately 18% after the malt tax was implemented relative to parishes without gley soil.
Not satisfied with only two identification strategies, Antman and Flynn then use a third, which is rainfall. Their data is limited to the counties around London (because that is where they have the rainfall data from). In this analysis, they find that:
...the effect of rainier barley growing seasons on parishes with few nearby water sources is positive and significant, indicating that summer deaths rise following particularly rainy barley growing seasons... [and] ...rainy barley-growing seasons lead to more summer deaths in areas where beer is most abundant, even controlling for the number of deaths occurring in the winter months.
So, the evidence seems consistent with beer drinking being associated with lower mortality, because in areas where beer drinking decreased (because of the increase in the malt tax) by a greater amount, mortality increased by more.
But not so fast. There are two problems here, when you compare across the tea drinking and beer drinking research. First, the data that they use is not consistent. The tea drinking paper uses all deaths in each parish. The beer drinking paper uses only summer deaths, arguing that summer deaths are more likely to be from water-borne causes. If that is the case, why use all deaths in the tea drinking paper? What happens to the results from each paper when you use the same mortality data specification?
Second, the increase in the Malt Tax was in 1780. The decrease in the tea tax (which the tea drinking paper relies on) was in 1784. The two tax changes are awfully close together timewise, and disentangling their effects would be difficult. However, neither paper seems to account for the other properly. The beer drinking paper includes tea imports as a control variable, but in the tea drinking paper it wasn't tea imports, but tea imports interacted with water quality that was the key explanatory variable (and the timing of the tea tax change interacted with the water quality variable). The tea drinking paper doesn't really control for changes in beer drinking at all.
That second problem is the bigger issue, and creates a potentially problematic omitted variable problem in both papers. If you don't include changes in tea drinking in the beer drinking paper, and the two tax changes happened around the same time, how can you be sure that the change in mortality was due to tea drinking, and not beer drinking? And vice versa for failing to include changes in beer drinking in the tea drinking paper.
However, maybe things are not all bad here. Remember that the two effects are going in opposite directions. It is possible that the decrease in the tea tax increased tea drinking, and mortality reduced, while the increase in the malt tax decreased beer drinking, and mortality increased. However, then we come back to the first problem. Why use a measure of overall mortality in the tea drinking paper, and a measure of only summer mortality in the beer drinking paper, when both papers are supposed to be looking at changes in mortality stemming from water-borne diseases?
Hopefully now you can see why I have my doubts about the tea drinking paper, as well as the beer drinking paper. Both are telling an interesting story, but the inconsistencies in data and approach across the two papers should make use extra cautious about the results, and leave us pondering the question of whether the results are causal or simply correlation.
[HT for the beer paper: The Dangerous Economist]
Read more:
No comments:
Post a Comment