We don't know the answer to the trade-off question in relation to the coronavirus (as I noted in this post), but perhaps looking at past pandemics can help us to infer what the trade-off is likely to be like. As one example, this new (and revised) working paper by Sergio Correia (Federal Reserve Board), Stephan Luck (Federal Reserve Bank of New York), and Emil Verner (MIT) looks at the impact of non-pharmaceutical interventions (NPIs) during the 1918 flu pandemic in the U.S. NPIs include "school, theater, and church closures, public gathering bans, quarantine of suspected cases, and restricted business hours".
Specifically, Correia et al. categorised the speed and intensity of NPIs at the city level for 43 U.S. cities, and look at how differences in the speed and intensity impacted on a measure of business disruptions. The results are a little surprising:
...when we compare cities with strict and lenient NPIs, we find that the increase in business disruptions in the fall and winter of 1918 was quantitatively similar across the two sets of cities. Our findings thus indicate that NPIs did not clearly exacerbate the economic downturn during the pandemic.
Further, we examine the economic impact of NPIs in the medium run. We find no evidence that cities that intervened earlier and more aggressively perform worse in the years after the pandemic, measured by local manufacturing employment and output and the size of the local banking sector.In other words, there was no difference in business disruption between cities that went 'hard and fast' and those that didn't. That suggests no trade-off between the economy and public health, or as Correia et al. put it:
...our results suggest that it is not a foregone conclusion that there is a trade-off between reducing disease transmission and stabilizing economic activity in a pandemic.This working paper has already been through at least one set of revisions, in response to some strong critiques such as this one by Andrew Lilley, Matthew Lilley, and Gianluca Rinaldi (all Harvard University). However, let's not get carried away and start believing there is no trade-off between public health and the economy in our current crisis. I still see some fairly major problems with this working paper.
Essentially, Correia et al. want us to believe that they have demonstrated no impact of NPIs on business disruptions. Proving a negative is of course impossible statistically. Proving a positive (e.g. that NPIs do impact business disruptions) is also impossible, but our normal statistical methods are at least geared towards demonstrating the strength (or weakness) of evidence in favour of an impact. As a result, we are generally much more likely to demonstrate that there is no impact when there really is, rather than the other way around.
One of the reasons we are more likely to show no impact where there really is impact is lack of statistical power. To have confidence in the results, and therefore improve the precision of the estimated effects, you need to have lots of data. Smaller datasets are less likely to show statistical significance, not because there is no effect, but because the measured effect is noisy. So, if you want to argue the absence of an effect ('no impact'), you need to argue that your statistical methods have enough statistical power to find an effect if there was one. Correia et al. don't really do this, although I don't have major concerns about the size of their dataset.
The strength (or weakness) of statistical evidence also depends on the quality of the data. Measurement error is typical problem, and when an explanatory variable is measured with error then the statistical estimates are more likely to show up as statistically insignificant. Correia et al.'s measure of NPI speed and intensity might not suffer from much in the way of measurement error - however, their measure does conflate the speed (how soon NPIs were put in place after each city exceeded twice its baseline mortality rate) and intensity (the cumulative sum of the number of days where school closure, public gathering bans, and quarantine/isolation of suspected cases were in place).
More damaging for the paper is the measure of business disruptions, which is based on a coding of city-level qualitative reports of business activity taken from the weekly Bradstreet’s - A Journal of Trade, Finance, and Public Economy. While Correia et al. are clear in how they code this variable in their analysis, it is not at all clear how Bradstreet's constructed this evaluation in the first place. Moreover, some of the Correia et al. coding is likely to induce measurement error, particular how they handle relative descriptors in the data (e.g. "improving" or "slowing down"), and even some of the absolute descriptors (e.g. "below normal" is coded as Bad, while "60 percent" is coded as Fair). The robustness of the analysis to these choices needs to be tested.
Finally, there was one interesting bright spot in the paper, but it surprised me that Correia et al. didn't follow through on it. Early on, they note that:
...cities that experienced outbreaks at later dates tended to implement NPIs sooner within their outbreak, as they learned from the experiences of cities affected earlier... Thus, as the flu moved from east to west, cities located further west were faster in implementing NPIs.It seems to me that the longitude of each city could be used to instrument for at least the speed measure of NPIs. Using instrumental variables reduces problems of measurement error (at least for the NPI variable - the business disruptions variable still needs some thorough checks on the robustness of results). Perhaps this is an opportunity for some follow-up work, either for these authors or for an enterprising student.
[HT: Marginal Revolution]
No comments:
Post a Comment