It may be time to reconsider weather variables as instruments

For many years, I was sceptical of instrument variables analysis. I expressed a little of this scepticism in one of my early posts on this blog in 2014. However, by then I was starting to come around to the idea, and encouraging my PhD students to consider using it in their work. However, I may have been shifted a little more back towards scepticism by this new working paper by Jonathan Mellon (West Point).

Mellon focuses on the use of weather variables as instruments, and demonstrates the problems associated with using them. However, before he gets that far, he has a very clear exposition of what instrumental variables entails, which is worth sharing. First, here's Figure 1 from the paper:

Then, the associated explanation:

Endogeneity is one of the most pervasive challenges faced by social scientists. Naively, we might assume the causal relationship between two social science variables 𝑋 and 𝑌 can be estimated by their observed relationship [first panel of Figure 1]... However, social scientists usually doubt this simple picture and believe most variables share unmeasured confounders 𝑈 (second panel). One strategy for conducting causal analysis in the presence of endogeneity is using an instrumental variable 𝑊 that causally affects 𝑋 but is uncorrelated with the error term... One of the most important assumptions for any instrumental variable estimation is the exclusion restriction that 𝑊 is associated with 𝑌 only through its relationship with 𝑋 (i.e. there are no other causal pathways from 𝑊 to 𝑌). The assumed DAG for the IV estimation is shown in figure 1’s third panel...

The fourth panel of Figure 1 also demonstrates a problem, where the instrumental variable W affects some other variable Z, which in turn has a direct effect the outcome variable Y.

Mellon's contribution in this paper is to draw attention to the fact that weather variables (mainly rainfall, but also other variables like temperature, wind speed or direction, sunlight, or various others) have been used in so many applications as the variable W, that they must surely have effects on almost every outcome variable Y that don't run only through the variable X. It's kind of an obvious point when you think about it, and Mellon uses the results from over 150 papers to illustrate it, concluding that:

Cunningham (2018) argues that a good instrument should have a “certain ridiculousness”. Until the secret endogenous route to causation is explained, the link between the instrument and outcome seem absurd. In a world where Australians and Californians cannot leave their houses for months at a time due to forest fires, and 1-3 billion people are projected to be left outside of historically-habitable temperature ranges... linkages between weather and the social world are just not ridiculous enough.

Mellon uses weather instruments as his example, but the point he is making is broader. We need to be much more critical of the instrumental variables that are employed. He even offers a simple literature-search-based algorithm for determining whether a proposed instrumental variable is likely to fail the exclusion restriction, which can be used alongside the usual theoretical justification for its use. 

Certainly, it is time to reconsider whether weather variables are valid instruments. Only time (and further criticism along the lines that Mellon has advanced) will determine whether we should be equally sceptical of instrumental variables analysis more generally.

[HT: Marginal Revolution]