It may be time to reconsider weather variables as instruments

For many years, I was sceptical of instrument variables analysis. I expressed a little of this scepticism in one of my early posts on this blog in 2014. However, by then I was starting to come around to the idea, and encouraging my PhD students to consider using it in their work. However, I may have been shifted a little more back towards scepticism by this new working paper by Jonathan Mellon (West Point).

Mellon focuses on the use of weather variables as instruments, and demonstrates the problems associated with using them. However, before he gets that far, he has a very clear exposition of what instrumental variables entails, which is worth sharing. First, here's Figure 1 from the paper:

Then, the associated explanation:

Endogeneity is one of the most pervasive challenges faced by social scientists. Naively, we might assume the causal relationship between two social science variables 𝑋 and 𝑌 can be estimated by their observed relationship [first panel of Figure 1]... However, social scientists usually doubt this simple picture and believe most variables share unmeasured confounders 𝑈 (second panel). One strategy for conducting causal analysis in the presence of endogeneity is using an instrumental variable 𝑊 that causally affects 𝑋 but is uncorrelated with the error term... One of the most important assumptions for any instrumental variable estimation is the exclusion restriction that 𝑊 is associated with 𝑌 only through its relationship with 𝑋 (i.e. there are no other causal pathways from 𝑊 to 𝑌). The assumed DAG for the IV estimation is shown in figure 1’s third panel...

The fourth panel of Figure 1 also demonstrates a problem, where the instrumental variable W affects some other variable Z, which in turn has a direct effect the outcome variable Y.

Mellon's contribution in this paper is to draw attention to the fact that weather variables (mainly rainfall, but also other variables like temperature, wind speed or direction, sunlight, or various others) have been used in so many applications as the variable W, that they must surely have effects on almost every outcome variable Y that don't run only through the variable X. It's kind of an obvious point when you think about it, and Mellon uses the results from over 150 papers to illustrate it, concluding that:

Cunningham (2018) argues that a good instrument should have a “certain ridiculousness”. Until the secret endogenous route to causation is explained, the link between the instrument and outcome seem absurd. In a world where Australians and Californians cannot leave their houses for months at a time due to forest fires, and 1-3 billion people are projected to be left outside of historically-habitable temperature ranges... linkages between weather and the social world are just not ridiculous enough.

Mellon uses weather instruments as his example, but the point he is making is broader. We need to be much more critical of the instrumental variables that are employed. He even offers a simple literature-search-based algorithm for determining whether a proposed instrumental variable is likely to fail the exclusion restriction, which can be used alongside the usual theoretical justification for its use. 

Certainly, it is time to reconsider whether weather variables are valid instruments. Only time (and further criticism along the lines that Mellon has advanced) will determine whether we should be equally sceptical of instrumental variables analysis more generally.

[HT: Marginal Revolution]

The impact of remote learning in Brazilian high schools

Last week, I wrote a post expressing some frustration with a paper that purported to show the effect of online teaching on student learning, but really only showed the effect of online revision materials. I lamented that:

We do need more research on the impacts of online teaching and learning. However, this research needs to actually be studies of online teaching and learning, not studies of online revision.

Now, this new article by Guilherme Lichand, Carlos Alberto Doria, Onicio Leal-Neto (all University of Zurich), and João Paulo Cossi Fernandes (Inter-American Development Bank), published in the journal Nature Human Behaviour (open access) is much more of what I was looking for, and what we need. Lichand et al. look at the effect of the lockdown-induced shift to remote learning on students in high schools in São Paulo State, Brazil. They follow a similar difference-in-differences strategy to the paper I referred to last week, but instead of comparing students from schools with different access to materials, before and during the pandemic lockdowns, they compare students' performance between the first quarter and fourth quarter of the school year in 2020 (when the lockdowns were in place for the last three quarters) with 2019 (when there were no lockdowns). Their data covers over 8.5 million quarterly observations of 2.2 million students enrolled in sixth through twelfth grades. That also allows Lichand et al. to do a further comparison between students in middle schools and high schools, because:

...some municipalities allowed in-person optional activities (psycho-social support and remedial activities for students lagging behind) to return for middle-school students and in-person classes to return for high-school students...

So, comparing middle school and high school students' performance in Q1 and Q4 between districts that did and did not allow a return to in-person classes for high school students (in a 'triple differences' model) allows a further test of the effect of in-person classes. However, the results on this latter analysis are not as robust, as Lichand et al. don't know precisely which schools went back to in-person schooling (only which districts would allow it). In both sets of analyses, their outcome variables are the risk of dropout, and standardised test scores (and they observe test scores for about 83.3 percent of the sample).

In the first set of comparisons, they find that:

...remote learning might have had devastating effects on student dropouts, as measured by the dropout risk, which increased significantly during remote learning, by roughly 0.0621 (s.e. 0.0002), a 365% increase (significant at the 1% level...)... this result is suggestive of student dropouts within secondary education in the State having increased from 10% to 35% during remote learning...

The differences-in-differences strategy, in turn, uncovers dramatic [learning] losses of 0.32 s.d. (s.e. 0.0001), significant at the 1% level, a setback of 72.5% relative to the in-person learning equivalent.

Those are some huge negative effects of remote learning. Turning to the second comparison, of the effect of returning to in-person classes on student learning, they find:

...positive treatment effects on learning, fully driven by high-school students. In municipalities that authorized high-school classes to return from November 2020 onwards, test scores increased on average by 0.023 s.d. (s.e. 0.001, significant at the 1% level...), a 20% increase relative to municipalities that did not.

So, students managed to recover over half of the learning losses when in-person classes resumed. This is the good news part of this paper, especially since:

In municipalities that authorized schools to reopen for in-person academic activities in 2020, the average school could have done so for at most 5 weeks.

So, it didn't take long to erase much of the negative impact of remote teaching on learning. However, there was no significant effect on dropout risk, so presumably students who were likely to drop out did not reconsider their choice once schools had returned to in-person instruction.

The evidence is becoming clearer, and these results are in line with those from the literature on university-level students. Remote teaching has had a substantial negative effect on student learning. However, what was missing from this paper was an analysis of the heterogeneous effects between good students and not-so-good students. I expect that the dropout risk, and probably the learning losses, were heavily concentrated in the latter. What would have been most interesting would be whether the recovery in learning after the return to in-person teaching was also concentrated among the low-performing students. Perhaps future studies will help to reveal that.

Read more:

• Online vs. blended vs. traditional classes
• Flipped classrooms work well for top students in economics
• Flipped classrooms are still best only for the top students
• Online classes lower student grades and completion
• Online classes may also make better students worse off
• Meta-analytic results support some positive effects of using videos on student learning
• New review evidence on recorded lectures in mathematics and student achievement
• Meta-analytic results may provide some support for flipping the classroom
• The value of an in-person university education
• The pandemic may have revealed all we need to know about online learning
• Live streamed video lectures and student achievement
• Low-performing students, online teaching, and self-selection
• A slightly more optimistic take on online learning during the pandemic
• Research-based instructional strategies and flipped classrooms in economics
• Using microcommitments and social accountability to improve online and hybrid learning
• What pandemic schooling tells us about online learning
• We should be careful not to conflate the effect of online revision with the effect of online teaching

Gib delivery workers may be due for a payday

The New Zealand Herald reported last week:

Workers who deliver hundreds of tonnes of Gib to building sites across Auckland each day are striking for better pay.

About 40 truck drivers and labourers are picketing outside the Penrose base of the delivery company CV Compton.

They want an 11 per cent pay rise, but the company has offered much less...

It took time to train workers to deliver the plasterboard but they often lasted less than a week on the job because it was heavy labour, [Driver assistant James] Ramea said...

Gib was in demand and those delivering the plasterboard were working hard, [First Union organiser Emreck Brown] said.

"Prices of Gib has increased in the last couple of years and this year it has increased significantly. We need some support from the company just to help the members who're helping the company."

In a search model of the labour market, each match between a worker and an employer creates a surplus, which is then shared between the worker and the employer. The share of the surplus (and hence, the wage for the job) will depend on the relative bargaining power of the worker and the employer. If the worker has relatively more bargaining power, then they will receive a higher share of the surplus, in the form of a higher wage.

In this case, there is reason to believe that the workers' bargaining power has increased. That isn't because "those delivering the plasterboard were working hard", or even because it takes "time to train workers to deliver the plasterboard but they often lasted less than a week on the job because it was heavy labour". Those factors likely haven't changed recently.

What has changed is two things. First, the unemployment rate is low. Low unemployment increases the relative bargaining power of workers, because if a worker leaves their job (or refuses an employment offer), the employer then has to start the process of searching for a new worker all over again. The employer would face the search costs of the time, money, and effort spent searching for a worker and evaluating potential matches.

Second, because the "Prices of Gib has increased in the last couple of years", the value that the workers create for the employer have increased. That in itself doesn't affect wages in a search model of the labour market (although it does in a supply and demand model, where the demand for labour is based on the value of the marginal product of labour). However, because the workers are threatening to strike, the costs of the strike to the employer are likely higher because of the high value of gib deliveries foregone. That also increases the relative bargaining power of the workers.

None of this is to say that the gib delivery workers are going to see a huge increase in their wages. Employers tend to retain most of the bargaining power. However, the gib delivery workers have a bit more bargaining power than they would have had until relatively recently, and should be able to leverage that additional bargaining power for better wages and conditions.

The South Korean kimchi crisis

The Washington Post reported this week (possibly paywalled for you):

In the foothills of the rugged Taebaek range, Roh Sung-sang surveys the damage to his crop. More than half the cabbages in his 50-acre patch sit wilted and deformed, having succumbed to extreme heat and rainfall over the summer.

“This crop loss we see is not a one-year blip,” said Roh, 67, who has been growing cabbages in the highlands of Gangwon province for two decades. “I thought the cabbages would be somehow protected by high elevations and the surrounding mountains.”

With its typically cool climate, this alpine region of South Korea is the summertime production hub for Napa, or Chinese cabbage, a key ingredient in kimchi, the piquant Korean staple. But this year, nearly half a million cabbages that otherwise would have been spiced and fermented to make kimchi lie abandoned in Roh’s fields. Overall, Taebaek’s harvest is two-thirds of what it would be in a typical year, according to local authorities’ estimates.

The result is a kimchi crisis felt by connoisseurs across South Korea, whose appetite for the dish is legendary. The consumer price of Napa cabbage soared this month to $7.81 apiece, compared with an annual average of about $4.17, according to the state-run Korea Agro-Fisheries Trade Corp.

The effects of poor weather on the markets for cabbage and kimchi can be easily analysed using the supply and demand model that my ECONS101 class covered the week before last. This is shown in the diagram below. Think about the market for cabbage first. The market was initially in equilibrium, where demand D0 meets supply S0, with a price of P0 and a quantity of cabbage traded of Q0. Bad weather reduces the cabbage harvest, decreasing supply to S1. This increases the equilibrium price of cabbage to P1, and reduces the quantity of cabbage traded to Q1.

Now consider the market for kimchi. The costs of producing kimchi have increased. That leads to a decrease in the supply of kimchi. The diagram for the market for kimchi is the same as that for cabbage, with the equilibrium price increasing, and the quantity of kimchi traded decreasing. At least, that is the case for kimchi made from cabbage. Kimchi can also be made from other vegetables. The Washington Post article notes that:

The fermented pickle dish can also be made from radish, cucumber, green onion and other vegetables.

What happens in the markets for kimchi made from radishes? That is shown in the diagram below. Radish kimchi is a substitute for cabbage kimchi. Since radish kimchi is now relatively cheaper than cabbage kimchi, some consumers will switch to using radish kimchi. The effect is shown in the diagram below, where the radish kimchi market is initially in equilibrium with a price of PA, and a quantity of radish kimchi traded of QA. This increases the demand for radish kimchi from DA to DB, increasing the equilibrium price of radish kimchi from PA to PB, and increasing the quantity of radish kimchi traded from QA to QB.

The South Korean kimchi crisis is echoing through all types of kimchi, even if it is just the cabbages that are affected.

[HT: Marginal Revolution]