Thursday, 29 September 2022

The impact of remote learning in Brazilian high schools

Last week, I wrote a post expressing some frustration with a paper that purported to show the effect of online teaching on student learning, but really only showed the effect of online revision materials. I lamented that:

We do need more research on the impacts of online teaching and learning. However, this research needs to actually be studies of online teaching and learning, not studies of online revision.

Now, this new article by Guilherme Lichand, Carlos Alberto Doria, Onicio Leal-Neto (all University of Zurich), and João Paulo Cossi Fernandes (Inter-American Development Bank), published in the journal Nature Human Behaviour (open access) is much more of what I was looking for, and what we need. Lichand et al. look at the effect of the lockdown-induced shift to remote learning on students in high schools in São Paulo State, Brazil. They follow a similar difference-in-differences strategy to the paper I referred to last week, but instead of comparing students from schools with different access to materials, before and during the pandemic lockdowns, they compare students' performance between the first quarter and fourth quarter of the school year in 2020 (when the lockdowns were in place for the last three quarters) with 2019 (when there were no lockdowns). Their data covers over 8.5 million quarterly observations of 2.2 million students enrolled in sixth through twelfth grades. That also allows Lichand et al. to do a further comparison between students in middle schools and high schools, because:

...some municipalities allowed in-person optional activities (psycho-social support and remedial activities for students lagging behind) to return for middle-school students and in-person classes to return for high-school students...

So, comparing middle school and high school students' performance in Q1 and Q4 between districts that did and did not allow a return to in-person classes for high school students (in a 'triple differences' model) allows a further test of the effect of in-person classes. However, the results on this latter analysis are not as robust, as Lichand et al. don't know precisely which schools went back to in-person schooling (only which districts would allow it). In both sets of analyses, their outcome variables are the risk of dropout, and standardised test scores (and they observe test scores for about 83.3 percent of the sample).

In the first set of comparisons, they find that:

...remote learning might have had devastating effects on student dropouts, as measured by the dropout risk, which increased significantly during remote learning, by roughly 0.0621 (s.e. 0.0002), a 365% increase (significant at the 1% level...)... this result is suggestive of student dropouts within secondary education in the State having increased from 10% to 35% during remote learning...

The differences-in-differences strategy, in turn, uncovers dramatic [learning] losses of 0.32 s.d. (s.e. 0.0001), significant at the 1% level, a setback of 72.5% relative to the in-person learning equivalent.

Those are some huge negative effects of remote learning. Turning to the second comparison, of the effect of returning to in-person classes on student learning, they find:

...positive treatment effects on learning, fully driven by high-school students. In municipalities that authorized high-school classes to return from November 2020 onwards, test scores increased on average by 0.023 s.d. (s.e. 0.001, significant at the 1% level...), a 20% increase relative to municipalities that did not.

So, students managed to recover over half of the learning losses when in-person classes resumed. This is the good news part of this paper, especially since:

In municipalities that authorized schools to reopen for in-person academic activities in 2020, the average school could have done so for at most 5 weeks.

So, it didn't take long to erase much of the negative impact of remote teaching on learning. However, there was no significant effect on dropout risk, so presumably students who were likely to drop out did not reconsider their choice once schools had returned to in-person instruction.

The evidence is becoming clearer, and these results are in line with those from the literature on university-level students. Remote teaching has had a substantial negative effect on student learning. However, what was missing from this paper was an analysis of the heterogeneous effects between good students and not-so-good students. I expect that the dropout risk, and probably the learning losses, were heavily concentrated in the latter. What would have been most interesting would be whether the recovery in learning after the return to in-person teaching was also concentrated among the low-performing students. Perhaps future studies will help to reveal that.

Read more:

at

No comments:

Post a Comment