The impact of the coronavirus pandemic on university students and grades

The coronavirus pandemic, and the associated lockdowns and stay-at-home orders, had a substantial impact on teachers and learners at all levels. The anecdotal evidence for this is legion, and there is a growing academic literature as well (as one example, see my working paper on the impacts on New Zealand university students, co-authored with Barbara Fogarty-Perry and Gemma Piercy, here).

Like us, many researchers took the opportunity to field surveys of their students (our New Zealand survey was part of an international project - see here). One such researcher was Núria Rodríguez-Planas (City University of New York), who has had two new articles published, both based on the impact on students at Queens College (QC) at the City University of New York. The first article was published in the journal Economics of Education Review (ungated earlier version here), and focuses on survey evidence on the impacts reported by students (similar to my study linked above). The survey was fielded between July and September 2020, and received 3163 responses (a response rate of about 20 percent).

Now, measuring the impact of the pandemic and lockdowns is tricky, because we don't know what would have happened in the absence of the pandemic. Rodríguez-Planas argues that (emphasis is hers):

...to the extent that the student has private information on how his academic performance and economic wellbeing were prior to the pandemic, I can estimate an individual-level subjective treatment effect by asking students directly how the pandemic has changed their experience or perceptions. This is indeed what I did in the online survey...

It's a pretty weak justification, but we'll just accept it at face value, noting that we really don't know what the counterfactual is (and it would be very difficult to establish one, since all students everywhere were affected similarly). In terms of subjective impact then, Rodríguez-Planas finds that:

...as many as 34% of QC students considered dropping a class during the spring 2020 semester. The most frequent reasons students gave were being concerned that their grade would jeopardize their future financial assistance (27%), the need to work (9%), and the need to care for a sick family member (7%). In addition, some students also stated getting sick with COVID-19 (5%) or the need to move due to the pandemic (3%). Consistent with students’ concerns that the pandemic would hurt their grade and jeopardize their future financial assistance, 32% of the sample reported having difficulties maintaining their desired level of academic performance because of the pandemic and 22% reported having difficulties maintaining financial aid due to the pandemic.

However, the more interesting results arise when Rodríguez-Planas compares low-income students (proxied by those who had ever received a Pell grant) and high-income students (those who had never received a Pell grant). She finds that:

...Pell recipients were 12 percentage points more likely to consider withdrawing a class during the spring 2020 semester than students who never received the Pell grant. This represents a 41% increase relative to the average effect for never Pell recipients of 28.5%. This greater withdrawing consideration during the spring 2020 semester is driven by Pell recipients more likely to: (1) be concerned that their grade would jeopardize financial assistance (64%); (2) care for a family member (44%); and (3) get sick with COVID-19 (49%) than the never Pell recipients. These estimates are statistically significant at the 5% level or lower...

Pell recipients were 20% more likely to report losing a job due to COVID-19 and 17% more likely to report being laid-off or furloughed because of COVID-19 than never Pell recipients. They were also 17% more likely to report a reduction in wage and salary earnings than students who never received the Pell grant... Importantly, Pell recipients were 65% more likely to have faced food and shelter insecurity, and 17% more likely to have experienced difficulties to replace a lost job or internship than never Pell recipients. All these impacts are statistically significant at the 5% level or lower.

Then, turning to a comparison of first-in-family students with others, and transfer students with others, Rodríguez-Planas finds that:

First-generation students are more likely to report difficulties with academic performance and continuing college education than their counterfactual. They are also more likely to report changing their graduation plans by postponing graduation and considering graduate school than their counterfactual. Similarly, transfer students are more likely to report difficulties with continuing college education, changing their graduation plans by considering graduate school or taking more classes than their counterfactual.

So, in summary this paper tells us that the pandemic had a negative impact on students. However, this impact was not the same for all students. Low-income students and first-in-family students faced the most serious impacts, but were impacted in different ways. Low-income students faced larger financial impacts because of their financial vulnerability, while first-in-family students faced larger educational impacts because of their educational vulnerability. There is nothing that that we should not have been able to anticipate.

Rodríguez-Planas's second article was published in the Journal of Public Economics (sorry, I don't see an ungated version of this one online), and focuses on the impact on student grades, using administrative data from over 11,000 QC students over the period from Spring 2017 to Spring 2020. In particular, Rodríguez-Planas focuses on the difference in impact between low-income and high-income students (again proxied by Pell grant receipt or not). Using a difference-in-differences framework, she finds that:

...lower-income students outperformed higher-income ones during Spring 2020 as they earned a 5.1% higher GPA and they failed 28% fewer credits (or exercised fewer NC grades) than their wealthier counterparts. Both effects are statistically significant at the 5% level or lower. Lower-income students’ higher relative academic performance during the first semester of the pandemic is largely associated with the flexible grading policy as the GPA differential by income status vanishes (β2 is close to zero and not statistically significant) when I estimate the effect of the pandemic with the GPA prior to students exercising the flexible grading option.

To clarify, higher-income students performed better in Spring 2020 than in other semesters, but the increase in performance was even greater for lower-income students. That's because lower-income students took greater advantage of flexible grading options where they otherwise would have received a low grade for a particular course. Rodríguez-Planas goes on to examine the mechanisms using data from her survey, and finds that:

...because of the pandemic, top-performing lower-income students were 9.7 percentage points more likely to report asking for an incomplete than their bottom-performing lower-income peers and 5.8 percentage points more likely to ask for an incomplete than their higher-income peers... Both estimates are significant at the 1% level.

Rodríguez-Planas concludes that:

...the flexible grading policy was able to counteract negative shocks, especially among the most disadvantaged students. Because low-income students regularly face idiosyncratic challenges, the results in this paper suggest that a higher use of the pass/fail grade (if not for all courses, for certain courses) may support students during critical moments.

I found this second article and its conclusion interesting, not because it was about the impact on students, but because it illustrated the impact on grades. Grades have an element of subjectivity to them, and they can be manipulated by scaling, by grading policies, or by the strictness or leniency of individual lectures. The impact of the flexible grading option chosen by QC was of particular interest because switching to pass/fail grading was something I advocated for at Waikato (unsuccessfully), when we first went into lockdown in 2020. In my view, this would have reduced student stress, reduced teacher stress, avoided the rampant grade inflation that eventually occurred as students were scaled upwards (even when the grade distribution was already high due to the move to online open book assessment), and kept student GPAs sensible and informative. I would suggest that this was a good learning opportunity for the institution, but hopefully we never find ourselves in that situation again!