When we first went into lockdowns in 2020, and teaching and assessment all shifted online for an extended period, I advocated for a shift to pass/fail grading. In my view, we couldn't have the same level of confidence that letter grades would adequately represent the relative strengths or weaknesses of students, either within or between papers. Lecturers were under pressure and assessment was sub-optimal in the new environment. Students were under pressure and adjusting to learning from home in conditions that were often not conducive for good learning. Under such circumstances, an A+ grade becomes almost meaningless.
The lockdowns clearly presented an extraordinary situation. In general, I am not a fan of pass/fail grading of papers (except for Masters and PhD theses, where letter grades make little sense). The reason is the incentive effects that pass/fail grading creates, relative to letter grading. When students receive a letter grade, students across the whole distribution have an incentive to work a little bit harder, because additional effort can raise their letter grade. However, when grading is pass/fail, only students at the margin of passing or failing the paper have an incentive to work a bit harder. Students who are clearly passing the paper have little incentive to work harder, since there is little reward for doing so. Or, at least, there is little extrinsic reward for doing so. Some students will remain intrinsically motivated to work hard, or see working harder and learning at a higher level to be rewarding in the sense of better job opportunities at graduation.
Is there evidence to support these incentive effects? This recent NBER Working Paper (ungated version here), by Kristin Butcher, Patrick McEwan, and Akila Weerapana (all Wellesley College) provides some evidence in support of the theory. They look at what happened when Wellesley College moved all first-year courses to mandatory pass/fail grading in Fall 2014. As they explain:
Beginning in Fall 2014, the College further implemented a shadow grading policy for first-year, first-semester students. Under this policy, transcripts record a pass if students receive a letter grade of D or above, and a fail if they receive an F. However, students are privately notified of the letter grade. The policy has two objectives. The first is to encourage students to take courses outside their usual preferences. Curricular exploration might foster—in the short- and longer-run—increased student engagement with fields in which they are under-represented (such as women in mathematically-intensive STEM majors). The second is to promote successful transitions from high school to college, thereby preventing leaves of absence and drop out.
Importantly, lecturers still recorded the underlying letter grade. So, Butcher et al. are able to compare the grades that students earned under pass/fail grading, with the grades that students in earlier semesters earned under letter grading. Based on a sample of 38,214 student-by-course observations covering the period from 2004 to 2019, they find that:
There are more consequential results for student grades. The policy lowered the average grade points of first-semester students by 0.13 or about 23% of a standard deviation in pre-policy grades, although effects on the cumulative grade point average were small and not statistically distinguishable from zero.
No surprises that letter grades were lower as a result of the change in policy. However, the lack of an effect on cumulative GPA is surprising. Butcher et al. offer that:
There are several possible explanations for the grade effects on first-year students. The first is compositional: students sorted into lower-grading STEM courses which mechanically lowered average grades of first-semester students. However, we show that compositional changes explain a reduction of less than 0.01, implying a substantial role for within-course effects.
We next consider the possibility that grade reductions are due to lower quality instruction... However, the estimates are not sensitive to the inclusion of controls for quality proxies. We further show that the policy did not affect the grade performance of all students in a course, as might be expected if quality uniformly declined. Rather, it lowered the grades of first-semester students relative to later-semester students enrolled in the same courses.
Finally, we rule out the possibility that the policy led instructors to arbitrarily modify their grading standards for first-semester students relative to later-semester students...
The remaining and most plausible explanation for the policy-induced reduction in grades is that students covered by mandatory pass/fail grading exerted less effort relative to letter-graded students.
Butcher et al. support their final assertion that lower grades were a result of reduced effort by showing that course evaluations did not change. They argue that if grades were lower, but students were exerting the same effort, students would respond by evaluating their courses more negatively. That argument is not entirely convincing for me. However, Butcher et al. provide some additional:
...descriptive evidence from a faculty survey in 2017, which provided detailed examples of how students reduced effort, including attendance and course preparation.
Of course, the perception of student effort by lecturing staff may be affected by students' grades, rather than the other way around. There is little that Butcher et al. can do without a more objective measure of student effort. Perhaps if class attendance had been recorded, they could have used that?
At first glance, the lack of a statistically significant effect on cumulative GPA is a little problematic. However, pass/fail grades don't contribute to cumulative GPA, so what that really shows is that there is no continuing effect of the change to pass/fail after students move on to courses that are letter graded. The incentive effects are purely concentrated in the courses that are graded pass/fail, and if those courses are prerequisites for courses that students complete later in their degree, the lower effort in their pass/fail courses doesn't appear to hold them back.
Overall, the takeaway message from this paper is that universities should be very cautious about employing a pass/fail grading system, as it appears to disincentivise student learning within the courses where it is applied.
[HT: Marginal Revolution]
No comments:
Post a Comment