Many years ago (I forget exactly when), I introduced extra credit into my ECON110 class (which is what is now ECONS102). The idea was to provide an incentive for students to attend class, since they could earn extra credit for completing various in-class exercises. A couple of years later, I briefly changed the way that I framed the extra credit, from being "extra marks that would be gained from attending", to "extra marks that would be lost by not attending".
If students were purely rational, the change from 'gain framing' to 'loss framing' the extra credit should have had no impact on student attendance. However, I was looking to exploit the fact that most people are quasi-rational, rather than purely rational. Quasi-rational decision-makers are loss averse, meaning that they value losses more than equivalent gains. For a loss averse person, losing $20 makes them unhappy to a greater extent than winning $20 makes them happy.
Does a change from 'gain framing' to 'loss framing' work? That is the question that this new article by Antal Ertl, Éva Holb (both Eötvös Lóránd Science University), and Barna Bakó (Corvinus University of Budapest), published in the Journal of Economic Behavior and Organization (open access), tries to answer. They use data from a field experiment at Corvinus University of Budapest, where students enrolled in a compulsory macroeconomics course for business students were randomised into one of three conditions: (1) Gain group, which earned points in each of four tests and the final examination as usual; (2) Loss group, which started each test and the final exam with full points, but had points deducted for each incorrect answer; and (3) Hybrid group, which was the same as the Gain group for the tests, but switched to the loss framing for the final examination.
Ertl et al. have a sample of 321 students who consented to be part of the research, completed an initial questionnaire at the start of the term, and earned a non-zero grade. Randomisation was conducted at the level of the tutorial group (so all students in a tutorial were in the same treatment), in such a way that each teacher had groups across more than one treatment. One wrinkle in their analysis is that the best three out of the four tests would count towards a student's grade, meaning that students may end up putting differential effort into each test, depending on how they have performed in the other tests already completed. So, in addition to looking at the effect of treatment on each test mark individually, Ertl et al. look at the effect on the 'best three' tests collectively, as well as the exam mark.
If randomisation were perfect and the treatment groups were balanced, the comparison between the Loss group and the Gain group would demonstrate the overall effect of loss framing on student performance. The comparison between the Loss group and the Hybrid group for the final exam, compared with the same comparison for the best three tests, would demonstrate whether students adjust in such a way that the loss framing has less impact over time (because the Hybrid group would be in their first loss-framed assessment, while the Loss group would be in their fifth such assessment). The treatment groups weren't perfectly balanced, with students sorting into tutorial groups in part based on whether they worked part-time. So, Ertl et al. control for working part-time, the tutorial day and time, and the tutorial group teacher, as well as other demographic and background variables.
In their main analysis, they find support for the positive effects of loss framing:
For the Loss treatment, the effect on the average of the Best 3 Tests is 3.2 percentage points, although the difference is not statistically significant. The treatment effect on the Final Test score, however, shows a large difference of 9.6 percentage points when not controlling for Best 3 Tests’ scores, i.e., how well students did throughout the semester before the Final Test.
After controlling for performance in the best three tests, the effect of the loss framing on performance in the final examination is a statistically significant 7.8 percentage points. Turning to the comparison of the Loss and Hybrid groups, Ertl et al. find that:
...the estimated effect sizes for Loss and Hybrid are essentially the same for the Final Test, once we take into account how well students did perform throughout the semester...
These results are consistent with loss framing leading to better student performance, and there being no novelty effect - the effect of loss framing doesn't appear to decline over time. Ertl et al. go on to show that the effects are similar for both male and female students, but larger for students who did not take advanced mathematics in high school than for those that did. They also show that the treatment did not seem to negatively affect students' perceptions of the course, because the teaching evaluations were similar for the different treatment groups.
Finally, Ertl et al. do provide a note of caution in their conclusion:
previous studies have highlighted possible psychological and motivational costs associated with loss framing... These findings suggest that the mechanism by which loss framing improves performance may, at least in part, operate through heightened tension and concern about avoiding mistakes rather than through enhanced intrinsic motivation. Moreover, in extreme cases, loss-framed grading may even produce adverse effects — for example, low-performing students might become discouraged early in the semester after ‘‘losing’’ too many points. Once it becomes apparent that only a passing grade is attainable at best, the loss-framed structure may make this limitation increasingly salient, potentially exacerbating anxiety and disengagement. Over time, this could have broader implications for students’ well-being and their willingness to enroll in courses or programs that employ such systems.
Ertl et al. don't directly test for these effects, but they should be a concern. We may be able to improve student performance through loss-framing assessments, but that might come at a cost to student mental health and wellbeing.
And that brings me back to the example I started with, from my ECON110 class. When I switched extra credit from gain-framed to loss-framed, student attendance in class did improve slightly. However, the bigger impact seemed to be the number of students who would contact me by email, seeking special consideration for missing the extra credit, offering to provide medical certificates or other evidence to explain their absence, and asking for extra chances to complete the in-class exercises. It turned out to be administratively much more costly for me, and so the change was short-lived (to the extent that I cannot even remember which year I tried this in). Those reactions could suggest a negative psychological effect of the switch from gain framing to loss framing.
So, not all interventions that are effective for promoting student performance should be adopted. We need to carefully consider both the benefits and the costs of the intervention first. Taking advantage of student loss aversion might be worth exploring further, but I would want to see a wider evaluation that included student wellbeing outcomes before adopting it.
No comments:
Post a Comment