In my ECONS102 class this week, among other things we covered a bunch of concepts in behavioural economics. One such concept was loss aversion - the idea that people value losses much more than equivalent gains (in other words, they like to avoid losses much more than they like to capture otherwise equivalent gains). Loss aversion seems to explain a lot of quasi-rational behaviour (however, as a concept, loss aversion is contested - more on that in another post later this week). People do seem to adjust their behaviour to try and avoid losses.
One example is the answering of multiple choice questions in tests and exams, but only where a wrong answer results in a penalty (subtracting marks from the overall test or exam score). I never grade multiple choice in that way, but many academics do. The argument they put forward is that, when incorrect answers are penalised, it creates a disincentive to students guessing. However, because the penalty creates a loss, it may be that loss averse students avoid answering when they are a little bit unsure, even if their not-quite-sure guess would have been the correct answer.
So, how much does loss aversion affect students' multiple choice answering behaviour? That is the question that this recent article by Heiko Karle (Frankfurt School of Finance and Management), Dirk Engelmann (Humboldt-Universität zu Berlin) and Martin Peitz (University of Mannheim), published in the Scandinavian Journal of Economics (ungated earlier version here), sets out to answer. Karle et al. use data from 646 students, combining an experimental-based measure of loss aversion (that the students completed early in the semester) with the results of a 30-question multiple-choice examination (held some three months later), where:
There are four possible answers to each question: a correct answer gives three points, no answer one point, and an incorrect answer gives zero points, as in the exam in our data set.
Notice that, on the face of it, there is no penalty for an incorrect answer, since an incorrect answer receives zero points. However, choosing not to answer yields one point, so considering not answering as the status quo reference point, choosing an incorrect answer makes a student worse off by one point (a loss, which they would try to avoid). Karle et al. hypothesise that students who are more loss averse will answer fewer questions, and will get more of the questions that they do answer correct. Both hypotheses are obvious from loss aversion - students try to avoid the loss, so are more likely not to answer questions when they are unsure, so more loss averse students will answer fewer questions. That means that the questions that more loss averse students do answer are those that they are surer about, and so they are more likely to get those answers correct. Karle also test a related hypothesis, that students who are more loss averse get fewer questions correct overall, which depends on how many the students answer (fewer, when the students are more loss averse) and how many of the ones they do answer are answered correctly (more, when the students are more loss averse). What to expect for this overall hypothesis is unclear.
In addition to loss aversion, Karle et al. measure students' self-confidence, which is based on the difference between students' estimates of the percentage of their own correct answers to a set of general interest questions and the average percentage of other students’ correct answers. Students who are more self-confident can be expected to be more likely to answer questions in the multiple choice exam.
Now, using their data, Karle find that:
...loss aversion and confidence have a negative and positive effect, respectively, on the number of answered questions... This effect is statistically significant at the 1 percent level... Our estimates suggest that loss neutral students answer approximately two questions more than otherwise identical students in the highest category of loss aversion (and 5/3 more than those in the middle category).
So, more loss averse students answer fewer questions than less loss averse students, as expected. Next:
We do not find a statistically significant effect (at the 5 percent level) of loss aversion or confidence on the ratio of correct answers per questions answered. However, the coefficient of loss aversion (but not strong loss aversion) turns positively significant at the 10 percent level when considering loss aversion and confidence together...
This is very weak support (or, rather, no support) for the hypothesis that more loss averse students get more of the questions that they do answer correct. Combining the two hypotheses, I'm sure you can guess for the overall hypothesis, that Karle et al. find that:
...loss aversion and confidence have a negative and positive effect, respectively, on the dependent variable [the number of correct questions overall]... Our estimate suggests that, ceteris paribus, students in the highest category of loss aversion give approximately 1.5 fewer correct answers than otherwise identical students who are loss neutral.
Karle et al. then spend a bit of effort trying to determine whether the effect of loss aversion on question answering behaviour is causal or not, and their results suggest that it is causal for some, but not all, students. I don't find those results as convincing as the overall takeaway that students' loss aversion is related to how they answer questions.
Nevertheless, these results are interesting even putting aside the question of causality. Loss aversion isn't something that is easy to change, so even a correlation between loss aversion and question answering behaviour is potentially important. And it may be even more important given that Karle et al. show that female students in their sample are more loss averse than male students. So, the effect of this style of grading multiple choice questions is a bias against female students, decreasing female grades relative to male grades. That's the last thing we need in economics. That may have contributed to this:
In our setting, according to a university directive, the differential treatment of wrong responses and no responses was no longer allowed after the academic year 2013/2014, which is the exam year we used in this paper.
Coming back to my earlier point, many academics like the style of grading that doesn't award 'free' marks to student guessing. However, there is a trade-off. If students are penalised for guessing, they are likely also penalised based on how loss averse they are. So, if multiple choice questions come with a penalty, the exam score will be more precise for each question (in terms of telling the grader whether students actually were confident in the answer) but also more biased overall (because more loss averse students will get fewer questions correct than less loss averse students). The trade-off seems untenable to me. I'll continue to use multiple choice marking that implicitly includes a reward for guessing.
No comments:
Post a Comment