Could early identification of students at risk of failure be a bad thing?

One of my Summer Research Scholarship students this year, Jacinda Herring, was working on a project on identifying  the characteristics of Waikato Management School students at risk of not completing their degree. The point of the project is, essentially, that if we can identify high-risk students before they start their degree, then we can better target pastoral care or other interventions that might help increase students' chances of successful degree completion. It seems to me there is little to argue against this approach, which is why I am pursuing research in this area with my students.

However, a recent interview with Jeffrey Alan Johnson (Utah Valley University) published in the Christian Science Monitor argues convincingly that perhaps we should pause before we get carried away with 'profiling' our students. Johnson says:

We've got an early warning system [called Stoplight] in place on our campus that allows instructors to see what a student’s risk level is for completing a class. You don’t come in and start demonstrating what kind of a student you are. The instructor already knows that. The profile shows a red light, a green light, or a yellow light based on things like have you attempted to take the class before, what’s your overall level of performance, and do you fit any of the demographic categories related to risk. These profiles tend to follow students around, even after folks change how they approach school. The profile says they took three attempts to pass a basic math course and that suggests they’re going to be pretty shaky in advanced calculus...

When I told my students I have Stoplight data, they were worried about what I thought of them coming into the class. It definitely bothered them. They wondered if instructors will think they need help, or dismiss them because it looks like they won’t succeed and it’s better to prioritize other students.

So it seems that maybe there are valid concerns about making this data available to teaching staff. Lecturers' attention is a scarce resource, and if lecturers know which students are more likely to fail a given course, then they may divert their attention away from most-at-risk students to students who are more likely to pass. Of course, the counter-argument is that maybe some lecturers would divert their attention towards those who are on the margin of passing the course (to the extent that pass rates, or helping individual students to pass, are important to lecturers). But the risk of profiling isn't a good reason not to have a system for identifying at-risk students. Perhaps the data on risk level could be made available only to student advisors or those engaged in pastoral care, which would minimise the risk to students of their interactions with lecturers being biased by preconceptions of their likelihood of passing. In any case, my aim is to press on with research into at-risk students later this year. 

Coming back to Jacinda's project, she compiled administrative data from all Waikato Management School students who commenced study between 2008 and 2011 (n = 2033). Each student was then classified into three categories: (1) completed any degree (not necessarily the one they started in); (2) still studying (in 2014); and (3) did not complete any degree and not still studying. She then used chi-square tests and logistic regression models to compare the first group with the latter two (combined).

What did she find? In the final (multivariate) specification of the logistic regression model (which only included data we would have known before the students commenced study, and data that are available for all students):

• Students aged 25 years and over (at first enrolment) had significantly lower odds of degree completion than those aged 19 years and under;
• Male students had significantly lower odds of degree completion than female students;
• Asian students had significantly higher odds of degree completion than all other ethnic groups, and Maori and Pacific Island students had the lowest odds of degree completion;
• Domestic students had significantly lower odds of degree completion than international students;
• Special admission (or provisional entrance) students had significantly lower odds of degree completion than other students; 
• Students who initially completed the Certificate of University Preparation (CUP) had significantly lower odds of degree completion than other students; and
• Students initially enrolled in conjoint degrees had significantly lower odds of degree completion than students enrolled in single degrees.

The results are altogether not surprising if you have any experience with tertiary education, except perhaps for the last one. Most of the time it is top-achieving students who enrol in conjoint degrees. However, many students enrol in conjoint degrees because they can't decide on a single degree that they want to specialise in and so try to do a bit of everything. Conjoint degrees take longer to complete, and as such require a higher level of commitment to study - this may lead to discouragement and a higher level of non-completion. So, at the least there is one take-away from Jacinda's work, which is that maybe we need to target more pastoral care or mentoring and role models for conjoint degree students.

I expect that Jacinda and I will write up her analysis as a Department of Economics working paper in the near future.

[HT for the CSM article: Marginal Revolution]