Some students (perhaps many) think that taking the easy route to a degree is the best way to achieve their goals (of graduating, and getting a good job). However, as I have written about before, choosing more difficult majors, or more difficult classes, provides a signalling advantage to students who are willing to put in the effort. That post was based on the theory of adverse selection and signalling, but now I have some good empirical results to back it up.
Those results come from this working paper by Michael Insler (US Naval Academy), Alexander McQuoid, Ahmed Rahman (Lehigh University), and Katherine Smith (US Naval Academy). They look at 21 years of data from students at the US Naval Academy (1997 to 2017, with over 51,000 observations), where the sequencing of core courses is imposed on students, and where students are randomly allocated to sections and to lecturing staff. That gives them the advantage of being able to follow students across courses, and see how lecturing staff in early courses affect performance in later courses, knowing that there will be no selection bias, because students cannot choose their courses or lecturers. And interestingly:
In a sequential learning framework we explore the channels through which instructor treatment in an initial period influences performance in the follow-on course in the sequence. Specifically, we decompose the “value-added” of each first-semester instructor into hard-skill and soft-skill components, each of which affects the student’s subsequent performance...
Given that instructors determine the final grades of their students, there are both objective and subjective components of any academic performance measure. For a subset of courses in our sample, however, final exams are created, administered, and graded by faculty who do not directly influence the final course grade. This enables us to disentangle faculty impacts on objective measures of student learning within a course (grade on final exam) from faculty-specific subjective grading practices (final course grade).
Using the objectively determined final exam grade, we measure the direct impact of the instructor on the knowledge learned by the student. We will refer to this dimension of faculty quality as the “hard skills channel”...
Beyond this hard skills channel though, faculty can also shape student behaviors that are important for longer-run success. These may include how to allocate time to study, how to learn independently and without much hand-holding, and how to distinguish between easy and difficult subject areas. We will label the effect by which faculty may impair such skills as the “soft standards channel”. When faculty set expectations that students do not need to put in significant effort to succeed in a discipline, and reward such behavior with easier grades, such low expectations may harm student performance in follow-on courses. To disentangle this effect from the hard skills channel, we use the subjective measure of professor quality stemming from the instructor-determined final course grade to separately identify the soft standards channel.
Essentially, in the first part of the paper, Insler et al. try to disentangle the effects of having a lecturer who teaches well (and leads to a high, objectively measured, exam grade) from the effects of having a lecturer who sets low standards (and leads to a high, more-subjectively measured, coursework grade). The outcomes are measured in the second course in each sequence. Insler et al. find that:
...the impact of soft standards... has a negative effect on sequential learning, while the impact of hard skills... has a positive effect.
Comparing STEM (Science, Technology, Engineering and Mathematics) and non-STEM courses, they find:
...no differential effect for the hard skills channel, but we do find that the soft standards channel operates differently between STEM and non-STEM courses. The sequential impact of the soft standards effect for a non-STEM course is -0.14 compared to -0.08 for a STEM course.
They also find differences by gender:
In terms of the hard skills channel, female students who have faculty that are higher quality teachers do even better in follow-on courses than men who have the same high quality faculty. When looking at the soft standards channel, however, the impact of lower standards faculty is significantly larger for women when compared to men.
Insler et al. then introduce data on student perceptions into the analysis. The data are drawn from ratemyprofessor.com, and include a measure of overall quality (essentially a measure of how popular the lecturer is) and a measure of the level of difficult of the lecturer's courses. Including those measures in their analysis, Insler et al. find that:
The impact of a professor who is deemed difficult is better for subsequent learning in follow-on courses, and the magnitude of the effect is about four times larger than the impact of a professor with a higher overall rating.
So, it is better for students to take a difficult course, than to take a course with a popular lecturer. However, it is possible for lecturers to be popular and challenging for students, or to be unpopular and easy. Insler et al. look across combinations of popularity and difficulty, noting that:
Faculty that are both well-liked and considered very difficult (top 25% of each RMP rating distribution) are likely what most faculty aspire to be: challenging and demanding, but generating devotion and enthusiasm based on superior teaching. Such unicorns are extremely rare in the data, making up just under 2% of the overall faculty, and just over 1% of the total number of observations. We find no evidence that these faculty impact sequential learning, although this may be related to the small sample size. The other three groupings (“High Difficulty, Low Overall”; “Low Difficulty, Low Overall”; “Low Difficulty, High Overall”), however, are all statistically different from the excluded group of faculty who are in the middle of the distribution on at least one of these dimensions.
Which faculty are most associated with sequential learning? Those faculty who bundle together characteristics of high difficulty and low likability. One interpretation of this finding is that poor teachers are considered to be difficult by students because of lack of clarity in lecture and course structure, and this experience pushes students to invest in studying on their own to learn the material, resulting in deeper learning which is carried through to the next semester. However, we suspect that the issue is more likely to be explained by faculty who demand a lot of their students, forcing students to exert costly effort. This learning by effort leads to deeper learning and sequential success, but also engenders animosity towards the professor, resulting in high difficulty and low overall ratings...
...faculty who are considered very easy and poor overall do notable damage to sequential learning. These faculty are in fact likely to be poor teachers, who perhaps minimize effort themselves, resulting in an easy and poor course.
Reflecting on my own teaching, I like to think I challenge students, and I know that my teaching evaluations are among the top in my School. I don't think I'm a unicorn though? Still:
Anyway, I'm hopefully not among the worse group, where Insler et al. note that:
...the grouping that appears to be most problematic are those with high overall ratings and low difficulty ratings. These faculty severely harm sequential learning, and more perniciously, are likely to be faculty who are praised by administrators for achieving high engagement from students (expressed through high overall opinions by students). These faculty - which we dub “a Seemingly Conscientious and Hardworking Mentor, an Obtuse and Perfunctory Educator” or Schmopes... - are deeply problematic because they damage student learning and are elevated within the university system as role model faculty.
Why 'Schmopes'? Footnote 7 in the paper notes that:
Inspired by discussions with our students who justified the value of this type of professor by commenting that they give hope to students, one rather perspicacious student retorted with “hope, schmope.”
The paper goes further, to show that the 'hard skills' channel has persistent effects over time, but the 'soft standards' channel decays over time. So, that may be some good news. If students encounter a 'soft standards' lecturer early in their studies, a 'hard skills' lecturer can get them back on track. However:
...it is easy to imagine that soft standards may influence student choices in other dimensions, such as choice of major. Encountering a lenient instructor early on in a college career may influence a student to choose a major that is not well suited to the student’s comparative advantage...
Finally, Insler et al. show that the effects of soft standards are higher for extroverted students (presumably because introverted students can better overcome the soft standards effect through self-study), and for students who are more 'feeling' than 'thinking' on the thinking/feeling scale (presumably because those who are more 'feeling' than 'thinking' are influenced more greatly by their lecturers).
Overall, there is a lot of depth to this paper, and the lack of selection bias means that this is a paper that we should pay a lot of attention to. By that, I mean firstly that lecturers and university administrators should pay attention to it, because tough lecturers who set high standards may not be well liked by students, but are much better for their learning. We should therefore interpret student evaluations of teaching very carefully (if at all, as they have a lot of problems, such as those I have outlined here and here). Second, students should pay attention too. Taking easy courses, or courses with lecturers who are known to give easy grades, may actually make future courses more difficult to achieve well in because of the lower learning in the earlier courses (and this is on top of any negative signal that easy courses provide to future employers).
The final word: Beware the schmopes!
[HT: Marginal Revolution]
No comments:
Post a Comment