Teacher quality, and why students should avoid easy classes - Beware the schmopes!

Some students (perhaps many) think that taking the easy route to a degree is the best way to achieve their goals (of graduating, and getting a good job). However, as I have written about before, choosing more difficult majors, or more difficult classes, provides a signalling advantage to students who are willing to put in the effort. That post was based on the theory of adverse selection and signalling, but now I have some good empirical results to back it up.

Those results come from this working paper by Michael Insler (US Naval Academy), Alexander McQuoid, Ahmed Rahman (Lehigh University), and Katherine Smith (US Naval Academy). They look at 21 years of data from students at the US Naval Academy (1997 to 2017, with over 51,000 observations), where the sequencing of core courses is imposed on students, and where students are randomly allocated to sections and to lecturing staff. That gives them the advantage of being able to follow students across courses, and see how lecturing staff in early courses affect performance in later courses, knowing that there will be no selection bias, because students cannot choose their courses or lecturers. And interestingly:

In a sequential learning framework we explore the channels through which instructor treatment in an initial period influences performance in the follow-on course in the sequence. Specifically, we decompose the “value-added” of each first-semester instructor into hard-skill and soft-skill components, each of which affects the student’s subsequent performance...

Given that instructors determine the final grades of their students, there are both objective and subjective components of any academic performance measure. For a subset of courses in our sample, however, final exams are created, administered, and graded by faculty who do not directly influence the final course grade. This enables us to disentangle faculty impacts on objective measures of student learning within a course (grade on final exam) from faculty-specific subjective grading practices (final course grade).

Using the objectively determined final exam grade, we measure the direct impact of the instructor on the knowledge learned by the student. We will refer to this dimension of faculty quality as the “hard skills channel”... 

Beyond this hard skills channel though, faculty can also shape student behaviors that are important for longer-run success. These may include how to allocate time to study, how to learn independently and without much hand-holding, and how to distinguish between easy and difficult subject areas. We will label the effect by which faculty may impair such skills as the “soft standards channel”. When faculty set expectations that students do not need to put in significant effort to succeed in a discipline, and reward such behavior with easier grades, such low expectations may harm student performance in follow-on courses. To disentangle this effect from the hard skills channel, we use the subjective measure of professor quality stemming from the instructor-determined final course grade to separately identify the soft standards channel.

Essentially, in the first part of the paper, Insler et al. try to disentangle the effects of having a lecturer who teaches well (and leads to a high, objectively measured, exam grade) from the effects of having a lecturer who sets low standards (and leads to a high, more-subjectively measured, coursework grade). The outcomes are measured in the second course in each sequence. Insler et al. find that:

...the impact of soft standards... has a negative effect on sequential learning, while the impact of hard skills... has a positive effect.

Comparing STEM (Science, Technology, Engineering and Mathematics) and non-STEM courses, they find:

...no differential effect for the hard skills channel, but we do find that the soft standards channel operates differently between STEM and non-STEM courses. The sequential impact of the soft standards effect for a non-STEM course is -0.14 compared to -0.08 for a STEM course.

They also find differences by gender:

In terms of the hard skills channel, female students who have faculty that are higher quality teachers do even better in follow-on courses than men who have the same high quality faculty. When looking at the soft standards channel, however, the impact of lower standards faculty is significantly larger for women when compared to men.

Insler et al. then introduce data on student perceptions into the analysis. The data are drawn from ratemyprofessor.com, and include a measure of overall quality (essentially a measure of how popular the lecturer is) and a measure of the level of difficult of the lecturer's courses. Including those measures in their analysis, Insler et al. find that:

The impact of a professor who is deemed difficult is better for subsequent learning in follow-on courses, and the magnitude of the effect is about four times larger than the impact of a professor with a higher overall rating.

So, it is better for students to take a difficult course, than to take a course with a popular lecturer. However, it is possible for lecturers to be popular and challenging for students, or to be unpopular and easy. Insler et al. look across combinations of popularity and difficulty, noting that:

Faculty that are both well-liked and considered very difficult (top 25% of each RMP rating distribution) are likely what most faculty aspire to be: challenging and demanding, but generating devotion and enthusiasm based on superior teaching. Such unicorns are extremely rare in the data, making up just under 2% of the overall faculty, and just over 1% of the total number of observations. We find no evidence that these faculty impact sequential learning, although this may be related to the small sample size. The other three groupings (“High Difficulty, Low Overall”; “Low Difficulty, Low Overall”; “Low Difficulty, High Overall”), however, are all statistically different from the excluded group of faculty who are in the middle of the distribution on at least one of these dimensions.

Which faculty are most associated with sequential learning? Those faculty who bundle together characteristics of high difficulty and low likability. One interpretation of this finding is that poor teachers are considered to be difficult by students because of lack of clarity in lecture and course structure, and this experience pushes students to invest in studying on their own to learn the material, resulting in deeper learning which is carried through to the next semester. However, we suspect that the issue is more likely to be explained by faculty who demand a lot of their students, forcing students to exert costly effort. This learning by effort leads to deeper learning and sequential success, but also engenders animosity towards the professor, resulting in high difficulty and low overall ratings...

...faculty who are considered very easy and poor overall do notable damage to sequential learning. These faculty are in fact likely to be poor teachers, who perhaps minimize effort themselves, resulting in an easy and poor course.

Reflecting on my own teaching, I like to think I challenge students, and I know that my teaching evaluations are among the top in my School. I don't think I'm a unicorn though? Still:

 Anyway, I'm hopefully not among the worse group, where Insler et al. note that:

...the grouping that appears to be most problematic are those with high overall ratings and low difficulty ratings. These faculty severely harm sequential learning, and more perniciously, are likely to be faculty who are praised by administrators for achieving high engagement from students (expressed through high overall opinions by students). These faculty - which we dub “a Seemingly Conscientious and Hardworking Mentor, an Obtuse and Perfunctory Educator” or Schmopes... - are deeply problematic because they damage student learning and are elevated within the university system as role model faculty.

Why 'Schmopes'? Footnote 7 in the paper notes that:

Inspired by discussions with our students who justified the value of this type of professor by commenting that they give hope to students, one rather perspicacious student retorted with “hope, schmope.”

The paper goes further, to show that the 'hard skills' channel has persistent effects over time, but the 'soft standards' channel decays over time. So, that may be some good news. If students encounter a 'soft standards' lecturer early in their studies, a 'hard skills' lecturer can get them back on track. However:

...it is easy to imagine that soft standards may influence student choices in other dimensions, such as choice of major. Encountering a lenient instructor early on in a college career may influence a student to choose a major that is not well suited to the student’s comparative advantage...

Finally, Insler et al. show that the effects of soft standards are higher for extroverted students (presumably because introverted students can better overcome the soft standards effect through self-study), and for students who are more 'feeling' than 'thinking' on the thinking/feeling scale (presumably because those who are more 'feeling' than 'thinking' are influenced more greatly by their lecturers).

Overall, there is a lot of depth to this paper, and the lack of selection bias means that this is a paper that we should pay a lot of attention to. By that, I mean firstly that lecturers and university administrators should pay attention to it, because tough lecturers who set high standards may not be well liked by students, but are much better for their learning. We should therefore interpret student evaluations of teaching very carefully (if at all, as they have a lot of problems, such as those I have outlined here and here). Second, students should pay attention too. Taking easy courses, or courses with lecturers who are known to give easy grades, may actually make future courses more difficult to achieve well in because of the lower learning in the earlier courses (and this is on top of any negative signal that easy courses provide to future employers).

The final word: Beware the schmopes!

[HT: Marginal Revolution]

The cost-benefit principle and investing in AI technology

In the first week of my ECONS102 class, we talk about rational behaviour as a useful starting point (or optimum) for considering decision-making. There are three characteristics of rational behaviour:

1. People (and firms, and government) have objectives (goals);
2. People choose the correct way to achieve their objectives (i.e. they compare benefits and costs, and they learn from past mistakes); and
3. People respond to the incentives they face (people’s behaviour may change if the costs and/or benefits of the alternatives change).

The second of those characteristics of rational behaviour relates to the cost-benefit principle: A rational decision-maker will take an action if, and only if, the incremental (extra) benefits from taking the action are at least as great as the incremental (extra) costs. Now, not everyone is rational in all decisions and at all times. But, deviations from rationality invariably make decision-makers worse off. Consider the cost-benefit principle. If a decision-maker takes an action where the incremental costs exceed the incremental benefits, they are making themselves worse off (because they would be better off by doing nothing at all instead).

So, that brings me to this article in The Conversation today by Evan Shellshear and Len Coote (both University of Queensland), which looks at the decision about whether to invest in some artificial intelligence (AI) application. The example they use is a farmer investing in an AI application that offers cost savings through reducing crop inputs. Shellshear and Coote offer some clear advice, deriving directly from the cost-benefit principle:

Invest if the extra profit is greater than the “opportunity cost” – the benefit you can gain from spending your money another way, or by not spending the money.

And Shellshear and Coote offer a simple flowchart for decision-makers to follow:

Peddlers of AI are great at selling the technological benefits of AI. However, that doesn't mean that AI should be adopted everywhere and for all purposes. A rational decision-maker must recognise that AI must pass the cost-benefit test. Applying Shellshear and Coote's simple flowchart, if you can't quantify the benefits (or gains), or you can't quantify the costs, or having quantified the benefits and costs you find that the costs outweigh the benefits, you should not adopt the AI. As they conclude:

Using an economic framework of worth, rather than an engineering claim of possibility, is the first step to make better decisions. Doing so reduces the prospect of another AI winter, and increases the chance of real gains contributing to a more prosperous and sustainable world.

Devon Zuegel on inflation

One of the interesting (or disturbing, maybe) things about teaching university economics over the last decade or more in a country like New Zealand is that our students have never experienced high inflation. Never. For incoming first-year students next year, mostly born in 2003 or 2004, the inflation rate has only been above five percent (on an annual basis) in two quarters (see the data here) in their lifetimes - in the September quarter of 2008 (5.1 percent) and in the June quarter of 2011 (5.3 percent). The latest data (for the September quarter of 2021) had annual inflation at 4.9 percent. For much of the last two decades (and more) the inflation rate has been under two percent per year. Individual prices may change, but the general price level overall barely moves at all (and it is the change in the general price level that defines inflation).

So, when we teach first-year students about the costs of inflation, we are talking to an audience about something they have never really experienced and can't recognise in the world around them. It's almost like this:

They have no idea, and so menu costs, shoe leather costs, and other costs of inflation are difficult for students to connect with their own experience. We can't use examples from the New Zealand context to illustrate these costs, because New Zealand really hasn't faced those costs in appreciable terms for decades. And jumping straight to examples of periods of hyperinflation (like Weimar Germany, Zimbabwe in the 2000s, or Venezuela more recently), makes inflation seem even more other-worldly to students.

So, I found it interesting to read this perspective from Devon Zuegel on inflation from earlier this week, drawing in part on their experience in Argentina. There is lots of good bits to Zuegel's post, and I encourage you to read it, but I want to focus on two bits on the costs of inflation. First, here (emphasis is theirs):

In turn, systemic uncertainty reduces people's willingness to make long-term investments. (For example, high-inflation Argentina has almost no mortgage industry.) This drawback in investment isn't predicted by the model that my friend had in mind, because the model doesn't take into account the uncertainty that inflation causes or its psychological impacts.

Inflation makes people uncertain about the future value (and purchasing power) of money. It makes lenders less likely to lend money (because they can't be sure about the value of what they will get paid back). You might argue that lenders can build higher expectations about inflation rates into the nominal interest rate they charge to borrowers. This involves recognising the importance of the Fisher equation: real interest rate ≈ nominal interest rate - inflation rate. If the inflation rate is higher, lenders will need to charge a higher nominal interest rate in order to receive the same (target) real interest rate. However, high inflation is also inherently more unstable, and therefore less predictable, so it is difficult for lenders to determine what interest rate they should charge. If they are risk averse, they could err on the side of caution and charge a higher interest rate to protect themselves, but that will deter borrowers and reduce investment in the economy.

Second, Devon notes that inflation creates real harms for everyday people in the economy (emphasis is theirs):

Wage adjustments are not just slow but also uneven across the economy. For example:

• A waiter might do okay when their tips are a percentage of prices, because as long as the restaurant's owner updates prices consistently (which they're very motivated to do), the tip-based wages will adjust accordingly. Their base pay will not adjust so quickly though, because the restaurant owner is unlikely to updated wages as fast as menu prices.

• A retiree with a pension is in a really tough spot, because pensions are rarely (if ever?) indexed to inflation, so over time their income gets eroded to zero.

• Architects are in a tough spot too. They usually charge large lump fees, so if they give you a quote at the beginning of the year and then inflation hits, the real value of the quote they gave you went way down by the time you actually pay for their services.

As a general rule, inflation disproportionately harms people who can't easily adjust their income upwards. Fixed contracts and fixed incomes are especially vulnerable. Wages also don't automatically adjust—you generally need to advocate for yourself to get a raise—so if you lack negotiating skills in a high-inflation economy, you're at a significant disadvantage.

Anyone who can't adjust their income easily is going to be harmed by high inflation. As Zuegel notes, this extends from those on fixed incomes (retirees) if their pensions do not automatically adjust, to people with annual salary reviews (since it takes a year before their salary is revised to account for changes in the cost of living) to contractors. However, it may even extend to day labourers, if their employers are not able to adjust prices frequently and pass on higher wages as a result. Having wages kept low while prices increase also benefits employers, but makes workers want to change jobs in order to lock in a higher wage rate. This 'employee churn' imposes costs on the employer as well as the economy overall.

Inflation imposes costs on people. New Zealanders may have forgotten about these costs (or never experienced them if they are young), but that doesn't make those costs any less real.

[HT: Marginal Revolution]

Visual imagination and learning economics

I've always tried to limit the extent of maths in first-year business economics (in my ECONS101 class). It really isn't necessary for general business or management students to understand the maths underlying the economic models, if they can understand the intuition and apply the models. And burying the lead in maths simply makes it more difficult for many students to connect to the important concepts. The economics majors can always pick up the mathematics in their intermediate classes, having worked through the intuition earlier.

On the other hand, I've made extensive use of diagrams, especially in my ECONS102 class. I've even devised new diagrams to illustrate particular models or concepts (like the models of media bias described in this working paper, currently under review at a good journal). While I've been quite careful about reducing problems of students' maths anxiety, I've been less concerned about students' problems with visual representations of economic models (in the form of diagrams or graphs).

However, this 2020 article by David Fielding, Viktoria Kahui, and Dennis Wesselbaum (all University of Otago), published in the journal New Zealand Economic Papers (sorry I don't see an ungated version online), has made me pause for thought. Fielding et al. look at the relationship between visual imagination (measured using the Vividness of Visual Imagery Questionnaire) and performance in an undergraduate macroeconomics class examination. Specifically, they look at performance on a mathematical exam question, many graphical questions, and other (non-mathematical, non-graphical) questions, and disaggregate the results for male and female students. They find that:

...male students with poor visual imagination perform significantly worse on examination questions that are predominantly graphical, and the size of this effect is greater for those students who performed poorly on a different, mathematical question. This suggests that poor performance in graphical questions is a product of poor visual imagination and weakness in mathematics... We find similar (but only marginally significant) results for male performance on nonmathematical, non-graphical questions. One explanation for our results is that visual imagination enhances students’ ability to manipulate economics diagrams and hence the students’ examination performance...

We do not find any significant results for women, but this may just reflect (i) the fact that women make up a relatively small proportion of our sample and (ii) the fact that as in previous studies, there are fewer women with very poor visual imagination, so the range of variation for women is smaller than it is for men...

One way of interpreting these results is that students who are good mathematically and have good visual imagination have a real advantage in studying economics. Having one or the other (good mathematical skills or good visual imagination) can make up for a deficit in the other. However, students with poor mathematical skills and poor visual imagination are going to really struggle.

Having read this paper, none of that strikes me as surprising. However, it does raise an important question: how can we engage the (male) students who struggle both with maths and diagrams, so that they can understand the key economics concepts? Some economic models are easier than others to explain narratively, without recourse to either mathematics or diagrams. However, it takes a lot more effort on the part of the teacher (or, to be fair, the textbook writer) to use a narrative format well. There are a number of popular economics books that do this well (see some of my past book reviews), but integrating those into a course would require a lot of skill. However, a textbook like The Economics of Public Issues, which I use in my ECONS102 class, is a good option.

I'm hopeful that Fielding et al. will give us some further guidance on how we can improve teaching for those students. We also need further corroboration that visual imagination is only a problem for male students, and not for female students. Fielding et al. only had 28 female students in their sample, so it was quite underpowered statistically to tell us much. Anyway, this article sounds like it is the beginning of a bigger research project:

In the next stage of the research project, the authors intend to use a larger sample of students, and to measure a number of different determinants of examination performance, as well as administering the VVIQ. This will facilitate a broader study of the ways in which the vividness of visual imagery interacts with other factors in determining students’ ability to understand and explain concepts in economics.

I look forward to reading that future research.