There are plenty of online resources available to teachers, to help improve their teaching, or to reduce the up-front costs of developing lessons. At university level, most textbook publishers provide a plethora of additional teaching resources including pre-prepared PowerPoint slides, learning activities, readings, case studies, test banks, and more. There are also websites where teachers can share their lesson plans and other materials with other teachers.
For the most part, I have resisted the temptation to use all of the resources available to me as a teacher, and I suspect I am not alone. There are several reasons for this. First, in relation to textbook resources, I feel like they are a thinly-disguised attempt to lock teachers into using the same textbook in perpetuity. They create a switching cost for lecturers wanting to change textbooks, since the lecturer would then need to develop new resources to replace those from the old textbook. And the textbook resources are really not that good, anyway. The PowerPoint slides simply parrot details which students can read for themselves from the textbook, reducing any value-add that students may get from lectures. The test banks and case studies tend to be error-laden, having largely been written by poorly-paid graduate students, and subjected to minimal, if any, quality checks (and I say this as a former poorly-paid graduate student who updated the instructor resources for the Gans, King and Mankiw Principles of Economics textbook some years ago). The resources provided by teachers for other teachers are much better quality, but tend to be somewhat idiosyncratic and require tailoring to the class they will be used in, and that tailoring is not without cost to the lecturer. For my part, I have adapted a number of in-class exercises and experiments for my classes, as well as developing my own.
Finally, it is worth wondering, does using pre-prepared resources affect student learning at all? Pre-prepared off-the-shelf lessons or activities may save the teacher some time, and that time might be used to improve the pedagogy that is employed in the classroom, including by improving the quality of discussion or examples or applications that they use (or, off-the-shelf lessons may free up the teacher to engage in more relaxing non-teaching activities, leaving them refreshed and engaged and better able to attend to students' learning needs). On the other hand, off-the-shelf lessons or activities may not be well adapted to the learning needs of a particular classroom, and therefore not improve student learning at all.
The value of off-the-shelf lessons is tested in this 2018 article by Kirabo Jackson and Alexey Makarin (both Northwestern University), published in the American Economic Journal: Economic Policy (ungated earlier version here). Jackson and Makarin conducted a randomised experiment using Mathalicious resources for high school mathematics teachers. As they explain:
Under our experiment, teachers were randomly assigned to one of three treatment conditions. In the license-only condition, we informed teachers that these lessons were high quality and that they had free access to them. To promote lesson adoption, some teachers were randomly assigned to the full treatment condition in which teachers received email reminders to use the lessons and were invited to an online social media group focused on lesson implementation (in addition to the license-only offerings). Finally, teachers randomly assigned to the control condition continued business-as-usual.
The sample frame was large:
All three Virginia districts agreed to participate: Chesterfield, Henrico, and Hanover. Across all grade levels, 59,186 students were enrolled in 62 Chesterfield public schools, 50,569 students were enrolled in 82 Henrico public schools, and 18,264 students were enrolled in 26 Hanover public schools in the 2013–2014 school year (NCES). All grades 6 through 8 math teachers in these districts were part of the study.
The sample size was also large, covering 363 teachers and 27,613 students in the 2013-14 school year. The key outcome variable was standardised maths test scores. The random assignment of teachers to the treatment conditions (after some non-randomised teachers were excluded from the sample) allows causal estimates of the effects of the experimental treatments to be estimated. Jackson and Makarin find that:
Students of teachers in the license-only and the full treatment groups experienced a 0.06σ and 0.09σ test score increase relative to those in the control condition, respectively. The full treatment effect is statistically significant at the 1 percent level, and has a similarly sized effect as that of moving from an average teacher to one at the eightieth percentile of quality, or reducing class size by 15 percent...
Those are sizeable effects, but not all teachers improved outcomes for their students equally, as:
...the benefits of online lesson use are the largest for the least effective teachers (as measured by teacher/classroom value added). We theorize that this is due largely to lesson quality improvements being largest for weaker teachers. We also find suggestive evidence that lesson provision had larger effects for first-year teachers, implying that the off-the-shelf lessons may have provided some time savings for these teachers.
So far, so good. So, what was it about the off-the-shelf lessons that led to these improvements, especially for the least effective and first-year teachers? Exploring the mechanisms, Jackson and Makarin find that:
Students from the full treatment group are 0.175σ (p-value < 0.05) more likely to agree that their math teacher promotes deep understanding. Also, consistent with off-the-shelf lessons freeing up teacher time to exert more effort in complementary teaching tasks, student agreement with statements indicating that their math teacher spends more one-on-one time with them is 0.144σ higher in the full treatment condition than in the control condition (p-value < 0.05). While the results are consistent with the time savings hypothesis, we cannot rule out that the increases in one-on-one time are due to changes in classroom practices due to using the new lessons...
Interestingly, the off-the-shelf lessons appear to have given teachers either greater time, or greater confidence, to focus on the real-life applications of maths, as well as more time to spend one-on-one with students.
Finally, Jackson and Makarin present some back-of-the-envelope calculations on the cost-effectiveness of the lessons, showing that there may be:
...a benefit-cost ratio of 939. Because of the low marginal cost of the intervention, it is extraordinarily cost effective.
Pre-prepared off-the-shelf lessons are not for every teacher (despite the evidence it provides, this study hasn't convinced me that I need to use them). But clearly, they have the potential for a positive impact on student learning. They may also have proven useful in the transition to online teaching and learning enforced by the pandemic, and in the current era of teaching both online and in-person. However, that would require a different evaluation.
No comments:
Post a Comment