Flipped classrooms work well for top students in economics

Earlier this year I blogged about two AER Papers and Proceedings papers that compared online only, blended (e.g. flipped) and traditional classes. Here's what I said then:

These two papers give quite complementary findings. The Swoboda and Feiler paper found a significant positive effect of their blended learning approach (compared with face-to-face), while Alpert et al. find no significant effect of blended learning...

Now I would really like to know what the distributional effects of blended learning are. My expectation is that it will probably work well for keen, motivated students, who are almost certain to watch the lectures before class. These are the students who currently read the textbook, and additional resources (like the lecturer's blog!). These students will likely benefit most from the change in approach, and gain significantly from the interactive learning in class, which is what the blended learning approach is designed to facilitate. The less-motivated students, who may watch some of the videos before class but often don't, will not benefit as much, or may actually be made worse off by the switch to blended learning.

Which brings me to this new paper by Rita Balaban and Donna Gilleskie (both UNC Chapel Hill), and Uyen Tran (University of Chicago), published in the latest issue of the Journal of Economic Education (sorry I don't see an ungated version anywhere). In the paper the authors compare a flipped classroom model (where students watch lectures online before attending class where more active learning approaches, such as problem-based learning, are employed) with a traditional lecture-based model. There were nearly 400 students in each semester. Unfortunately, the research design is not clean because the two models were employed in different semesters. The authors demonstrate that there are no observable differences between the students, but I would also be concerned that the lecturer (who is one of the co-authors) knew that the study was being undertaken during the second semester (when the blended learning approach was used), and put in greater effort (a genuine concern for any single-blinded trial).

Notwithstanding my concern about the single-blinded nature of the approach, the results are interesting and very positive for the blended learning approach:

The values of average percent correct among all common questions for the traditional and flipped classroom formats suggest that the course redesign led to a (statistically significant) 6.9-percentage-point improvement in student performance (i.e., a difference-in-means result).

Once they control for student characteristics, the results remain similar, with an overall increase of about one-half of a standard deviation. This is quite a large impact. When they disaggregate the results by question type, they find:

Our results indicate that the flipped classroom does not differently impact performance on knowledge questions (objective 1), which require memory, recognition, and recall. We find that the flipped format significantly improves performance on comprehension questions (objective 2) by one-quarter of a standard deviation...

With regard to performance on application questions (objective 3), the flipped classroom boosts performance by 0.74 standard deviations on average...

On analysis questions (objective 4), we find gains of 0.47 standard deviations. The ability to analyze involves differentiating, organizing, and attributing.

So, as one might expect, the main impacts are on students being better able to apply their learning (it's worth noting that there were only three questions in the exam in the knowledge domain, and only three questions in the analysis domain).

Finally, the authors looked at the results by performance level (using quantile regression). They find:

that the flipped classroom had slightly different effects on students (depending on their position within the performance distribution) for different types of questions. Overall, students in the top 25 percent of the distribution appear to benefit more from the flipped format than those below the 75th percentile, although all students benefit substantially.

Which brings me back to my initial disquiet at the flipped classroom model. This paper has done little to dissuade me that it benefits the motivated top-performing students and could make things worse for the unmotivated marginal student. I wouldn't necessarily take this study as representative of the average university student. The average exam result was 80.7 percent, which struck me as rather high until I looked at the average SAT scores for the sample, and found that the average student was in about the 90th percentile on the SAT. So, even the students in the bottom of this class are relatively good students in the overall scheme of things. To add to that, the lecture attendance was over 90 percent on average. So not only are these mostly top-achieving students, they are well-motivated top-achieving students. Exactly the students I would expect to benefit from the flipped classroom. I guess I'm still waiting for the research that will convince me that the flipped classroom model will have positive outcomes for the marginal (or even the median) student that I teach.