Online vs. blended vs. traditional classes

I've been a bit resistant to the increased trend towards 'flipped classrooms' and blended learning. For the most part, my resistance was due to a lack of robust research demonstrating that flipped classrooms were more effective (or even equally effective) when compared with 'traditional' face-to-face classes. My classes aren't the traditional chalk-and-talk in any case - my students know that they will get a lot of exercises to do in class, especially in ECON110. So, I remain to be convinced that flipping my classroom would benefit my students (moreover, I worry that there would be distributional consequences - more on that at the end of this post).

Anyway, two papers in the new AER Papers and Proceedings issue caught my eye on this topic. The first paper (ungated version here), by Aaron Swoboda and Lauren Feiler (both Carleton College) compares a blended learning approach with a traditional face-to-face class in introductory microeconomics. They describe the two approaches they employ as follows:

The blended sections required students to read textbook chapters, watch videos of lecture material created by the authors, and answer basic comprehension questions online before coming to a class session on a topic. During the class session, the professors began by answering questions about the out-of-class materials and giving mini-lectures targeted to troublesome items, but students spent much of their time engaged in group problem-solving, simulations, and discussion activities. After the session, students were assigned a more challenging set of online questions covering that day’s topic and preparatory materials for the next topic. Online homework assignments were completed using Sapling Learning and provided instant grading and feedback. The control courses primarily used “chalk-and-talk” and traditional written homework assignments.

In other words, this paper doesn't simply compare the flipped classroom with a traditional approach. Both approaches had the same number of contact hours between teachers and students. So, the blended approach essentially involved the students engaging in more learning hours, because in additional to classroom time they were watching video material. On top of that, the blended learning treatment group had online questions with feedback. So, it's not possible to isolate whether any impact on learning arises from the flipped classroom approach, the additional materials, or some combination of the two. Having said that, Swoboda and Feiler find:

The mean student scored 20.5 points on the TUCE [Test of Understanding of College Economics] at the end of the term compared to 15.0 on the pretest. The difference in TUCE scores is significantly different from zero at conventional levels.

...students in the traditional courses improved by roughly four points out of 30, while students in the blended courses improved by 6 points.

So with the blended learning approach students improved their economics knowledge by more. The additional increase was about half a standard deviation in the initial scores, which is reasonably substantial. However, it isn't possible to isolate whether this was because of the blended approach itself, or the additional learning hours devoted to the paper, or the online tests that provided ongoing feedback. Another issue with the paper is that students could choose whether they were in the blended learning class or the face-to-face class. You'd expect students who thought they would do better in the blended learning class to choose that option (though that is by no means certain - students might well convince themselves that blended learning is a good thing, but then not follow through on watching the videos, thereby reducing their learning opportunities).

Which brings me to the second paper (sorry I don't see an ungated version anywhere), by William Alpert, Kenneth Couch, and Oskar Harmon (all University of Connecticut). They take what I consider to be a more robust approach (also in introductory microeconomics), where students are randomly assigned to a treatment (or control), and they compare face-to-face with both a blended learning approach and a fully online approach. They explain:

The experimental design randomly assigned students to one of three delivery modalities: classroom instruction, blended instruction with some online content and reduced instructor contact, and purely online instruction. The traditional section met weekly for two 75-minute sessions, alternating between a lecture and discussion period. The blended section met weekly with the instructor for a 75-minute discussion period. As a substitute for the lecture period, students were given access to online lecture materials. The online section had class discussion in an online asynchronous forum, and for the lecture period students were given access to the same online lecture materials as students in the blended section. The online materials were developed using best practice standards from the Higher Ed Program Rubric for online education as described on the website of the Quality Matters (2014) organization. For the three arms of the experiment, lectures, discussions, and other instructional content were prepared and delivered by the same instructor.

So, in this case the blended learning approach had reduced contact hours, so any observed effect would likely not be because of enforced additional learning hours for students. Their outcome measure is performance on a common final examination. They find:

There, students are still found to score 4.2 (t-statistic = 2.68) points lower in the online section than the face- to-face variant. The sign of the impact of participating in the blended course is negative but the parameters are not significantly different than zero at conventional levels across the three columns.

In other words, the students in the online only treatment did demonstrably worse than students in the face-to-face class (by about half a letter grade). Students in the blended learning treatment performed similarly to those in the face-to-face class.

Alpert et al. also look at differential attrition between the three classes. They find:

From the point students received permission to enroll to course completion, potential participation in the face-to-face section declined from 175 to 120 students (30 percent). For the blended course section, the decline was from 172 randomized students to 110 completers (36 percent). The largest decline was observed for the online arm where 172 students were assigned to the course and 93 completed (46 percent).

In other words, students were most likely to drop out of the online class, and a greater proportion of students dropped out of the blended learning class than the face-to-face class. Once they control for this attrition, the negative effects of the online only class are even larger (about a full letter grade), but there is still no significant difference between the face-to-face and blended learning class.

These two papers give quite complementary findings. The Swoboda and Feiler paper found a significant positive effect of their blended learning approach (compared with face-to-face), while Alpert et al. find no significant effect of blended learning. However, since Swoboda and Feiler had additional learning hours (i.e. the same number of classroom hours plus video watching beforehand), it is plausible that their significant effects were due to the enforced additional learning hours. The higher attrition in the blended learning class for Alpert et al. is a bit of a concern though.

Now I would really like to know what the distributional effects of blended learning are. My expectation is that it will probably work well for keen, motivated students, who are almost certain to watch the lectures before class. These are the students who currently read the textbook, and additional resources (like the lecturer's blog!). These students will likely benefit most from the change in approach, and gain significantly from the interactive learning in class, which is what the blended learning approach is designed to facilitate. The less-motivated students, who may watch some of the videos before class but often don't, will not benefit as much, or may actually be made worse off by the switch to blended learning. So I'd expect to see a similar mean (or median) result in the class (or higher if the number of classroom hours remained the same), but a wider distribution around that mean.

At the least, some more research work is needed to convince me to move to a blended learning approach (having said that, it is essentially the approach we use in teaching ECON100 in summer school!).