Saturday, 13 November 2021

Live streamed video lectures and student achievement

As I have noted many times on this blog (see the links at the bottom of this post), I strongly believe that online learning has heterogeneous effects on student learning and achievement. As I said in my most recent post on this topic:

Students who are motivated and engaged and/or have a high level of 'self-regulation' perform at least as well in online learning as they do in a traditional face-to-face setting, and sometime perform better. Students who lack motivation, are disengaged, and/or have a low level of self-regulation flounder in online learning, and perform much worse.

The problem with a lot of the research that tries to establish the effects of online or blended or hybrid learning on student achievement is that it doesn't distinguish between the effects on students at the top of the ability distribution and the effects on students at the bottom of the ability distribution. So, it doesn't really tell us a lot about how a change in teaching practice will affect the whole distribution of student achievement - it might only tell us what will happen at the middle of the distribution, which often isn't very helpful.

One recent exception is this recent article by Paula Cacault (EPFL), Christian Hildebrand (University of St. Gallen), Jeremy Laurent-Lucchetti, and Michele Pellizzari (both University of Geneva), published in the Journal of the European Economic Association (open access, but just in case there is an ungated earlier version here). Cacault et al. investigated the effect of live streamed video lectures on student attendance behaviour and their achievement across eight compulsory management, statistics, and economics courses at the University of Geneva. Students in each class (nearly 1500 students in total) were given access to a live stream of lectures in some weeks, but not others. As they explain:

Based on the enrolment lists of each course from the e-learning platform, we first randomly assigned students to three groups. A first group of students (15% of all students) never had access to the streaming service and we label this group the Never-access. Another 15% of the students were given access to the service in all the weeks of the term and we label this group the Always-access. The remaining 70% of students were given access to streaming only some weeks at random and we label this group the Sometimes-access. Every week, a varying share of students in this group was given access. In the Spring semester 2017, we randomly assigned weekly access to 50% of the sometimes-access group. In the Fall semester 2017, we decided to vary this share between 20% and 80%...

Physical attendance in the classroom was always possible and students could freely decide to go to class in person even in the weeks when they had access to streaming...

Cacault et al. then look at student performance in the final examinations for questions drawn from lectures where they had live streaming access against those from lectures where they didn't. First, they find that the take-up of live streaming is low:

Using information from the server of the streaming platform, we can identify the students who actually accessed the service. On average, only about 5% of the students (i.e., including those who had no access) used the service at least once in each week...

Combining information on assignment and usage we also construct measures of take-up, that is, the share of students with access who logged into the platform. This share is on average around 10%–11%, ranging across weeks from a minimum of 6.7% to a maximum of 13%.

So, very few students made use of the live streaming. Cacault et al. find that there are no differences in take-up between students at different 'ability levels' (defined by their academic performance in high school). They also find only a small effect on classroom attendance (which was measured from photos taken of the classroom):

...for every 100 students who are offered lectures via live streaming about 8 of them do not show up in class.

So, the effect on learning should be pretty small, given that few students make use of the live streaming option. Indeed, Cacault et al. find that:

Results indicate that on average there is no detectable effect of the experimental assignment, nor of actual usage of the streaming platform. However, once we look at effects by ability groups, we uncover large heterogeneity, with a sizeable negative ITT [intent-to-treat] effect on the low-ability students and a positive effect on the high-ability students...

The magnitudes of these estimates are sizeable. For students in the bottom 20% of the ability distribution, having access to the streaming platform (regardless of whether one uses it or not) lowers the share of correct answers by approximately 2 percentage points over an average of about 55%. The positive effect at the top of the ability distribution is even larger and in the order of about 2.5 percentage points. The ATT [average effect of treatment on the treated] estimates are very large: around -18 percentage points for the low-ability and about +25 percentage points for the high-ability students.

The ATT results are the effects on the students who actually participated in the live streaming, rather than just being given the option to do so, which explains the much larger effect.

Now, these results relate to live streaming, so we should be cautious about over-interpreting them as applying to all online learning. However, overall these results accord with those I have discussed on the blog earlier - online learning options tend to make more motivated, higher ability students better off, and less motivated, lower ability students worse off. Cacault et al. suggest a possible mechanism that explains this difference in effects:

Consider a situation in which the streaming technology is not available. Assume that in normal times most students attend lectures in person, but when the cost of going to class is too high, the good students tend to stay at home and study on their own. This happens because very good students can read the material in the book and understand most of it easily, even without the professors presenting and explaining it, whereas students of lower ability would have a harder time learning in autonomy and prefer attending.

Introducing the streaming technology in this context would allow all students to use it when the cost of going to class happens to be high. The good students replace own study with streaming, which improves learning and leads to the observed positive effect on exam performance. The low-ability students watch the streamed lectures instead of going to class, which is a more effective (but also more costly) mode of learning, and eventually perform worse. The students in the middle range of the ability distribution, substitute streaming to attendance for small shocks and streaming to no-attendance for larger shocks. Hence, average effect on grades tend to be close to zero, as we see in our data...

It's possible that is the mechanism at play in their context, where live streaming is available but the lectures are not recording, but I'm unsure how that translates to other contexts, particularly where the lectures are recorded. I do think we need more research on this area, but in particular, we need research on how to mitigate the negative impacts of online learning on students at the bottom of the ability and motivation distribution. If universities are serious about an enduring shift to online learning, this is a problem that needs urgent attention.

Read more:

Read more:

at

No comments:

Post a Comment