Some interesting points on US transport infrastructure

Transport infrastructure has been big news in the US this year, with the Biden administration's plans for US$312 billion in new spending. So, I was interested to read this 2020 NBER Working Paper (ungated version here) by Gilles Duranton (University of Pennsylvania), Geetika Nagpal, and Matthew Turner (both Brown University). Duranton et al. first document the quantity and quality of the interstate highway network, bridges, public transit buses, and subways. Second, they look at how (primarily public) spending on transport has evolved over time, and the implied unit costs of different transport modes. Third, they summarise the literature on the effects of infrastructure on economic growth and congestion.

First, a digression. I also found these two factoids really interesting:

The New York subway system carries about 71% of all subway riders and about 31% of all public transit riders in the whole country...

...a single combination truck causes about as much damage to a highway as about 2.1 commute hours of automobile traffic...

Having travelled on several public transit systems in the US, I had no idea how dominant the New York subway system was overall. And, there is definitely a case for trucks to be paying more towards the cost of maintaining roads (in the US, and here in New Zealand).

Anyway, getting back to the central theme of the paper, the overall conclusion is that:

Massive investments in transportation infrastructure seem to draw support from across the political spectrum. These policies are often motivated by claims that our current infrastructure is crumbling or that such investments will spur economic growth. The available evidence does not support these claims.

In other words, they find little evidence that infrastructure is crumbling, and while there is evidence that transport infrastructure affects the location of economic activity, there is little evidence that it affects economic growth overall. Infrastructure investment also doesn't reduce congestion, because additional lane miles of roadway encourage more driving. So, I guess there is good news if you are a local planner in a peri-urban area trying to steal a bit of economic activity away from the urban area, but no such good news for a national planner.

In all the arguments over transport infrastructure in New Zealand, it may be that the current government's reluctance to invest in more road infrastructure will be neutral in terms of its impact on economic growth. That doesn't mean that we should be investing in crazy cycle bridge projects.

[HT: Marginal Revolution, last year]

Meta-analytic results may provide some support for flipping the classroom

The research article that I referenced in my post yesterday pointed me to this 2019 article by David van Alten, Chris Phielix, Jeroen Janssen, and Liesbeth Kester (all Utrecht University), published in the journal Education Research Review (ungated version here). They report on a meta-analysis on the effect of 'flipping the classroom' on student outcomes. Flipping the classroom is an approach where, as van Alten et al. note:

...students study instructional material before class (e.g., by watching online lectures) and apply the learning material during class.

Under this definition, 'instructional material' might include online instructional videos, computer-based tasks, or simply reading material. This broader definition is problematic if what you really want to know is whether new technologies are having a positive impact on learning. It turns out the definition isn't a problem in this paper, because 110 of the 115 interventions (reported across 114 research articles) included in the review used online videos as the before-class component, so there is likely to be little bias arising from the inclusion of other modes. All included studies were published in the period from 1990 to 2016.

Van Alten et al. focused on three outcomes: (1) assessed learning; (2) perceived learning; and (3) student satisfaction. The difference between assessed and perceived learning is that assessed learning is based on the quantitative results of an assessment exercise (e.g. an exam, or a standardised test), while perceived learning is based on how much the students thought they learned. They found that:

The average effect size for assessed learning outcomes (g=0.36) was found to be significant... The average effect on perceived learning outcomes was nearly identical (g=0.36), but not significant (p=.13)... For student satisfaction, a trivial and non-significant effect size (g=0.05) was found.

It is interesting that students were not satisfied with the flipped classroom approach on the whole. The perceived learning results were pretty noisy, but the large positive effect on assessed learning outcomes doesn't really accord with my understanding of the literature to date (see the links at the end of this post). Van Alten et al. dug a bit deeper into their results by looking at the characteristics of the studies, and in terms of assessed learning outcomes they found that:

...studies that shortened the classroom time of the flipped condition had a significantly lower (p=.027) average effect than studies in which the classroom time in both conditions was equal (with a difference of g=−0.26, while accounting for all the other variables in the model). In addition, adding quizzes in the flipped condition also showed a significant (p=.044) difference with studies where quizzes were not added or already applied in the traditional condition (with a difference of g=0.19, while accounting for all the other variables in the model).

In other words, the positive outcomes from flipping the classroom arise when the before-class tasks are additional to the classroom learning time. It is encouraging the students to do additional work, and that might be driving the effects, rather than the flipping of the classroom on its own. That also flies in the face of one of the rationales for flipping the classroom, that it reduces the time commitment required of teaching staff. If the same number of contact hours are required, along with the preparation of videos as well, then that requires more time.

There are some problems with the meta-analysis, and one arises from the nature of the included studies. Only about a quarter of the studies employed randomisation, and so the other studies lack a bit of experimental rigour. It would have been interesting to see what the results look like when limited only to the randomised studies (although, to be fair, randomisation wasn't statistically significant in the meta-regression, so perhaps it wouldn't make much difference).

The other issue is heterogeneity, which is an issue I have raised several times before (e.g. see here or here). Blended learning works well for highly-engaged high-achieving students, but it could well be net negative for less-engaged low-achieving students. It would be nice to see one of these meta-analyses or systematic reviews of the literature engage with a really important issue. On a related note, it would be interesting to see the meta-regression results of these analyses stratified by the average SAT scores of the samples (or similar measure of student ability). It could be that the studies that find the most positive effects tend to be undertaken in universities where the student body is (on average) higher ability and more engaged. Until we see some analysis along these lines, my critique of blended learning and flipped classrooms (and online approaches more generally) remains live.

Read more:

• Online vs. blended vs. traditional classes
• Flipped classrooms work well for top students in economics
• Flipped classrooms are still best only for the top students
• Online classes lower student grades and completion
• Online classes may also make better students worse off
• Meta-analytic results support some positive effects of using videos on student learning
• New review evidence on recorded lectures in mathematics and student achievement

New review evidence on recorded lectures in mathematics and student achievement

Last year I posted about new meta-analytic research that showed some positive effects of using video recordings as part of teaching. However, regular readers of this blog will know that I am sceptical of the positive effects of video lectures and blended learning. Mostly, that is because I believe that it neatly splits the class into two parts: (1) a group of students that watches (most or all of) the recordings, and whose learning may improve; and (2) a group of students that doesn't watch the recordings, and quickly falls behind. When you combine both groups together in a single analysis, you get the average effect, which will be small and may be positive or negative, depending on the mix of the two groups of students in the class being studied.

In a new article published in the Mathematics Education Research Journal (ungated, and there is a non-technical summary of the research available on The Conversation), Euan Lindsey and Tanya Evans (both University of Auckland) review the literature on the use of lecture capture in mathematics. Lecture capture as Lindsey and Evans define it is fairly broad as a concept, including:

...“synchronized audio and visual recordings of live lectures, which students can download to view at their own leisure”... but also includes... online lectures— recordings of the lecture content done by the lecturer (not necessarily during the live lecture) which have a one‐to‐one correspondence with a live lecture from the course. Sometimes those online lectures can be split into many shorter videos—on average, 7 min long...

So, this definition covers what I was doing in my class before the coronavirus pandemic (recording live in-class lectures), as well as how most lecturers approached lecturing during the pandemic lockdowns (recording whole lectures, or short videos). Lindsey and Evans base their review on 16 studies published over the period from 2010 to 2020, and look at student perceptions of lecture capture (LC), its effect on attendance, and its effect on learning. On the first point (student perceptions), they find that:

Students see immense value in LC because of the flexibility it provides, this being the most obvious and widely accepted perception of LC... Because the lecture information can be accessed at any time, LC availability is perceived to facilitate a better study/work/life balance and provide equitable access to content for students who have other commitments...

So, it's all good from the students' perspective. In relation to attendance though:

Of the 16 relevant studies this review identified, 10 did not consider attendance, none reported a positive impact, three sources reported insignificant/no change, and four reported a negative impact on attendance...

In all studies that explicitly reported a drop in attendance, the total attendance reductions were in the range of 23–30%.

In that case, lecturers' concerns that lecture capture reduces class attendance are borne out (at least, in some studies), although that hasn't been my experience (until this year!). Finally, the most important result (in my opinion) relates to the effect on learning:

Of the 16 studies investigated in this review, six did not report on attainment, one concluded the neutral impact of LC (with the exception of students who were not following up on their intentions to watch LC after missing a lecture—this practice was strongly associated with poor grades...), one concluded a positive relationship, and nine concluded a negative relationship between LC and attainment...

Of the eight studies that reported a negative impact on attainment, all found that regular substitution of live lecture attendance with LC was associated with lower achievement...

Despite the inconsistency number of cited studies with negative findings (was it eight, or nine?), the overall conclusion is clear - lecture capture likely reduces student achievement. There are a couple of important caveats though:

...these studies did see benefits to groups of students who used LC supplementarily... but once a student used LC as their primary learning tool, they underperformed compared to their peers attending lectures...

Some studies found that students who were already performing poorly were the most affected by the introduction of LC...

The second point accords well with my impression of the impact of lecture capture, especially when, as Lindsey and Evans note:

Broadly, there appear to be two types of LC users: students who supplement attendance and students who substitute attendance. The latter group seems to be getting disproportionally larger, considering the trend over the decade.

I suspect that those two groups would correlate pretty strongly with the two groups of students I mentioned at the beginning of this post. However, as can be inferred from this post, it no doubt depends on how you go about the lectures, and what works in-class is unlikely to work as well online. And that suggests that the live capture approach is best only when used to supplement attendance.

Read more:

• Online vs. blended vs. traditional classes
• Flipped classrooms work well for top students in economics
• Flipped classrooms are still best only for the top students
• Online classes lower student grades and completion
• Online classes may also make better students worse off
• Meta-analytic results support some positive effects of using videos on student learning

Book review: Prisoner's Dilemma

I just finished reading William Poundstone's 1992 book, Prisoner's Dilemma. The subtitle is John von Neumann, Game Theory, and the Puzzle of the Bomb. Von Neumann is recognised (along with Oskar Morgenstern) as the founders of game theory, as outlined in their 1944 book Theory of Games and Economic Behavior (although, as Poundstone's book notes, some of the key ideas date back to work by the French mathematician Emile Borel in the 1920s, some years before von Neumann began his work on game theory).

I was expecting the book to be mostly a biography of von Neumann, but that is only the beginning. The book ranges widely over the origins and early applications of game theory, from zero-sum games to non-zero-sum games, and of course the prisoners' dilemma itself (along with 'chicken' games and some other games of note). A fairly large chunk of the book is actually devoted to the goings-on at the RAND Corporation, the 'original' think tank, set up by Douglas Aviation and initially funded by the US Air Force. As Poundstone notes:

The RAND Corporation's original purpose was to perform strategic studies on intercontinental nuclear war. RAND hired many of the scientists leaving wartime defense work, and took on as consultants an even larger orbit of stellar thinkers.

And later:

Many of the social scientists hired or consulted were economists who had fallen under the spell of game theory. It was at RAND rather than in the groves of academia that game theory was nurtured in the years after von Neumann and Morgenstern's book. In the late 1940s and early 1950s, few of the biggest names of game theory and allied fields didn't work for RAND< either full-time or as consultants. 

Game theory, as the study of strategic interactions, is of course central to RAND's original purpose. And, as it turns out, so was John von Neumann. The biography sections of the book are very good at outlining von Neumann's gifts and accomplishments, of which there were many. This quote highlights how highly Poundstone clearly thought of him:

Von Neumann's brilliant accomplishments can only hint at the academic and public perception of him. To many of his associates at Princeton, the Pentagon, the RAND Corporation, and elsewhere, von Neumann was a living legend whose reputation was spread by innumerable anecdotes, some of which appeared in the popular press.

Poundstone shares many of these anecdotes. If the passage above sounds little hyperbolic to you, then you'll probably find many others of the like, such as:

The prisoner's dilemma has become one of the premier philosophical and scientific issues of our time. It is tied to our very survival.

It's an important game, yes, but I'm not sure I would go quite that far. Even with much more recent memories of the Cold War (remember, this book was written in 1992).

Other than the occasional hyperbole, the biographical section of the book seemed to me to lack flow, jumping around chronologically. That makes it more difficult than it should be to keep the sequence of the story in mind. I realise that Poundstone was doing double-duty in trying to write both about von Neumann and about developments in game theory, but I felt that I could have understood a bit more with some more logical sequencing. Other readers might not be so easily distracted.

The latter parts of the book go into some depth on prisoners' dilemma games and other social dilemmas, and the lack of appropriate solutions to these dilemmas (after all, if there were simple solutions, there would be no dilemma). The explanations here are good, but sometimes they can be a little circular. And the conclusion will not satisfy everyone:

The ability to see "opponents" as fellow beings frequently transforms a nominal prisoner's dilemma into a much less troublesome game. The only satisfying solution to the prisoner's dilemma is to avoid prisoner's dilemmas.

Despite these issues, I did enjoy reading this book. I've enjoyed reading a number of biographies in recent years, and I can add this one to the list. And I enjoy game theory, so fleshing out a bit more detail on its origins was helpful to me. If you have similar preferences, then you will also get a lot out of this book.