The promise of a personalised, AI-augmented textbook, and beyond

In the 1980s, the educational psychologist Benjamin Bloom introduced the 'two-sigma problem' - that students who were tutored one-on-one using a mastery approach performed on average two standard-deviations (two-sigma) better than students educated in a more 'traditional' classroom setting. That research is often taken as a benchmark for how good an educational intervention might be (relative to a traditional classroom baseline). The problem, of course, is that one-on-one tutoring is not scalable. It simply isn't feasible for every student to have their own personal tutor. Until now.

Generative AI makes it possible for every student to have a personalised tutor, available 24/7 to assist with their learning. As I noted in yesterday's post though, it becomes crucial how that AI tutor is set up, as it needs to ensure that students engage meaningfully in a way that promotes their own learning, rather than simply being a tool to 'cognitively offload' difficult learning tasks.

One promising approach is to create customised generative AI tools, that are specifically designed to act as tutors or coaches, rather than simple 'answer-bots'. This new working paper by the LearnLM team at Google (and a long list of co-authors) provides one example. They describe an 'AI-augmented textbook', which they call the 'Learn Your Way' experience, which:

...provides the learner with a personalized and engaging learning experience, while also allowing them to choose from different modalities in order to enhance understanding.

Basically, this initially involves taking some source material, which in their case is a textbook, but could just as easily be lecture slides, transcripts, and related materials from a class. It then personalises those materials to the interests of the students, adapting the examples and exercises to fit a context that the students find more engaging. For example, if the student is an avid football fan, they might see examples drawn from football. And if the student is into Labubu toys, they might see examples based on that.

The working paper describes the approach, reports a pedagogical evaluation performed by experts, and finally reports on a randomised controlled trial (RCT) evaluating the impact of the approach on student learning. The experts rated the Learn Your Way experience across a range of criteria, and the results were highly positive. The only criterion where scores were notably low was for visual illustrations. That accords with my experience so far with AI tutors, which are not good at drawing economics graphs, in particular (and is an ongoing source of some frustration!).

The RCT involved sixty high-school students in Chicago area schools, who studied this chapter on brain development of adolescents. Half of the students were assigned to Learn Your Way, and half to a standard digital PDF reader. As the LearnLM Team et al. explain:

Participants then used the assigned tool to study the material. Learning time was set to a minimum of 20 minutes and a maximum of 40 minutes. After this time, each participant had 15 minutes to complete the Immediate Assessment via a Qualtrics link.

They then did a further assessment three days later (a 'Retention Assessment'). In terms of the impact of Learn Your Way:

The students who used Learn Your Way received higher scores than those who used the Digital Reader, in both the immediate (p = 0.03) and retention (p = 0.03) assessments.

The difference in test outcomes was 77 percent vs. 68 percent in the Immediate Assessment, and 78 percent vs. 67 percent in the Retention Assessment. So, the AI-augmented textbook increased student learning and retention by about 10 percentage points in both immediate learning and in the short term (three days). Of course, this was just a single study with a relatively small sample size of 60 students in a single setting, but it does offer some promise for the approach.

I really like this idea of dynamically adjusting content to suit students' interests, which is a topic I have published on before. However, using generative AI in this way allows material to be customised for every student, creating a far more personalised approach to learning than any teacher could offer. I doubt that even one-on-one tutoring could match the level of customisation that generative AI could offer.

This paper has gotten me thinking about the possibilities for personalised learning. Over the years, I have seen graduate students with specific interests left disappointed by what we are able to offer in terms of empirical papers. For example, I can recall students highly interested in economic history, the economics of education, and health economics in recent years. Generative AI offers the opportunity to provide a much more tailored education to students who have specific interests.

This year, I'll be teaching a graduate paper for the first time in about a decade. My aim is to allow students to tailor that paper to their interests, by embarking on a series of conversations about research papers based on their interests. The direction that leads will be almost entirely up to the student (although with some guidance from me, where needed). Students might adopt a narrow focus on a particular research method, a particular research question, or a particular field or sub-field of economics. Assisted by a custom generative AI tool, they can read and discuss papers, try out replication packages, and/or develop their own ideas. Their only limits will be how much time they want to put into it. Of course, some students will require more direction than others, but that is what our in-class discussion time will be for.

I am excited by the prospects of this approach, and while it will be a radical change to how our graduate papers have been taught in the past, it might offer a window to the future. And best of all, I have received the blessing of my Head of School to go ahead with this as a pilot project that might be an exemplar for wider rollout across other papers. Anyway, I look forward to sharing more on that later (as I will turn it into a research project, of course!).

The ultimate question is whether we can use generative AI in a way that moves us closer to Bloom’s two-sigma benefit of one-on-one tutoring. The trick will be designing it so that students still do the cognitive work. My hope (and, it seems, the LearnLM team’s) is that personalisation increases students' engagement with learning rather than replacing it. If it works, this approach could be both effective and scalable in a way that human one-on-one tutoring simply can’t match.

[HT: Marginal Revolution, for the AI-augmented textbook paper]

Neuroscientific insights into learning and pedagogy, especially in the age of generative AI

In May last year, my university's Centre for Tertiary Teaching and Learning organised a seminar by Barbara Oakley of Oakland University, with the grand title 'The Science of Learning'. It was a fascinating seminar about the neuroscience of learning, and in my mind, it justified several of my teaching and learning practices, such as continuing to have lectures, to emphasise students' learning basic knowledge in economics, and retrieval practice and spaced repetition as learning tools.

Now, I've finally read the associated working paper by Oakley and co-authors (apparently forthcoming as a book chapter), and I've been able to pull out further insights that I want to share here. The core of their argument is in the Introduction to the paper. First:

Emerging research on learning and memory reveals that relying heavily on external aids can hinder deep understanding. Equally problematic, however, are the pedagogical approaches used in tandem with reliance on external aids—that is, constructivist, often coupled with student-centered approaches where the student is expected to discover the insights to be learned... The familiar platitude advises teachers to be a guide on the side rather than a sage on the stage, but this oversimplifies reality: explicit teaching—clear, structured explanations and thoughtfully guided practice—is often essential to make progress in difficult subjects. Sometimes the sage on the stage is invaluable.

I have resisted the urge to move away from lectures as a pedagogical tool, although I'd like to think that my lectures are more than simply information dissemination. I actively incorporate opportunities for students to have their first attempts at integrating and applying the economic concepts and models they are learning - the first step in an explicit retrieval practice approach. Oakley et al. note the importance of both components, because:

...mastering culturally important academic subjects—such as reading, mathematics, or science (biologically secondary knowledge)—generally requires deliberate instruction... Our brains simply aren’t wired to effortlessly internalize this kind of secondary knowledge—in other words, formally taught academic skills and content—without deliberate practice and repeated retrieval.

The paper goes into some detail about the neuroscience underlying this approach, but again it is summarised in the Introduction:

At the heart of effective learning are our brain's dual memory systems: one for explicit facts and concepts we consciously recall (declarative memory), and another for skills and routines that become second nature (procedural memory). Building genuine expertise often involves moving knowledge from the declarative system to the procedural system—practicing a fact or skill until it embeds deeply in the subconscious circuits that support intuition and fluent thinking...

Internalized networks form mental structures called schemata, (the plural of “schema”) which organize knowledge and facilitate complex thinking... Schemata gradually develop through active engagement and practice, with each recall strengthening these mental frameworks. Metaphors can enrich schemata by linking unfamiliar concepts to familiar experiences... However, excessive reliance on external memory aids can prevent this process. Constantly looking things up instead of internalizing them results in shallow schemata, limiting deep understanding and cross-domain thinking.

This last point, about the shallowness of learning when students rely on 'looking things up' instead of relying on their own memory of key facts (and concepts and models, in the case of economics), leads explicitly to worries about learning in the context of generative AI. When students rely on external aids (known as 'cognitive offloading'), then learning becomes shallow, because:

...deep learning is a matter of training the brain as much as informing the brain. If we neglect that training by continually outsourcing, we risk shallow competence.

Even worse, there is a feedback loop embedded in learning, which exacerbates the negative effects of cognitive offloading:

Without internally stored knowledge, our brain's natural learning mechanisms remain largely unused. Every effective learning technique—whether retrieval practice, spaced repetition, or deliberate practice—works precisely because it engages this prediction-error system. When we outsource memory to devices rather than building internal knowledge, we're not just changing where information is stored; we're bypassing the very neural mechanisms that evolved to help us learn.

In short, internalized knowledge creates the mental frameworks our brains need to spot mistakes quickly and learn from them effectively. These error signals do double-duty: they not only help us correct mistakes but also train our attention toward what's important in different contexts, helping build the schemata we need for quick thinking. Each prediction error, each moment of surprise, thus becomes an opportunity for cognitive growth—but only if our minds are equipped with clear expectations formed through practice and memorization...

Learning works through making mistakes, recognising those mistakes, and adapting to reduce those mistakes in future. Ironically, this is analogous to how generative AI models are trained (through 'reinforcement learning'). When students offload learning tasks to generative AI, they don't get an opportunity to develop the underlying internalised knowledge that allows them to recognise mistakes and learn from them. Thus, it is important for significant components of student learning to happen without resorting to generative AI (or other tools that allow students to cognitively offload tasks).

Now, in order to encourage learning, teachers must provide students with the opportunity to make, and learn from, mistakes. Oakley et al. note that:

...cognitive scientists refer to challenges that feel difficult in the moment but facilitate deeper, lasting understanding as “desirable difficulties... Unlike deliberate practice, which systematically targets specific skills through structured feedback, desirable difficulties leverage cognitive struggle to deepen comprehension and enhance retention...

Learning is not supposed to be easy. It is supposed to require effort. This is a point that I have made in many discussions with students. When they find a paper relatively easy, it is likely that they aren't learning much. And tools that make learning easier can hinder, rather than help, the learning process. In this context, generative AI becomes potentially problematic for learning for some (but not all) students. Oakley et al. note that:

Individuals with well-developed internal schemas—often those educated before AI became ubiquitous—can use these tools effectively. Their solid knowledge base allows them to evaluate AI output critically, refine prompts, integrate suggestions meaningfully, and detect inaccuracies. For these users, AI acts as a cognitive amplifier, extending their capabilities.

In contrast, learners still building foundational knowledge face a significant risk: mistaking AI fluency for their own. Without a robust internal framework for comparison, they may readily accept plausible-sounding output without realizing what’s missing or incorrect. This bypasses the mental effort—retrieval, error detection, integration—that neuroscience shows is essential for forming lasting memory engrams and flexible schemas. The result is a false sense of understanding: the learner feels accomplished, but the underlying cognitive work hasn’t been done.

The group that benefits from AI as a complement for studying is not just those who were educated before AI became ubiquitous, but also those who learn in an environment where generative AI is explicitly available as a complement to learning (rather than a substitute). To a large extent, it depends on how generative AI is used as a learning tool. Oakley et al. do provide some good examples (and I have linked to some in past blog posts). I'd also like to think the AI tutors I have created for my ECONS101 and ECONS102 students assist with, rather than hamper, learning (and I have some empirical evidence that seems to support this, which I have already promised to blog about in the future).

Oakley et al. conclude that:

Effective education should balance the use of external tools with opportunities for students to internalize key knowledge and develop rich, interconnected schemata. This balance ensures that technology enhances learning rather than creating dependence and cognitive weakness.

Finally, they provide some evidence-based strategies for enhancing learning (bolding is mine):

• Embrace desirable difficulty—within limits: Encourage learners to generate answers and grapple with problems before turning to help... In classroom practice, this means carefully calibrating when to provide guidance—not immediately offering solutions, but also not leaving students floundering with tasks far beyond their current capabilities...
• Assign foundational knowledge for memorization and practice: Rather than viewing factual knowledge as rote trivia, recognize it as the glue for higher-level thinking...
• Use procedural training to build intuition: Allocate class time for practicing skills without external aids. For instance, mental math exercises, handwriting notes, reciting important passages or proofs from memory, and so on. Such practices, once considered old-fashioned, actually cultivate the procedural fluency that frees the mind for deeper insight...
• Intentionally integrate technology as a supplement, not a substitute: When using AI tutors or search tools, structure their use so that the student remains cognitively active...
• Promote internal knowledge structures: Help students build robust mental frameworks by ensuring connections happen inside their brains, not just on paper... guide students to identify relationships between concepts through active questioning ("How does this principle relate to what we learned last week?") and guided reflection...
• Educate about metacognition and the illusion of knowledge: Help students recognize that knowing where to find information is fundamentally different from truly knowing it. Information that exists "out there" doesn't automatically translate to knowledge we can access and apply when needed.

I really like those strategies as a prescription for learning. However, I am understandably biased, because many of the things I currently do in my day-to-day teaching practice are encompassed within (or similar to) those suggested strategies. I'll work on making 'guided reflection' a little more interactive in my classes this year, as I have traditionally made the links explicit for the students, rather than inviting them to make those links for themselves. We have been getting our ECONS101 students to reflect more on learning, and we'll be revising that activity (which happens in the first tutorial) this year to embrace more of a focus on metacognition.

Learning is something that happens (often) in the brain. It should be no surprise that neuroscience has some insights to share on learning, and what that means for pedagogical practice. Oakley et al. take aim at some of the big names in educational theory (including Bloom, Dewey, Piaget, and Vygotsky), so I expect that their work is not going to be accepted by everyone. However, I personally found a lot to vindicate my pedagogical approach, which has developed over two decades of observational and experimental practice. I also learned that there are neuroscientific foundations for many aspects of my approach. And, I learned that there are things I can do to potentially further improve student learning in my classes.

This week in research #112

Here's what caught my eye in research over the past week:

• Mati et al. find that the Russia-Ukraine war resulted in an immediate 21 percent reduction in the daily growth rate of the Euro-Ruble exchange rate, and that the steady-state effect translates to a 26 percent reduction in growth
• Masuhara and Hosoya review the COVID-19-related performance of OECD countries as well as Singapore and Taiwan in terms of deaths, vaccination status, production, consumption, and mobility from the early part of the pandemic to the end of 2022, and conclude that Norway was the most successful in terms of balancing deaths, production, and consumption
• Neprash, McGlave, and Nikpay (with ungated earlier version here) quantify the effects of ransomware attacks on hospital operations and patient outcomes, finding that attacks decrease hospital volume by 17-24 percent during the initial attack week, with recovery occurring within 3 weeks, and that among patients already admitted to the hospital when a ransomware attack begins, in-hospital mortality increases by 34-38 percent
• Tsivanidis (with ungated earlier version here) studies the world’s largest Bus Rapid Transit system in Bogotá, Colombia, and finds that low-cost "feeder" bus systems that complement mass rapid transit by providing last-mile connections to terminals yield high returns, but that welfare gains would have been about 36 percent larger under a more accommodative zoning policy
• Janssen finds that the 2023 Bud Light boycott led to a large drop in Bud Light volume (34-37 percent), partial switching into other beer, and a net decline in total ethanol purchases of roughly 5.5-7.5 percent of pre-boycott intake
• Krishnatri and Vellakkal (with ungated earlier version here) find that alcohol prohibition in Bihar, India, led to significant increases in caloric, protein, and fat intake from healthy food sources, as well as a decline in fat intake from unhealthy food sources
• Geruso and Spears (open access) document the worldwide fall in birth rates, and the unlikely prospects of a reversal to higher fertility in the future

Americans' beliefs about trade, and why compensation matters

Do people understand trade policy? Or rather, do they understand trade policy the way that economists understand it? Given current debates in the US and elsewhere, it would be fair to question people's (or politicians') understanding of trade policy, and to consider what it is about trade that generates negative reactions. After all, the aggregate benefits of free trade are one of the things about which economists most agree.

Last year, Stefanie Stantcheva won the John Bates Clark Medal (which is awarded annually to the American economist under age 40 who has made the most significant contributions to the field). Stantcheva's medal-winning work included three main strands, one of which was the use of "innovative surveys and experiments to measure what people know". One of the papers from that strand of research is this 2022 NBER Working Paper (revised in 2023), which describes Americans' understanding of trade and trade policy and importantly, it answers the question of why people support trade (or not).

The paper reports results from three large-scale surveys in the US run between 2019 and 2023, with a total sample size of nearly 4000. The surveys also included experiments that primed respondents to think about trade from particular angles. Overall, Stantcheva is interested in teasing out the factors that affect Americans' support for trade policies. Essentially, she tests the mechanisms that are described in Boxes I-V in Figure 2 from the paper:

Box I picks up views on whether trade lowers prices and increases variety for consumers. Box II picks up the threats from increasing trade to workers in import-competing sectors. Those two boxes together constitute self-interest as an effect on people's views on trade policy. Their views might also be affected by broader social and economic concerns, such as trade's efficiency effects (Box III), its distribution impacts (Box IV), and patriotism, partisanship, or geopolitical concerns (Box V).

Before we turn to the specific results on the mechanisms, it is worth considering Americans' overall views on trade first. Stantcheva reports that:

Most respondents (63%) are supportive of more free trade and decreasing trade restrictions in general... Only 36% believe that import restrictions are the best way to help U.S. workers.

Nevertheless, there is support for more targeted trade restrictions. 40% of respondents believe the US should restrict food imports to ensure food security. 54% think the US should protect their “infant” industries. 78% support protection of key consumer products, namely food items and cars. 50% believe the US should restrict trade in key sectors, such as oil and machinery...

And general knowledge about trade policy is not too bad, as:

...almost 80% of respondents know what an import tariff is, but just around half know what an import quota is. Two-thirds of respondents appear to understand the basic price effects of tariffs and export taxes, i.e., that an import tariff on imported goods will likely raise the price of that good and that an export tax will increase the price of the taxed good abroad. The final question... considers a scenario in which the US can produce a good (“cars”) at a lower cost than the foreign country. Respondents are asked whether, under some circumstances, it would still make sense to import cars from abroad. 68% of respondents agree that it could make sense. This suggests that respondents either understand the concept of comparative advantage or have in mind some model of love-for-variety or quality differential.

So far, so good. How do Americans perceive the impacts of trade? Figure 9 Panel A reports perceptions related to the self-interest motivation (Boxes I and II from the figure above):

From the bottom of that figure, it is clear that a majority of Americans believe that they are better off from trade, but a substantial minority (39%) believe that they are worse off. Still focusing on the self-interest motivations (Boxes I and II), Stantcheva finds that:

In general, a respondent’s (objective) negative exposure to trade through their sector, occupation, or local labor market is significantly positively correlated with a feeling that trade has made them worse off and that it has negatively affected their job. People exposed to trade through their job also feel worse off as consumers and are less likely to believe that trade has reduced the prices of goods they buy, perhaps because they feel that their purchasing power is lower than it would otherwise be. Furthermore, college-educated respondents are significantly less likely to feel negatively impacted in their role as consumers and workers.

Notice those results are mostly consistent with the figure above. What about consumer gains through reduced prices on imported products? Stantcheva reports that:

...the belief that prices decrease from trade is not significantly related to either support for trade or redistribution. Consistent with this lack of correlation, the experiment priming people to think of their benefits as consumers (precisely, the prices and variety of goods they purchase) does not move their support for trade either.

So, in terms of self-interest, Americans' support for trade is more negative when they are negatively affected as workers, but is not more positive when they are positively affected as consumers. In my ECONS102 class, we talk about the tension between the gains from trade and loss aversion. Every trade involves gaining something, in exchange for giving something up. However, quasi-rational decision-makers are affected much more by losses than equivalent gains (what we call loss aversion). So, loss aversion might mean that many profitable trades are not undertaken, because the decision-makers prefer to keep what they have, rather than giving it up for something that may be objectively worth more. In the case of Stantcheva's survey respondents, the workers who are negatively impacted experience a loss, which would be weighed much more heavily than the gain that a consumer receives.

An alternative explanation is salience. Job losses are very visible and impactful on the people who lose their jobs and those around them. Consumers' gains in terms of lower prices and increased variety, on the other hand, are not really as visible - many people wouldn't even notice them, unless they were pointed out to them. So even if people weren’t loss averse, attention would still be drawn disproportionately to the negative impacts of trade, rather than the positive. Taken altogether, Stantcheva's results here are not surprising.

What about the broader social and economic concerns, and their impact on views about trade? In terms of efficiency effects (Box III), Stantcheva reports that:

Respondents are generally optimistic about these effects. For instance, 61% of respondents think that international trade increases competition among firms in the US, 69% that it fosters innovation, and 62% that it generates more GDP growth.

Moreover:

...efficiency gains from trade are significantly associated with more support for free trade... This relation can be seen in the correlations and the experimental effects: the Efficiency treatment significantly improves support for free trade.

And interestingly:

Respondents who believe that trade can improve innovation, competitiveness, and GDP are more supportive of redistribution policy to help those who do not benefit from these efficiency gains.

Turning to distributional impacts (Box IV), Stantcheva reports that:

Overall, respondents know that trade can have adverse distributional consequences through the labor market. Just around half of all respondents believe that trade has, on balance, helped US workers. 79% of people think that trade is the reason for “unemployment in some sectors and the decline of some industries in the U.S..” More respondents (63%) believe that high-skilled workers could easily change their work sector if their jobs were destroyed by trade than that low-skilled workers could switch sectors (37%)...

Consequently, around two-thirds of respondents think that trade is a major reason for the “rise in inequality” in the US. Notably, despite being aware of the potential adverse distributional consequences of trade, a majority (62%) of respondents believe that, in principle, trade could make everyone better off because it is possible to “compensate those who lose from it through appropriate policies.”

It is interesting that so many people believe in the compensation principle (although I bet that few of them would know that term for it). And it turns out that belief in the compensation principle is really important, as:

...the strongest predictor of support for free trade is the belief that, in principle, losers can be compensated... free trade. As long as respondents believe that adverse consequences from trade on some groups can be dampened by redistributive policy, they are likely to support more free trade, even if they believe that there are adverse distributional consequences. The perceived distributional impacts of trade also substantially matter for support for compensatory redistribution. Respondents who believe that trade hurts low-income and low-skilled workers and that it fosters inequality support redistribution much more.

Finally, in terms of patriotism, partisanship, or geopolitical concerns (Box V), Stantcheva reports that:

...those who worry about geopolitical ramifications from trade restrictions, i.e., retaliatory responses, are more likely to support policies to compensate losers from trade rather than support outright trade restrictions. Patriotism is significantly correlated with support for trade restrictions in many industries and to protect U.S. workers, as well as with lower support for compensatory transfers...

Stantcheva draws a number of conclusions from her results, including:

First, respondents perceive gains from trade as consumers to be vague and unclear but perceive potential losses as workers to be concentrated and salient. Actual and perceived exposure to trade through the labor market is significantly associated with policy views...

Second, people’s policy views on trade do not only reflect self-interest. Respondents also care about trade’s distributional and efficiency impacts on others and the US economy...

Third, respondents’ experience, as measured by their exposure to trade through their sector, occupation, and local labor market, shapes their policy views directly (through self-interest) and indirectly by influencing their understanding and reasoning about the broader efficiency and distributional impacts of trade.

Overall, I take away from this paper that Americans have more correct views about trade than I suspected. Their support for trade is not determined simply by self-interest, but is more nuanced. However, negative impacts weigh far more heavily for those who are negatively impacted than the weight attached to positive impacts for those who are positively impacted. That may relate to loss aversion, and to the more concentrated nature of negative impacts compared with more diffuse positive impacts. That asymmetry also explains why a majority have positive views of trade (since fewer people will have been negatively impacted on the whole). The most surprising aspect to me, though, was the views on the compensation principle. Those results provide a clear policy prescription. To get more people on board with trade, making compensatory policy more explicit and salient may help to ensure that there is greater support for trade. On the other hand, politicians who want to exploit the negative views on trade might benefit from obscuring any such compensatory policies. Unfortunately, there are too many who are willing to do just that.

[HT: Marginal Revolution, last year]