This week in research #70

Here's what caught my eye in research over the past week:

• Burton (with ungated earlier version here) finds that smoking bans in bars in the US result in a 1-drink-per-month (5 percent) increase in alcohol consumption and no economically meaningful effects on smoking
• Perron and Hu (open access) find that, in the NHL, each additional locally born player completing a full season is associated with an increase in home game attendance by approximately 12,000 spectators and $4.8 million in additional revenue
• Dalal and Raju develop a theoretical model of the illegal drugs market, and show that, under risk aversion, increasing punishment costs (i.e., severity) is more effective than increased enforcement (i.e., certainty) and demand reduction is more effective than interdiction
• Faerber-Ovaska et al. (open access) test the accuracy of ChatGPT’s answers to multiple-choice and short essay questions from a widely used economics textbook, and find that ChatGPT scored a high D in multiple-choice questions and a low A in essay questions (I feel like the world has already moved on from this though, as I noted in this post)
• Cheah and Qi develop a theoretical model of the effect of broadcast revenue on competitive balance in sports, and their simulations with the model show that broadcast revenue allows teams with smaller home fan bases to narrow their performance gap against stronger teams (it must depend on revenue sharing rules though, surely?)
• Reimão et al. find that expert judges favour the first dish tasted in a blind test in the Great British Bakeoff (and other English-speaking versions of the show)

Book review: The Economists' Hour

Once upon a time, economists were backroom advisers, crunching numbers and developing theories, but rarely in the limelight and certainly not the central actors in political decision-making. However, as Binyamin Appelbaum outlines in his 2019 book The Economists' Hour, that all changed in the late 1960s. The title of the book references the period from 1969 to 2008, a period of unprecedented policy change (in the US and in other countries), and a period where economists had the ear of the key governmental decision-makers. As Appelbaum notes in the introduction to the book:

This book is a biography of the revolution. Some leading figures are relatively well-known, like Milton Friedman, who had a greater influence on American life than any other economist of his era, and Arthur Laffer, who sketched a curve on a cocktail napkin in 974 that helped to make tax cuts a staple of Republican economic policy. Others may be less familiar, like Walter Oi, a blind economist who dictated to his wife and assistants some of the calculations that persuaded Nixon to end military conscription; Alfred Kahn, who deregulated air travel and rejoiced in the cramped and crowded cabins on commercial flights as the proof of his success; and Thomas Schelling, a game theorist who persuaded the Kennedy administration to install a hotline to the Kremlin - and who figured out a way to put a dollar value on human life.

That paragraph neatly sums up the book. Each chapter is devoted to one particular aspect of policy that changed as a result of the influence of economists. Before reading the book, I had no idea of the important role that economists played in ending military conscription in favour of volunteer armed forces. I was, however, well aware of economists' role in deregulation of airlines, as well as deregulation of interstate trucking in the US, and of financial markets, and the development of monetary policy and the independence of central banks. Some particular parts are surprising, such as the relatively late impact of economists on antitrust regulation (only from the 1960s). However, like other areas covered in the book, economists drove a radical change in policy in that space:

The rise of economics transformed the role of antitrust law in American life. During the second half of the twentieth century, economists gradually persuaded the federal judiciary - and, to a lesser extent, the Justice Department - to set aside the original goals of antitrust law and to substitute the single objective of providing goods and services to consumers at the lowest possible prices.

Appelbaum describes in some detail the contributions of the key players in each case, including economists as well as political decision-makers and their other advisors. Some figures, such as Friedman and various US presidents, make many appearances, and often similar ideas come up across multiple chapters. This repetition might turn some readers off. However, it is difficult to see how the book might have been constituted in any other way, because the thread of each case would easily be lost if all the material were presented chronologically.

The book is incredibly well researched, with nearly 90 pages of footnotes. As is sometimes the case in books like this, particularly for readers that are familiar with the general story, the footnotes present details that are of more interest than the text itself. For example, consider this footnote on Milton Friedman, and real and nominal interest rates:

This is another example of a battle Friedman won so completely that his victory is largely forgotten. He insisted during the 1950s and the 1960s that there was a significant difference between real and nominal rates. Conventional economists disagreed... Today the distinction between real and nominal rates is universally understood to be significant.

Indeed, we teach the difference between real and nominal interest rates (and the relationship between them known as the 'Fisher equation'), but Friedman's battle to have this recognised is largely forgotten.

I really enjoyed that Appelbaum didn't limit the book to only considering the US case. Economists had important roles in reshaping the economies in Chile and Taiwan, and in deregulating markets across the developed world. Appelbaum writes a lot about deregulation in Iceland. If there is one missing element to the book, it would be the relative lack of attention paid to economists' roles in the transitional economies of former Communist countries such as Poland, Hungary, of the Soviet Union. However, New Zealand does make an appearance a couple of times, including this bit:

In December 1989, New Zealand passed a law making price stability the sole responsibility of its central bank, sweeping away a 1964 law that, characteristically for its time, had instructed the central bank to pursue a laundry list of goals including economic growth, employment, social welfare, and trade promotion. The man picked to lead New Zealand's experiment was an economist named Don Brash, who ran one of the nation's largest banks and then one of its largest trade groups, the Kiwifruit Authority...

Appelbaum is careful not to provide an overly rosy view of the role of economists, and the impacts of these changes. Indeed, in the introduction he warns that:

This book is also a reckoning of the consequences...

Markets make it easier for people to get what they want when they want different things, a virtue that is particularly important in pluralistic societies which value diversity and freedom of choice. And economists have used markets to provide elegant solutions to salient problems, like closing a hole in the ozone layer and increasing the supply of kidneys available for transplant.

But the market revolution went too far. In the United States and in other developed nations, it has come at the expense of economic equality, of the health of liberal democracy, and of future generations.

And almost as quickly as it began, perhaps, the economists' hour was over:

The Economists' Hour did not survive the Great Recession. Perhaps it ended at 3:00 p.m. on Monday, October 13, 2008, when the chief executives of America's nine largest banks were escorted into a gilded room at the Treasury. The government had tried to support the banks by purchasing bonds in the open market, but the market had collapsed, so the government decided to save the financial system by taking ownership stakes in the largest financial firms.

Or perhaps it was one of a dozen other moments during the financial crisis; it doesn't really matter which. In the depths of the Great Recession, only the most foolhardy purists continued to insist that markets should be left to their own devices...

However, it would be fair to note that economists continue to have a strong influence in policy, in other countries if not in the US (as the current furore over tariffs attests).

I really enjoyed this book, and if you have an interest in understanding how economics (and economists) came to have such an important influence on policy, I am sure that you will enjoy it too. Highly recommended!

Supply curves slope upwards... Nigerian cocoa edition

The New Zealand Herald reported last month:

Booming cocoa prices are stirring interest in turning Nigeria into a bigger player in the sector, with hopes of challenging top producers Ivory Coast and Ghana, where crops have been ravaged by climate change and disease.

Nigeria has struggled to diversify its oil-dependent economy but investors have taken another look at cocoa beans after global prices soared to a record US$12,000 ($21,000) per tonne in December.

“The farmers have never had it so good,” Patrick Adebola, executive director at the Cocoa Research Institute of Nigeria, told AFP.

More than a dozen local firms have expressed interest in investing in or expanding their production this year, while the British Government’s development finance arm recently poured US$40.5 million into Nigerian agribusiness company Johnvents.

When the price of a good increases, sellers become willing and able to supply more of the good. In general, sellers want to increase their profits. When the price of a good increases, it becomes more profitable to sell it, and so sellers want to sell more of it. [*] This intuition is embedded in the supply curve, as shown in the diagram below. When the price of cocoa is P0, sellers want to sell Q0 tonnes of cocoa. But when the price increases to P1, sellers want to sell Q1 tonnes of cocoa.

What might have caused the increase in the global price of cocoa? The New Zealand Herald article explains that:

Ivory Coast is by far the world’s top grower, producing more than two million tonnes of cocoa beans in 2023, followed by Ghana at 650,000 tonnes.

But the two countries had poor harvests last year as crops were hit by bad weather and disease, causing a supply shortage that sent global prices to all-time highs.

I'll refrain from drawing the global market for cocoa, but suffice to say that the high global price of cocoa is attracting Nigerian farmers to produce more, illustrating that the supply curve for Nigerian cocoa is upward sloping.

*****

[*] There are at least two other explanations for why the supply curve is upward sloping, and both relate to opportunity costs. First, as sellers produce more of the good, the factors of production (raw materials, labour, capital, etc.) become more scarce and so become more expensive. Also, less relevant (and so more costly) inputs begin to be used to produce the good. So, the opportunity costs of production increase, and as the sellers produce more the minimum price they are willing to accept increases more because their marginal cost is increasing. Second, when the price is low the opportunity cost of not selling is low, but as the price rises the opportunity cost of not selling rises, encouraging the sellers to offer more for sale. In other words, as the price increases, the sellers do less of not selling (yes, that is a double negative, and it was intentional). As the price increases, the sellers want to sell more.

Do economists act like the self-interested decision-makers from our models, and if so, why?

Economics models typically assume that decision-makers are self-interested, trying to maximise their own 'economic rent'. Does exposure to these models, and the assumption of self-interest, lead people who have studied economics to make more self-interested decisions? Or, are people who make more self-interested decisions more likely to study economics (perhaps because it accords with their already-established world view)?

These are questions that many studies have tried to grapple with (and which I have written about before, most recently in this 2023 post). What is needed is a good systematic review of the literature. We don't have that, but this 2019 article by Simon Hellmich (Bielefeld University), published in the journal The American Economist (sorry, I don't see an ungated version online), provides a review of the literature (up to 2019, of course).

Hellmich prefers the term "people trained in economics" rather than "economists", noting that much of the literature focuses on undergraduate students who have only taken one or a few courses in economics, and can hardly be considered "economists". Hellmich reviews the empirical literature that comes from both lab experiments and field experiments, although it is worth noting that most of the literature makes us of lab experiments. He draws three broad conclusions from the literature:

• People trained in economics behave more in accordance with the standard paradigms of their discipline in situations that are typically described in economic categories. They tend to prioritize their self-interest in games... but this is at least in part an outcome of their expectations about other peoples’ behavior and social interaction can strengthen their cooperativeness.

• Most of the experiments reviewed here involve economic decisions (i.e., involve the allocation of money); in most of the less obviously economic decisions, people trained in economics do not seem to be much less concerned with other people’s welfare and no more likely than other people to expect opportunism from other individuals. All in all... there is not much unambiguous support for the view that training in economics affects the fundamental preferences of people by making them more “selfish” or opportunistic.

• Most empirical evidence seems to be consistent with the self-selection assumption and more than half of the relevant studies—some of them providing high-quality evidence— seem to suggest that there are training effects... Probably both forces play a role.

In other words, the review doesn't really tell us much more than we already knew. People trained in economics behave in a more self-interested way, and part (or perhaps most) of the reason for that is the types of people who choose to train in economics. What Hellmich adds to this research question, though, is a concern about the way that previous research has tried to identify the effects, and in particular, the way that the research is framed (from the perspective of the research participants). He notes that:

...most of the experiments reviewed here lack sufficient consideration of the fact that human subjects in experiments do not mechanistically and passively respond to selected stimuli consciously created and controlled by the experimenter, and in so doing reflect their fundamental preferences. Instead, human subjects tend to interpret cues given to them—perhaps unconsciously— by the experimenter or the environment and what they might know about the theories underlying the experiment... In social dilemmas that involve decisions that are clearly identifiable as being of an economic nature (e.g., because they involve the allocation of money), people compete more than if this trait is less clear... In market-like contexts, there is broad acceptance of self-interest. It may even constitute the social norm to follow...

In other words, perhaps people trained in economics act differently in these experiments because the lab environment, and the wording of the decisions, induces them to apply their economics skills. This would explain why, in the field experiments conducted in more naturalistic settings, the behaviour of people trained in economics differs much less from other people than it does in the lab experiments. Hellmich is essentially arguing for more investigation of real-world decisions, and how they differ between people trained in economics and people who are not. That seems like a sensible suggestion.

However, the overwhelming result from Hellmich's review is that people trained in economics are "different" in meaningful ways (including higher levels of self-interest), and that difference should be recognised. He concludes that:

...as provisional steps, we should perhaps try to make students more aware of the fact that most economists understand key elements of neoclassical theory—like the homo economicus—as an instrument to explain macrophenomena rather than as a normative model of micro-behavior and how other elements of the “culture” of the discipline might make their judgments deviate from that of other groups.

In other words, our students (and other people) need to understand that self-interested behaviour is an assumption that we make in economic models, and not an ideal to strive for.

Read more:

• Does education in economics make politicians corrupt?
• The impact of business and economics education on moral competence
• This study tells us nothing about whether studying economics make students less ethical
• Does studying economics make you selfish?

Qantas tries to execute a break-out of Air New Zealand's locked-in customers

As I noted in this post last week, customer lock-in occurs when consumers find it difficult (costly) to change once they have started purchasing a particular good or service. Having locked-in consumers is quite profitable for firms. They can raise their prices without fear of losing those consumers, or they can leverage their locked-in status to sell them other things.

Of course, if another firm wants to compete with a firm that has locked in its consumers, the competing firm may need to find some way of breaking those consumers out of being locked in. That usually involves trying to lower the switching costs that are keeping the consumers locked in. We saw an example of this late last year, when Qantas made a bid to lure away Air New Zealand's frequent flyers, as reported in the New Zealand Herald in November:

Qantas is targeting Air New Zealand’s upper-tier Airpoints members as it looks to grow its loyalty programme here beyond one million members.

As part of an aggressive push into New Zealand, Qantas will fast-track Gold members of other airline loyalty programmes into its scheme.

Those who hold Gold or higher equivalent status with other ‘‘select airlines’' can fast-track to Qantas Gold by earning 100 status credits in 90 days on flights with Qantas, Jetstar and partner airlines.

Gold status is usually obtained by earning 700 status credits in a membership year.

In addition, participating members will get access to the airline’s network of Qantas Club lounges and extra checked baggage during the 90-day fast-track offer...

Qantas is also targeting a wider range of New Zealanders to ensure they take advantage of points they already have.

Qantas Frequent Flyer will remove the $60 join fee on its website later this month.

Loyalty schemes, like frequent flyer programmes, lock consumers in because if they switch to a different programme, they lose the benefits that their current programme provides, and their frequent flyer points or airmiles will eventually expire (those are the switching costs). Qantas is trying to reduce those switching costs by fast-tracking Air New Zealand Gold Airpoints members to Qantas Gold, meaning that consumers who switch wouldn't lose their frequent flyer benefits (or wouldn't lose them for long). The switching costs aren't eliminated, because their Air New Zealand frequent flyer points will eventually expire, but they are substantially reduced. The lower cost of switching would probably attract at least some Air New Zealand frequent flyers to make the switch. As the article notes:

Qantas made a similar offer to Air NZ Gold members in 2020 which [Qantas Loyalty chief executive Andrew] Glance said had been successful.

Taking advantage of switching costs and customer lock-in is an important way that firms use to increase their profitability. It isn't surprising that firms have discovered countermeasures to restrict their competitors' ability to lock-in customers. What might be more surprising is that Air New Zealand didn't appear to retaliate by offering a similar deal for Qantas frequent flyers!

This week in research #69

Here's what caught my eye in research over the past week:

• Altindag, Cole, and Filiz (with ungated earlier version here) find that students' academic performance is better when their race matches their teachers, but that this is only true for students who are younger than their teacher, and not for students who are a similar age or older than their teacher (role models clearly matter)
• Calamunci and Lonsky (open access) find that, between 1960 and 1993, an Interstate highway opening in a county led to an 8% rise in total index crime, driven by property crime (burglary, larceny, and motor vehicle theft)
• Achard et al. (open access) find that individuals living close to newly installed refugee facilities in the Netherlands developed a more positive attitude towards ethnic minorities and became less supportive of anti-immigration parties compared to individuals living farther away

Mobile phone providers and the repeated switching costs game

This week, my ECONS101 class covered pricing and business strategy, and one aspect of that is switching costs and customer lock-in. Switching costs are the costs of switching from one good or service to another (or from one provider to another). Customer lock-in occurs when customers find it difficult (costly) to change once they have started purchasing a particular good or service. The main cause of customer lock-in is, unsurprisingly, high switching costs.

As one example, consider this article from the New Zealand Herald last month:

A new Commerce Commission study has found the switching process between telecommunications providers is not working as well as it should for consumers...

The study found 50% of mobile switchers and 45% of broadband switchers ran into at least one issue when switching.

The experience was so bad that 29% of mobile switchers and 27% of broadband switchers said they wouldn’t want to switch again in future...

The commission’s latest consumer satisfaction report found that 31% of mobile consumers and 29% of broadband consumers have not switched because it requires ‘too much effort to change providers’...

Gilbertson said a lack of comprehensive protocols between the “gaining” service provider and the “losing” service provider was a central issue with the current switching process.

This led to a number of problems, including double billing, unexpected charges, and delays.

The difficulty of changing from one mobile phone provider to another is a form of switching cost. It's not a monetary cost, but the time, effort, and frustration experienced by consumers wanting to switch makes the process of switching costly. And because the process is costly, mobile phone consumers are locked into their current provider.

It is clear why a mobile phone provider would want to make it difficult (costly) for its consumers to switch away from it and use some other provider. However, why don't mobile phone providers try to make it easier to switch to using their service instead? Maybe they could have staff whose role is to help consumers to navigate the process of switching to their service. That would allow the mobile phone provider to attract consumers and capture a greater market share. The answer is provided by considering a little bit of game theory.

Consider the game below, with two mobile phone providers (A and B), each with two strategies ('Easy' to switch to, and 'Hard' to switch to). The payoffs are made-up numbers that might represent profits to the two providers.

To find the Nash equilibrium in this game, we use the 'best response method'. To do this, we track: for each player, for each strategy, what is the best response of the other player. Where both players are selecting a best response, they are doing the best they can, given the choice of the other player (this is the definition of Nash equilibrium). In this game, the best responses are:

1. If Provider B chooses to make switching easy, Provider A's best response is to make switching easy (since 3 is a better payoff than 2) [we track the best responses with ticks, and not-best-responses with crosses; Note: I'm also tracking which payoffs I am comparing with numbers corresponding to the numbers in this list];
2. If Provider B chooses to make switching hard, Provider A's best response is to make switching easy (since 8 is a better payoff than 6);
3. If Provider A chooses to make switching easy, Provider B's best response is to make switching easy (since 3 is a better payoff than 2); and
4. If Provider A chooses to make switching hard, Provider B's best response is to make switching easy (since 8 is a better payoff than 6).

Note that Provider A's best response is always to choose to make switching easy. This is their dominant strategy. Likewise, Provider B's best response is always to make switching easy, which makes it their dominant strategy as well. The single Nash equilibrium occurs where both players are playing a best response (where there are two ticks), which is where both providers make switching easy.

So, that seems to suggest that the mobile phone providers should be making switching to them easier. However, notice that both providers would be unambiguously better off if they chose to make switching hard (they would both receive a payoff of 6, instead of both receiving a payoff of 3). By both choosing to make switching easy, it makes both providers worse off. This is a prisoners' dilemma game (it's a dilemma because, when both players act in their own best interests, both are made worse off).

That's not the end of this story though, because the simple example above assumes that this is a non-repeated game. A non-repeated game is played once only, after which the two players go their separate ways, never to interact again. Most games in the real world are not like that - they are repeated games. In a repeated game, the outcome may differ from the equilibrium of the non-repeated game, because the players can learn to work together to obtain the best outcome.

So, given that this is a repeated game (because the providers are constantly deciding whether to make switching easier or not), both providers will realise that they are better off making switching harder, and receiving a higher payoff as a result. And unsurprisingly, that is what happens, and it doesn't require an explicit agreement between the players - the agreement is 'tacit' (it is understood by the providers without needing to be explicit). Each provider just needs to trust that the other providers will make switching hard (because there is an incentive for each provider to 'cheat' on this outcome). Any instance of cheating (by making switching easier) would be immediately known by the other providers, and the agreement would break down, making them all worse off. So, there is an incentive for all providers to keep switching hard for the consumers. Even a new entrant firm into the market, which might initially make it easy for consumers to switch to them in order to capture market share, would soon realise that they are then better off making switching more difficult (it is not so long ago (2009) that 2degrees was a new entrant in this market).

The Commerce Commission is correct that the difficulty of switching mobile phone providers (the switching cost) keeps consumers with their current provider (customer lock-in). The result is that the mobile phone providers can profit from increasing prices for their lock-in consumers. The only solution to this situation would be to find some way to force a breakdown of the tacit arrangement. Then the market would settle at the equilibrium of all providers making it easy to switch to them. This may be an instance where some regulation is necessary.

The emerging debate on Oprea's paper on complexity and Prospect Theory

Late last year, an article in the American Economic Review by Ryan Oprea caught my attention (and I blogged about it here). It purported to show that the key experimental results underlying Prospect Theory may in part be driven by the complexity of the experiments that are used to test them. These were extraordinary results. And when you publish a paper with extraordinary results, that could potentially overturn a large literature on a particular theory, then those results are going to attract substantial scrutiny. And indeed, that is what has happened with Oprea's paper.

The team at DataColada, most well-known for exposing the data fakery of Dan Ariely and Francesca Gino (and the resulting lawsuit, which was dismissed), have a new working paper, authored by Daniel Banki (ESADE Business School) and co-authors, looking at Oprea's results (see also the blog post on DataColada by Uri Simonsohn, one of the co-authors). To be clear before I discuss Banki et al.'s critique, they don't accuse Oprea of any misconduct. They mostly present an alternative view of the data and results that appears to contradict key conclusions that Oprea finds in his paper. Oprea has also provided a response to some of their critique.

I'm not going to summarise Oprea's original paper in detail, as you can read my comments on it here. However, the key result in the paper is that when presented with risky choices, research participants' behaviour was consistent with Prospect Theory, and when presented with choices that involved no risk at all but were complex in a similar way to the risky choices ('deterministic mirrors'), research participants' behaviour was also consistent with Prospect Theory. This suggests that a large part of the observed results that underlie Prospect Theory may arise because of the complexity of the choice tasks that research participants are presented with.

Banki et al. look at a number of 'comprehension questions' that Oprea presented research participants with, and note that:

...75% of participants made an error on at least one of the comprehension questions, such as erroneously indicating that the riskless mirror had risk.

Once the data from those research participants is excluded, Banki et al. show that research participant behaviour differs between lotteries and mirrors for the research participants who 'passed' the comprehension checks (by getting all four of the comprehension questions correct on their first try). This is captured in Figure 2 from Banki et al.'s paper:

The two panels on the left of Figure 2 show the results for the full sample, and notice that both lotteries (top panel) and mirrors (bottom panel) look similar in terms of results. In contrast, when the sample is restricted to those that 'passed' the comprehension checks, the results for lotteries and mirrors look very different. Which is what we would expect, if research participants are not 'fooled' by the complexity of the task.

Banki et al. provide a compelling reason why the results for the research participants who failed the comprehension checks looks the same for lotteries and mirrors: regression to the mean. As Simonsohn explains in the DataColada blog post, this arises because of the way that a multiple-price list works:

When the dependent variable is how much people value prospects, regression to the mean creates spurious evidence in line with prospect theory. When people answer randomly for 10% chance of $25, they overvalue it, because the “right” valuation is $2.50, and the scale mostly contains values that are higher than that. When people answer randomly for 90% chance of $25, they undervalue it, because the “right” valuation is $22.50 and the scale mostly contains values that are lower than that. Thus, random or careless responding will produce the same pattern predicted by prospect theory.

Oprea responds to both of these points, noting that:

...a range of imperfectly rational behaviors including noisy valuations, anchoring-and-adjustment heuristics, compromise heuristics and pull-to-the-center heuristics will all tend to produce prospect-theoretic patterns of behavior simply because of the nature of valuation. BSWW offer this possibility as an alternative to the Oprea (2024)’s account of his data, but in fact these are examples of exactly the types of cognitive shortcuts Oprea (2024) was designed to study.

In other words, Banki et al.'s results don't refute Oprea's results, but are very much in line with Oprea's. One thing that Oprea does take issue with is Banki et al.'s use of medians as the preferred measure of central tendency. Oprea uses the mean, and when reanalysing the data with the same exclusions as Banki et al., Oprea shows that the mean results look similar to the original paper. So, Banki et al.'s results are not simply driven by excluding the research participants who failed the comprehension checks, but also by switching from using the mean to using the median.

On that point, I'm inclined to agree with Banki et al. The median is often used in experimental economics, because it is less influenced by outliers. And if you look at Oprea's data, there are a lot of large outliers, which become quite influential observations when the mean is used as the summary statistic. However, the outliers are likely to be the observations you want to have the smallest effect on your results, not the largest effect.

Oprea also critiques Banki et al.'s interpretation of the comprehension questions. Oprea rightly notes that:

...it is important to emphasize that these training questions weren’t designed to measure beliefs (e.g., payoff confusion), and because of this they are poorly suited to the task BSWW repurpose it for, ex post. Indeed, evidence from the patterns of mistakes made in these questions suggests that overall training errors largely serve as a measure of the cognitive effort (an important ingredient in Oprea (2024)’s account) subjects apply to answering these questions, and that BSWW therefore substantially overestimate the level of payoff confusion with which subjects entered the experiment.

In other words, the 'comprehension questions' are not comprehension questions at all, but they are really 'training questions' that were used to train the research participants to understand the choice tasks that they would be presented with. And so, using those training questions overall as a measure of understanding misses the point, and seriously underestimates the amount of understanding of the task that research participants had by the time they had completed the training questions.

Oprea's response is good on this point. However, if the training questions had really done a good job of training the research participants, then all participants should have had a similar level of understanding by the end of the training questions, and there should be no detectable differences in behaviour between those with more, and those with fewer, 'failed' training questions. That wasn't the case - the behaviour of the research participants who made errors in training was much more likely to be the same for lotteries and mirrors than was the behaviour of research participants who made no errors. To clear this up, it would have been interesting to have research participants also complete 'comprehension questions' at the end of the experimental session, to see if they still understood the tasks they were being asked to complete. At that point, those failing the comprehension questions could be dropped from the dataset.

One point of Banki et al.'s critique that Oprea hasn't engaged with (yet, although he promises to do so in a future, more complete response), is their finding that a larger than 'usual' proportion of the research participants fail 'first order stochastic dominance' (FOSD). A failure of FOSD in this context means that a research participant valued a lottery (or mirror) lower than a similar lottery that was strictly better. For example, valuing a 90% chance of receiving $25 less than a 10% chance of receiving $25 is a failure of FOSD. Banki et al. show that:

We begin by examining G10 and G90. Violating FOSD here involves valuing the 10% prospect strictly more than the 90% one. Across all participants (N = 583), 14.8% violated FOSD for mirrors, and 13.9% for lotteries. These rates are quite high given that the prospects differ in expected value by a factor of nine.

Those failure rates are much higher than for other similar research studies. Banki et al. note an overall rate of 20.8 percent in the Oprea results, compared with an average of 3.4 percent across eight other highly cited studies. It will be interesting to see how Oprea responds to that point in the future.

This is an interesting debate so far. Oprea does a good job of summing up where this debate should probably go next:

Ultimately, however, these questions and ambiguities can only be fully resolved by further research. While BSWW’s critique has not convinced me that the interpretation offered in Oprea (2024) is mistaken, I am eager to see new experiments that deepen, alter, or even overturn this interpretation. First, concerns that the Oprea (2024)’s results are a consequence of the design being too confusing to yield insight can only really be resolved one way or another by followup experiments that vary his procedures, instructions and other design choices in such a way as to satisfy us that the Oprea (2024) results are (or are not) overfit to that design.

Indeed, more follow-up research is needed. Prospect Theory hasn't been overturned, yet (and as I noted in my earlier post, it is consistent with a lot of real-world behaviour). However, now we know that it may be vulnerable, and Oprea's paper provides a starting point for testing more thoroughly how much of the experimental results arise from complexity.

[HT: Riccardo Scarpa]

Read more:

• Are experimental measures of loss aversion and behaviour under risk just an artefact of complexity?