Saturday, 21 December 2024

This week in research #54

Here's what caught my eye in research over the past week (which, it appears, was a very quiet week in the lead-up to Christmas):

  • Bietenbeck (open access) finds that exposure to academically motivated classmates causes an increase in student achievement among elementary school students, using data from the Project STAR randomised controlled trial in Tennessee

Wednesday, 18 December 2024

Onshore windfarms vs. birds

Several times recently, I've had conversations with others about environmental objections to windfarms, and specifically about their impacts on birds. I've expressed surprise that anyone could believe that large, slow-moving wind turbines could be a threat to birds. It turns out, there is research that supports the negative impacts of wind turbines on birds (see here or here), but that research doesn't actually demonstrate that wind turbines cause a decrease in bird populations. The problem, of course, is that it isn't feasible to run a randomised controlled trial with wind turbines due to cost - placing wind turbines at random across some areas and not others, and comparing the effect on bird populations in both areas. The cost of such an experiment would be enormous.

Fortunately, there are statistical methods that we can use to try and estimate the causal effects. And that is what this new article by Meng et al., published in the Journal of Development Economics (ungated earlier version here) attempts to do. Specifically, they look at the effect of onshore windfarms on bird biodiversity at the county level in China. They have two measures of biodiversity:

Bird abundance is the average number of birds of a given species per checklist observed at a county-month-year-species level. Species richness is the total number of unique species observed in a given county at the month-year level, which better reflects the diversity of the bird populations.

To establish causality, they use a difference-in-differences (two-way fixed effects) model, which essentially compares the difference in bird biodiversity before and after a windfarm is installed, between counties with and without windfarms. However, Meng et al. go a step further, using an instrumental variables approach, instrumenting for the location of windfarms by the interaction between national-level growth in windfarms interacted with county-level average windspeed at 100 metres. That instrumental variables approach should mitigate issues arising from the correlation of wind turbine location and bird biodiversity.

The novelty of this paper is not just in the methods, but in the data that Meng et al. employ. To measure bird biodiversity, they make use of data:

...from the China Birdwatching Report (CBR, similar to the eBird Reference Dataset), a citizen science dataset consisting of reports from users, including information on individual bird trips and associated characteristics, such as the specific date and time, location of a specific trip, as well as species and quality of birds encountered...

Their dataset covers the period from 2015 to 2022, and includes data collated from over 33,000 checklists. They also control for a variety of other variables:

...including average bird observed duration, average temperature, average visibility, average wind speed, total precipitation, average ozone, percentage of natural park areas in the county, average population density, and average night light value.

Using this data and the two-way fixed effects approach, Meng et al. find that:

A one standard-deviation increases in wind turbines (approximately 84 turbines)... in a given county leads to a 9.75% decrease in bird abundance per checklist from the mean value of 5.38...

...while a one standard-deviation increases in wind turbines (approximately 84 turbines) in a given county decreases the number of unique bird species by 17.67% from the mean value of 66...

Meng et al. also find evidence that the impacts are greater on migratory birds than on resident birds (important given that China is a major migration pathway for migratory birds), and that the effects are larger in forested and urban/farmland than for grassland. There is also evidence that the impact is greatest for the largest bird species.

Finally, Meng et al. show that there are effects of windfarms on neighbouring counties (as well as the counties in which the windfarms are located), and that those effects are somewhat smaller in size. That made me wonder why those analyses were not the primary results in the paper, since it seems obvious that birds may move across county borders.

So, it does appear that windfarms might cause a decrease in bird biodiversity. Meng et al. even address a bunch of concerns that jumped out at me as I was reading the paper, especially that birdwatchers, anticipating that there would be fewer birds near windfarms, do less birdwatching in those locations. On that point, Meng et al. note that:

We do not find a significant impact of wind turbine installations on birdwatcher behaviors regarding the submitted number of checklists...

And they further support that with detailed mobile phone GPS data, showing that there were not fewer trips made to the areas of windfarms, relative to areas further away. However, a couple of concerns do remain, but they are rather technical. First, I wondered why Meng et al. used the interacted variable (national growth in windfarms interacted with windspeed), rather than just windspeed alone. They describe this as a "Bartik-like variable", but we should be cautious about whether Bartik instruments are appropriate (see here). Also, two-way fixed effects models have also come in for criticism recently (see here and here). I'm not going to drag you into the technical details (read the links if you're interested). But suffice to say, this will not be the last word on whether windfarms negatively impact bird biodiversity. However, the best quality study we have so far seems to suggest they do.

Tuesday, 17 December 2024

Licensing of economists, and other fortune tellers

There are certain examples I use in my classes where the origins are shrouded in mystery. They likely come from some obscure note I wrote to myself after reading something online. One of those examples is that there are some states in the US that require fortune tellers to be licensed. [*] I use this as an example of the ridiculousness of occupational licensing, which in many circumstances serves no real purpose other than creating a barrier to entry into the market. After all, what harm could befall consumers from receiving the services of an unlicensed fortune teller, that licensing would help to prevent?

It turns out that the fortune teller example is true. Here's the relevant website with links to the law, as well as this hilarious article which asks the most relevant questions:

How cool would that be to have a fortune tellers license? But then I started to wonder how the licensing process would work. Is there a written examination? Do they hand you a blank piece of paper and expect you to divine the questions and then answer them? Is the test multiple choice or essay? Who grades the essays? Other fortune tellers – kind of like bar exam? Is there a road test? Is reading tea leaves or your palm akin to parallel parking?

I was in New Orleans last month. Walking along Bourbon Street, you see a lot of fortune tellers. I could tell the phony ones. They were the ones that beckoned me over. If they could tell the future, then surely they would have known that I wasn't going to walk over to them, no matter how enthusiastic they waved at me?

Anyway, if fortune tellers are licensed in Massachusetts, does that mean economists should need a licence? After all, economists are regularly asked to tell the future - what is going to happen to GDP, unemployment, interest rates, exchange rates, etc.? Whether economists should be licensed or not isn't a crazy question - there have been calls for that in the past (see here and here). And the consequences of bad fortune telling are likely to be as bad, or worse, when an economist gets it wrong as when a palm reader does. Real risk of harm is the reason that governments license doctors, dentists, and nurses. If there is a real risk of harm from people making poor financial decisions on the advice of economists (or other fortune tellers), maybe they do have to be licensed after all?

[HT: Marginal Revolution]

*****

[*] Although, as it turns out, I have referred to licensing of fortune tellers before, with a relevant link (see here).

Read more:

Monday, 16 December 2024

How university students and staff used and thought about generative AI in early 2023

Somehow, this report languished in my to-be-read pile for over a year. By Natasha Ziebell and Jemma Skeat (both University of Melbourne), it explores the relatively early use of generative AI by university students and staff, based on a small (110 research participants - 78 students and 32 academic staff) survey conducted in April-May 2023.

While the results are somewhat dated now, given the pace of change in generative AI and the ways that university students and staff are engaging with it, some aspects are still of interest. For example, Ziebell and Skeat found that while over 78 percent of academic staff had used generative AI to that point, only 52 percent of students had done so. I think many of us would be surprised that students were not more experimental in their early use of generative AI. On the other hand, perhaps they were simply reluctant to admit to having used it, given that this was a study undertaken by a university that may sanction students for the use of generative AI in assessment?

The other aspect of the paper that still warrants attention are the opportunities and challenges identified by the research participants, which still seem to be very current. In terms of opportunities:

There were a range of opportunities identified for using generative AI as a tool for teaching and learning:

• to generate study materials (e.g. revision materials, quiz/practice questions)

• to generate resources (e.g. as a teacher)

• to summarise material (e.g. coursework material, research papers)

• to generate information (e.g. similar to Wikipedia)

• to provide writing assistance (e.g. develop plans and outlines, rewording and refining text, editing)

• for learning support (e.g. explaining questions and difficult content, as an additional resource, ‘using it like a tutor’)

• as a research tool (e.g. potential for integrating generative AI with library search tools)

• as a high efficiency time-saving tool (e.g. to sort data, gather information, create materials)

• to encourage creative thinking, critical thinking and critical analysis (e.g. students generate work in an AI program and critique it)

I don't think we've moved on substantially from that list of opportunities, and if a similar survey was conducted now, we would see many of the same opportunities are still apparent. In terms of challenges:

The key challenges identified by respondents can be summarised according to the following categories:

- Reliability of generative AI (inaccurate information and references, difficulty fact checking, misinformation)

- Impact on learning (e.g. misusing generative AI, not understanding limitations of the technology)

- Impact on assessment (e.g. cheating, difficulty detecting plagiarism, assessment design)

- Academic integrity and authenticity (e.g. risk of plagiarism, collusion, academic misconduct)

- Trust and control (reliance on technology rather than human thinking, concerns about future advancements)

- Ethical concerns (e.g. copyright breaches, equitable access, impact on humanity)

Unfortunately, just as the opportunities remain very similar, we are still faced with many of the same challenges. In particular, universities have been fairly poor at addressing the impact on learning and assessment, and in my view there is a distinct 'head-in-the-sand' approach to issues of academic integrity and authenticity. Many universities seem unwilling to step back and reconsider whether the wholesale move to online and hybrid learning and assessment remains appropriate in an age of generative AI. The support available to academic staff who are on the frontline dealing with these issues is superficial.

However, academic integrity and authenticity of assessment are only an issue if students are using generative AI tools in assessment. This report suggests that, in early 2023, only a minority of students were doing so. I don't think we can rely on that being the case anymore. One example from my ECONS101 class in B Trimester serves as an illustrative case.

This year (and for many prior years going back to at least 2005), we've had weekly quizzes in ECONS101 (and before that, ECON100). These quizzes this year had 12 questions, generally consisting of ten multiple choice questions and two (often challenging) calculation questions. These quizzes are each worth about one percent of students' grades in the paper, so they are fairly low stakes. Students have generally taken them seriously, and the median time to complete a quiz has been fairly stable at 15-20 minutes over the last few years. Until B Trimester this year, when the median time to complete started the trimester at over 17 minutes, but by the end of the trimester was down to 7 minutes. It isn't clear to me that it is possible to genuinely complete the 12 questions in 7 minutes. Around 16 percent of students completed the last Moodle quiz in four minutes or less. And it wasn't that those students were rushing the test and performing badly. The average score in the quiz for students completing it in four minutes or less was 86 percent (only slightly below the 92 percent average for students who took longer than four minutes). I'm almost certain that the culprit was one of the (now several) browser extensions that will automatically answer Moodle quizzes using generative AI. Needless to say, this year sees the end of Moodle quizzes that contribute to grades in ECONS101.

Anyway, I digress. It would be really interesting to see this sort of study replicated in 2025, and with a decent sample size - it is hard to say much with a sample of 110, split across students and staff. I imagine that we would see many of the same opportunities and challenges would be salient, but that the uses of generative AI have changed in the meantime, and students would now be at least as prolific as users of generative AI as are staff.

[HT: The Conversation, last year]

Sunday, 15 December 2024

The South Waikato compensating differential strikes again

The New Zealand Herald reported last month:

Robbie Winterson says he faces a battle every time he tries to hire new plumbers in his business.

He runs Go Fox Plumbing and Electrical in Putāruru in the South Waikato. When he puts a job ad on Seek or Trade Me, he says, he either gets no New Zealand applicants at all, or only people completely unsuitable for the job.

“We don’t get good-quality applicants at all. You just don’t get people applying.”

Winterson said plumbers would be paid more than electricians and his firm paid the same as they would get in Auckland, but it was still hard to hire.

The problem here is that Go Fox Plumbing is only offering to pay the same as plumbers would receive working in Auckland. To get quality applicants, they might have to offer more. They face essentially the same problem as South Waikato District Council looking for a health and safety manager in 2018, or a Tokoroa GP looking for a doctor in 2016. In all cases, despite offering what they thought was a competitive wage (compared with elsewhere), the employer was unable to attract good people for the role. 

Aside from difficulty in recruiting, there is one common thread there, and that's the location: South Waikato. Before we talk about why that's relevant (and potentially throw some shade on South Waikato), we need to review a bit of theory on compensating differentials.

Wages differ for the same job in different firms or locations. Consider the same job in two different locations. If the job in the first location has positive non-monetary characteristics (e.g. it is in an area that has high amenity value, where people like to live), then more people will be willing to do that job. This leads to a higher supply of labour for that job, which leads to lower equilibrium wages. In contrast, if the job in the second area has negative non-monetary characteristics (e.g. it is in an area with lower amenity value, where fewer people like to live), then fewer people will be willing to do that job. This leads to a lower supply of labour for that job, which leads to higher equilibrium wages. The difference in wages between the attractive job that lots of people want to do and the unattractive job that fewer people want to do is called a compensating differential. The compensating differential essentially compensates workers for working in jobs with negative non-monetary characteristics compared with working in jobs with positive non-monetary characteristics.

Now, if South Waikato employers (the council, or a GP, or a plumber) are all offering 'competitive' wages and unable to find quality workers, then that suggests that they are not offering wages that are high enough to compensate for the non-monetary characteristics of the job. If they have to offer a wage that is even higher than workers would receive in other locations, that suggests that there is a compensating differential at play. For some reason, it appears that workers perceive that working in South Waikato comes with negative non-monetary characteristics relative to other areas, and are unwilling to move and work there at the wages that are being offered.

Workers' perceptions have an important implication for South Waikato employers. If they want to attract quality applicants, then South Waikato employers either need to find some way to convince potential workers that there are positive reasons for living in South Waikato (South Waikato District Council is clearly trying their best), or they need to pay more to offset the perceived negative characteristics of living in South Waikato.

Read more:

Friday, 13 December 2024

This week in research #53

Here's what caught my eye in research over the past week:

  • Dentler and Rossi (open access) use data from Rochester, NY, and find that residents are willing to pay between 7% and 18% of house valuations to avoid crime
  • Kühn and Wolbring (open access) use deepfaked job application videos and find that more attractive applicants score higher in competence ratings and are more likely to be invited for a job interview than less attractive candidates, but that only men consistently profit from their looks, while women benefit from a beauty premium in female-typed, but not in male-typed jobs
  • Akyildirim et al. (including my colleagues Shaen Corbet and Greg Hou) find that a cyberattack leads stock returns to decrease by -0.24%, but the effect reverses in about two weeks
  • Aridor et al. (with ungated earlier version here) summarise the literature on the economics of social media
  • Courtemanche et al. find no evidence that e-cigarette licensure laws affect youth use of 'electronic nicotine delivery systems', using data from the US State Youth Risk Behavior Survey (useful evidence for the New Zealand context as well, I expect)
  • Cannistrà et al. (open access) evaluate the impact of an online game-based financial education tool, and find that it increases financial literacy levels by 0.313 standard deviations, in a sample across four countries
  • Karadas and Schlosky look at the stock trades of US Members of Congress from 2004 to 2022, and find that politicians trade more when Congress is in session and when geopolitical risk is high, as well as making more buy trades when economic policy uncertainty and equity market volatility are high
  • Ash et al. (open access) find that an increase of 0.05 rating points in Fox News viewership, induced by exogenous changes in channel placement, has increased Republican vote shares by at least 0.5 percentage points in recent presidential, Senate, House, and gubernatorial elections

And the latest paper from my own research (or, more accurately, this is the first paper from the thesis research of my PhD student Jayani Wijesinghe, on which I am a co-author along with Susan Olivia and Les Oxley):

  • Our new working paper investigates the trends and factors associated with state-level life expectancy and lifespan inequality (inequality in the length of life) in the US over the period from 1968 to 2020, finding a strong and statistically significant negative correlation between life expectancy and lifespan inequality, and that greater lifespan inequality is associated with state-level education, demographics, various health variables, and income inequality

Also new from the Waikato working papers series:

  • Gibson, Kim and Li examine relationships between nighttime lights (luminosity) and local economic growth for counties in China and the US and districts in Indonesia, and find that GDP-luminosity elasticities vary especially by spatial scale and metro status, and also by period and remote sensing source, meaning that claimed growth effects in previous studies based on nighttime lights may be quite inaccurate

Wednesday, 11 December 2024

A qualitative assessment of the implications of AI for New Zealand

Back in July, Treasury released an analytical note written by Harry Nicholls and Udayan Mukherjee on the implications of artificial intelligence for New Zealand. I read it last month, but didn't have time to write up my comments until now due to travelling. The analytical note is not a quantitative assessment on the economic or labour market or other impacts, but really just testing the waters and providing some broad overview of some of the issues. Fortunately (or unfortunately, depending on your perspective), this hasn't been superseded by subsequent quantitative analysis of the impacts, so it still provides a starting point for some discussion.

Nicholls and Mukherjee see three main issues: (1) the impacts of AI on productivity and investment; (2) the impacts on employment and the labour market; and (3) the development of regulatory approaches for AI. It may be my bias showing, but I think they miss the impact of AI on education and human capital development as a fourth main issue, but perhaps they see it as being subsumed under their first main issue (although they don't mention education at all, and mention human capital only once in the note).

The note discusses each of the three main issues in turn, but first it lays out frameworks for thinking through each issue. There are two (competing) approaches. The first is based on macroeconomic growth accounting framework, where:

AI can be thought of as a form of capital deepening, improving the quantity and quality of capital inputs. It can also be thought of as labour-augmenting technology that improves the quality of labour inputs. And it might be a form of technology that more effectively combines capital and labour, improving multi-factor productivity.

The second is a microeconomic approach, which sees:

generative AI as a form of workplace automation...

There are three types of effects [Acemoglu and Restrepo] identify in this framework:

• Displacement Effect: Automation makes it more efficient to produce some tasks by capital that were previously done by labour, so it reduces the share of labour in production.

• Reinstatement Effect: Automation can allow a more flexible allocation of tasks in production, and so can create a range of new tasks in which labour has a comparative advantage.

• Productivity Effect: Automation allows some of the tasks previously performed by labour to now be performed more cheaply by capital, and so increases the value-added in production and so potentially the overall demand for labour.

The benefit of this framework is that it helps to think through how the net impact of new automation technologies can be thought about as the result of the combination of the strength of these effects.

For example, the total effect of AI on labour demand is a combination of the Productivity Effect increasing the demand for labour and the Displacement Effect replacing labour from tasks it previously performed. Similarly, the impact of AI on wages could be thought of as the net change on marginal productivity from the Reinstatement Effect creating new tasks for labour demand and the Displacement Effect create new labour supply from replaced tasks.

I found the frameworks quite useful, but it is difficult to see how they can be reconciled (which, to be fair, is a common problem comparing macroeconomic and microeconomic frameworks). Nicholls and Mukherjee then turn to thinking through the three issues, but the frameworks seem to be a bit lost in those sections. However, they conclude by offering some potential areas for future research, which are:

• A deeper exploration of the policy levers that might accelerate the diffusion of AI, reducing the lag that characterises New Zealand’s diffusion of new technology.

• Investigating the likely impacts of AI on the productivity and competitiveness of particular sectors, or small-to-medium sized enterprises (SMEs), given their significance to the New Zealand economy.

• Taking a closer look at the implications of AI for employment and labour markets, with a particular focus on how these impacts should inform our policy settings around skills, immigration, and labour markets.

• Examining how AI intersects with our economic security, particularly if AI development is concentrated in a small number of large multinational overseas-based technology companies.

• Considering how AI could lift the productivity of New Zealand’s public sector, in order to maintain or enhance service levels in the face of pressures like an aging population.

Again, I would add the impacts on education and human capital development here, as well as research on the distributional consequences of AI. Those questions are important to understanding the future labour market impacts, as well as the need for changes to policy settings on taxes and transfers. Anyway, this is a good starting point as a think piece, and it is good to see that Treasury are actively thinking in this space, and hopefully there is more careful analytical work to come from them.

[HT: Inside Government, back in July]

Tuesday, 10 December 2024

Enforced class attendance might make better students worse off

Each trimester, I make it very clear to students that regular attendance at classes offers greater chances at success for them. Those statements are based on years of observational data on student performance in my papers - students who attend more regularly have higher pass rates, and better grades. Part of that is clearly selection bias - students who attend less often may have lives that are more complicated, making it difficult for them to keep up with studying. Alternatively, students who attend less often may be less motivated, and that affects both attendance and willingness to study. However, just before the pandemic, we ran an experiment in the ECONS101 class that provided some evidence that incentivised non-attendees to attend class, and that did appear to improve performance, suggesting that at least some of the effect of attendance on performance is causal (that study has still not been written up for publication, sadly).

On the other hand, I am very aware of the large (and still growing) literature comparing online, blended, and in-person teaching and learning modes (see this post, and the long list of links at the end of it). My takeaway from that literature is that teaching mode has no effect on student performance (or student learning) on average, but has important heterogeneous effects. More able (or more self-directed) students tend to do better in online and blended learning environments, whereas less able (or less self-directed) students tend to do worse in those environments. Based on that evidence, I encourage in-person attendance as I am somewhat more focused on ensuring that the less able students have the best opportunity to succeed. However, in the back of my mind, I do harbour some reservations that the top students are not necessarily getting the best learning experience as a result.

Those reservations seem to now have some support, from this 2023 article by Sofoklis Goulas (Brookings Institution), Silvia Griselda (Bocconi University), and Rigissa Megalokonomou (Monash University), published in the Journal of Economic Behavior and Organization (ungated earlier version here). They look at a relaxation of attendance rules at Greek high schools, and the effects on attendance and student performance. Specifically:

The education system in Greece has a very strict attendance policy. All high- school students attend classes back to back with short recesses in between from 8 am to 2 pm, Monday through Friday...

Until the end of the 2005-06 school year, every student was allowed a maximum of 50 unexcused and 64 hours (about 10 days) of excused class absences in a year. One absence is equal to one missed school period...

Before the beginning of the 2006-07 school year, the Ministry of Education announced a nationwide policy to encourage students’ autonomy (Gov. Gazette 65/A/30-3-2006). The policy provided eligible students with 50 additional excused class absences. Every student who had received a raw GPA higher than 75% the year before was eligible to take up more parent- approved absences in the current year.

Goulas et al. then use a difference-in-differences-in-differences approach, which involves comparing the difference between students who were eligible (greater than 75% GPA the previous year) and those that were not eligible, between Grade 11 (no relaxed policy) and Grade 12 (relaxed policy), between the 2006 and 2007 years (when the policy was implemented). That allows them to estimate the causal impact of the change in attendance policy on student outcomes, including high-stakes exam performance (in subjects that count towards university entrance) and low-stakes exam performance (in subjects that do not count towards university entrance). Goulas et al. find that:

Targeted students in the treated cohort took more absences and improved their high-stakes exam performance as a result of the increased autonomy policy. In particular, targeted students increased their total (excused) absences by four (three) additional class hours—roughly 0.09 (0.13) standard deviations—relative to non-targeted students during the year the increased autonomy policy was in effect. Targeted students' high-stakes exam performance increased by 0.07 standard deviations—or 0.019 standard deviations per missed period—due to the increased autonomy policy. Targeted students' low- stakes exam performance remained unaffected by the increased autonomy policy.

So, the policy, which was targeted at top students, appeared to increase their class absences (which was what it was designed to do), and improved those students' performance. Goulas et al. also show that: 

...the increased autonomy policy is found to be associated with an increase in university admission score of 0.13 standard deviations... also... increased autonomy is associated with being admitted to university degree programs of higher quality/selectiveness—an improvement roughly equivalent to two percentiles in the distribution of degree quality.

So, the relaxation of attendance for top students also affects their long-term prospects in a positive way. However:

The policy is also associated with an increase of roughly 0.12 standard deviations in non-targeted students’ excused absences (roughly three additional excused class absences) in the year the increased autonomy policy was introduced. The effect of the policy on non-targeted students’ total absences or on school performance is not statistically significant. The unexcused absences of targeted and non-targeted students are unaffected by the increased autonomy policy.

So, other students also had more absences, but their performance was not affected. At least they weren't made worse off. Goulas et al. then dig a little deeper, showing that the change in absences was greatest among the top quartile of eligible students (in terms of GPA). However, that analysis doesn't show that those top-quartile students perform any better than other eligible students (in fact, the effects seem to be largest for the lowest quartile of eligible students).

What might be causing these results? Goulas et al. identify five potential mechanisms:

A first mechanism through which increased autonomy could influence performance is because of the attendance-performance association, which may also differ across students... A second mechanism through which increased autonomy could affect performance is related to the effective class size. As students take more class absences due to the increased autonomy policy, the class size decreases and student performance increases. A third mechanism may be related to changes in peer characteristics due to the increased autonomy policy. As targeted students skip class more often, the effective peer characteristics may change, impacting student performance... A fourth channel of influence of increased autonomy on performance is through teachers. As targeted students skip class more often because of the increased autonomy policy, instruction might become more effective... A fifth channel of influence of the increased autonomy policy on performance is related to student motivation from agency as suggested by the Self Determination Theory... Students targeted by the increased autonomy policy might feel they are being treated as responsible individuals, capable of making optimal decisions on their own, potentially feeling motivated to perform higher...

Goulas et al. then show that there is little evidence to support the effective class size, effective peer characteristics, and the instruction quality mechanisms, because there are similar impacts on eligible students in classes with higher or lower proportions of eligible students. That leaves the attendance-performance mechanism and greater agency. Goulas et al. show that controlling for decreased attendance explains most of the effect of the policy change on student outcomes, which suggests that it is the change in attendance that most explains the change in student performance.

What can we take away from this study? Class attendance may well make the top students worse off in terms of their academic performance. However, it is likely that what students do with the spare time does need to also be considered. This policy change was directed at the top 25 percent of Greek high school students, and they were well aware that they had a high-stakes university entrance examination at the end of the year. And, the additional absences needed to be approved by parents, who are likely to maintain at least some control over their children's activities during their absences. In other contexts, and with less motivated (or less closely supervised) students, the impacts are unlikely to be the same.

Also, I'd be cautious before extending these results to the university context. In my experience, students who are not attending are likely to be working (or playing video games) rather than studying during that freed up time. And past studies have shown that working impacts negatively on student performance (see this post, and the links at the bottom of it). However, the study does give some food for thought - maybe strict attendance norms can be relaxed for top students who are clearly engaged in other outside learning activities?

Friday, 6 December 2024

This week in research #52

I started this week at the Australia New Zealand Regional Science International (ANZRSAI) Conference in Canberra. Unfortunately, I left my notebook in one of the conference rooms at the end of the second day, so most of my notes were lost. However, from memory (and the camera roll on my phone) here are some of the highlights I found from the conference:

  • Andrew Leigh (economist and Australian MP) gave a great keynote on inequality in Australia based on his revised book Battlers and Billionaires, although the most interesting bit was a graph on mentions of 'inequality' in Australian parliamentary debates, which is fairly low until a massive spike from 2011 onwards (I guess politicians didn't really care about inequality before then?)
  • Putu Widyastaman presented on the relationship between gentrification and prostitution in Jakarta, showing that gentrification reduces prostitution in an urban village (roughly equivalent to a neighbourhood), but increases prostitution in surrounding urban villages
  • John Madden presented on the economic impact of Victoria University's west Melbourne campus (although the counterfactual of 'what would have happened if Victoria University didn't have a campus in west Melbourne' is always going to be a challenge)
  • Robert Tanton presented on a geocoded synthetic Census that is being developed for Australia, allowing researchers to conduct research on Census data without access to the underlying Census data (which is not a problem in New Zealand, where access to the Census data is available through the secure Integrated Data Infrastructure)

Aside from the conference, here's what caught my eye in research over the past week:

  • Duan, Yuan, and Snyder find that a one standard deviation increase in the local sex ratio raises rural unmarried males’ likelihood of temporary migration in China by 3.6 percentage points
  • Vatsa and Pino (open access) find that petrol price shocks had a slightly delayed but persistent effect on one-year inflation expectations in New Zealand, whereas five-year inflation expectations were largely insensitive to these shocks
  • Meng et al. (with ungated earlier version here) find that a one-standard-deviation increase in wind turbines reduces bird abundance by 9.75% and leads to a 12.2% reduction in bird species richness at the county level in China
  • Arora and Roy (with ungated earlier version here) do not find any significant differences in student evaluations of teaching received by female and male professors, using an experimental approach in India
  • Abel et al. find that, in an experiment, investors are more likely to continue to follow financial advice from male professionals following advice that proves incorrect, but 47 percent less likely to follow financial advice from female professionals in the same circumstances
  • Cordes, Dertwinkel-Kalt, and Werner (open access) investigate the economics of 'loot boxes' in online and mobile games using an experimental study design, and find that common design features of loot boxes (such as opaque odds and positively selected feedback) double the average willingness-to-pay for lotteries
  • Klöcker and Daumann (open access) develop a theoretical model to explain success and dominance in international sports
  • De Acutis, Weber, and Wurm (open access) conduct a meta-analysis of 51 studies on the effect of gender quotas on company boards, and find that (among other results) stock market returns are negatively affected by quota policies (I feel like there is more to this than meets the eye on a quick skim-read though)
  • König et al. (open access) find that there is a statistically significant, but very small, positive association between short-term ambient temperature changes and individuals’ general willingness to take risks, using a large survey in Augsburg, Germany
  • Lin, Churchill, and Ackermann (open access) find using 14 waves of HILDA data in Australia that a 1 percent increase in the proportion of a postcode that has access to the national broadband network (NBN) is associated with a 1.573 increase in Body Mass Index and a 6.6 percentage point increase in the probability of being obese
  • Oprea challenges the existence of loss aversion by showing using experimental data that it may simply arise because of complexity of experimental tasks that are used to demonstrate loss aversion (which may also explain these results, I guess?)

Sunday, 1 December 2024

This week in research #51

It's been a slow week again this week, as I've been travelling for the Australia New Zealand Regional Science International (ANZRSAI) Conference in Canberra (more on that next week). Nevertheless, here's what caught my eye in research over the past week:

  • Borbely et al. (open access) find that a universal free school meal policy covering the first three grades of primary schools in Scotland had a small positive effect on attendance and a small negative effect on health-related absences (potentially important results given New Zealand's free school lunch policy and mooted changes)
  • Asanov, Schirmacher, and Bühren (open access) conduct a meta-analysis of 'lost letter' experiments (78 studies with an overall sample size of 53,504 letters from 18 countries), and find that the return rate is lower for political or deviant issues, stamped letters are more likely to be returned, but letters with money are not more likely to be returned, and a high socio-economic environment increases the chances of the return

Monday, 25 November 2024

Try as they might, the Australian Green party can't make university education free

The Australian Green party has proposed cancelling all student debt in Australia, as part of an aim for government to provide "free education for life". However, free education is not free. In an article in The Conversation earlier this month, Bruce Chapman (Australian National University) makes a case against the claim that cancelling student debt would make university education free. In Chapman's argument, someone has to pay the cost of providing education, and if it isn't students paying, then taxpayers will be the ones paying:

If there is no charge for a university degree, this means all taxpayers (including those without a university education) are fully subsidising graduates, who get lifetime advantages from their education.

In other words, calling for “free” universities is equivalent to supporting financial assistance going from the poor to the privileged.

Just about all political parties in Australia – and most governments around the world – agree this is not a not a wise idea.

Chapman is correct, of course. The cost doesn't go away just because students aren't paying some proportion of it. However, even if students are not paying any monetary cost, students are not receiving free education. That's because there are opportunity costs associated with education as well.

An opportunity cost is the cost of not pursuing the opportunity to do something else (the term dates back to the 19th Century Austrian economist Friedrich von Wieser). Studying (at all levels) takes time, and time is scarce. If a student devotes some of their scarce time to studying, they can't use that time for other activities that are valuable to them (and let's not get into the issues with students multitasking in class). A student might give up some leisure time, or work time, in order to complete their studies. The value of the time that they have given up is the opportunity cost of their education.

It turns out that the opportunity cost is the largest component of the cost of a university education. Think about it. A full-time student gives up at least three years of working in order to complete a university degree. If they are working while studying, chances are that they are working in a lower-paid job than they would have if they were instead working full time. The cost of their tuition fees pales in comparison to the cost of foregone income while they are studying.

None of this is to say that university education is not a good idea. For the majority of students, the benefits (higher lifetime income) far outweigh the costs (including opportunity costs) of education. There are also myriad social benefits from having a more educated population, which is why the government generally subsidises education for domestic students. That some of the benefits are private is one justification for students paying some of the costs of tertiary education (as Chapman notes in his article).

The Australian Green party can't legislate away the costs of education. Someone has to pay those costs, even if it isn't the students themselves. However, even if there were no monetary cost, the Australian Green party can't make education free for students, because even when education has no monetary cost, it will still have a large opportunity cost.

Read more:

Sunday, 24 November 2024

Book review: Understandable Economics

I used to review a lot of popular economics books, the kind that are written to explain key economic theories to general readers. Popular economics books vary widely in quality, but also in the ideological underpinning of the authors, and some authors are better than others at muting the underlying ideology. When they fail (or don't try) to avoid the ideology, to me it often gets in the way of a clear-headed explanation or omits key understandings of the real world. For example, that was the case for the market fundamentalist book Common Sense Economics (which I reviewed here). Since reading that book, I have mostly avoided popular economics theory books.

However, I couldn't avoid them forever, and I recently finished reading Howard Yaruss's Understandable Economics. Yaruss is absolutely not a market fundamentalist, and his sympathies to more left-wing ideals are clearly on display throughout the text. It can be a little preachy, but to me it didn't get in the way of understanding. Apparently, the original title of the book was to be 'Economics for Activists'. I'm glad Yaruss didn't choose that title, as I don't think it accurately conveys the content of the book.

The book is tightly focused on understanding the US economy, and much of the content relates to macroeconomics and public policy. Readers from other countries will therefore get less out of the book than US readers (although the general underlying principles are mostly the same for other countries). The examples are also very specific to the time that the book was published. This is a book for its time, and it will not remain current for too long.

Having said that, I really appreciated some of the commentary and some of the explanations. For example, Yaruss clearly explains why currency in a bank vault is not 'money' (as defined by economists):

Only currency in the hands of someone who can actually spend it, like a consumer, a business, the government, or a thief who stole currency from the bank and managed to get away, counts as money. So, although currency in bank vaults may look very much like money, it cannot be spent by anyone and, therefore, isn't money.

I also liked the example of an auction to explain money's role in creating inflation:

Imagine an auction where a fixed number of play "dollars" are divided among participants so that they can buy the goods on display. If the total number of play dollars were increased (remember, these play dollars have no value outside of this auction) without an increase in the number of goods for sale, people would be willing to bid and will, in fact, bid more and pay more for each good. Therefore, the bids, or prices, for each of these goods will go up - there will be inflation.

Back in the days when I hoped to one day write a book about the economics of the reality TV series Survivor, I had intended to use the 'Survivor Auction' (which has recently been revived for season 46 after many years absent) as an example. The explanation for why a Survivor contestant would pay $400 for a burger in the Survivor Auction would be almost identical to Yaruss's explanation above.

I also appreciated Yaruss's humour. Consider this quip about the averages:

If there is wide variation, an average can be very misleading. Think about the fact that the "average" adult human being has one breast and one testicle.

Overall, I enjoyed this book. However, I think that there are better books within the popular economics category that general readers would benefit from. However, in terms of clearly explaining the US economy and public policy, this book is great.

Saturday, 23 November 2024

Jared Cooney Horvath on how generative AI could harm learning

In a post last month about generative AI, I expressed some scepticism towards those among my colleagues who are trying to integrate generative AI into assessment (an "if you can't beat them, join them" solution to the impact of generative AI on assessment). I also expressed some hope that generative AI can be used in sensible ways to assist in student learning. Both of those views are contested. They certainly are not universally held among teachers.

In a recent article on the Harvard Business Publishing website, Jared Cooney Horvath outlines three critical problems generative AI poses for learning: (1) AI tools lack empathy, and an empathetic learner-teacher relationship is a strong contributor to learning; (2) while AI tools are good at retrieving information, in so doing they make having internal knowledge less important for students, and yet it is a broad internal knowledge that helps us to understand and solve problems; and (3) generative AI encourages multitasking, which is bad for learning.

On the latter point, Horvath concludes that:

It’s not that computers can’t be used for learning; it’s that they so often aren’t used for learning that whenever we attempt to shoehorn this function in, we place a very large (and unnecessary) obstacle between the learner and the desired outcome—one many struggle to overcome.

Finally, Horvath notes one positive for generative AI and learning:

There is one area of learning where generative AI may prove beneficial: cognitive offloading. This is a process whereby people employ an external tool to manage “grunt work” that would otherwise sap cognitive energy.

However, as noted above, when novices try to offload memorization and organization, learning is impaired, the emergence of higher-order thinking skills is stifled, and without deep-knowledge and skill, they’re unable to adequately vet outputs.

Experienced learners or experts can benefit from cognitive offloading. Imagine a mathematician using a calculator to avoid arithmetic, an event planner using a digital calendar to organize a busy conference schedule, or a lawyer using a digital index to alphabetize case files. In each of these scenarios, the individual has the requisite knowledge and skill to ensure the output meaningfully matches the desired outcome.

Horvath hasn't really changed my views on generative AI and learning. He does give some food for thought though, especially in relation to the value of created a finetuned AI designed to help with a particular course. If students use it as an interactive tutor, to help them develop their internal knowledge, then it is likely positive. However, if they use it purely to ask contingent questions, it may impair their ability to develop that internal knowledge and make them worse off. I wonder if there are particular learning tasks that can be used to encourage the former behaviour without too many students resorting to the latter? Clearly I have more thinking to do on this before I roll something like that out for my students.

{HT: Mary Low]

Read more:

Friday, 22 November 2024

This week in research #50

As I mentioned, last week I was at the North American Regional Science Congress in New Orleans. This isn't a science conference per se. Regional science is essentially a mix of economics, geography, sociology, and political science (and a bunch of other fields mixed in as well). As is often the case, there were more sessions that I wanted to attend than I could possibly attend, but here are some of the highlights I found from the conference:

  • My long-time friend and collaborator Matt Roskruge presented on the challenges of developing quantitative measures of Māori social capital (my takeaway was that it may be best to throw away the Western conceptions of social capital, and start over with a Te Ao Māori (Māori worldview) perspective, but apparently that has been done several times already)
  • Steven Deller presented on elder care and female labour force participation, showing that female labour force participation is lower in counties that have less access to elder care
  • Rosella Nicolini presented data that showed immigrants in rural areas are associated with increased GDP growth in Spain, while immigrants in urban areas are associated with decreased GDP growth
  • Rafael González-Val presented analysis of the impacts of the Spanish Civil War, showing a large (12 percent) reduction in industrial employment in provinces aligned with the Republicans, compared to those aligned with the rebels (although it must be noted that all of Spain's main industrial centres were aligned with the Republicans, so it may be no surprise that they declined relative to other regions)
  • Aurelie Lalanne presented some amazingly detailed data on urban growth in France, drawn from historical censuses that have been harmonised, and covering the period from 1800-2015

Aside from the conference, here's what caught my eye in research over the past week:

  • Davis, Ghent, and Gregory (with ungated earlier version here) use a simulation model (calibrated to real-world data) to show that the pandemic induced a large change to the relative productivity of working from home that substantially increased home prices and will permanently affect incomes, income inequality, and city structure
  • Galasso and Profeta find that reducing or eliminating time pressure decreases the math gender gap by up to 40 percent, and that time pressure contributes to the gap through increased anxiety rather than through students modifying their test-taking strategies
  • Mizzi (open access) looks at how economics teachers develop and utilise pedagogical content knowledge (the intersection of pedagogical knowledge and content knowledge) to assist their students’ engagement with disciplinary knowledge in economics (I feel like we should know more about this topic)
  • Liu et al. find that large increases in minimum wages have significant adverse effects on workplace safety, increasing work accidents by 4.6 percent, based on US state-level data
  • Matthes and Piazolo (open access) analyse data from over 40 seasons of professional road cycling races, and find that having a teammate in a group behind positively impacts win probability
  • Fernando and George find that home-team cricket umpires are less biased when working with a neutral colleague (one who is neither a national of the home nor the foreign team)
  • Chilton et al. find that there are large potential gains in better identifying exceptional students in law schools, if changes were made to certain personnel, course, and grading policies to improve the signalling quality of grades (and yet to me, it seems like most universities are on a policy trajectory to reduce the quality of grades as a signal)

Thursday, 21 November 2024

Will New Zealand finally deal with excess demand for access to tourist destinations?

New Zealand has long had a problem with excess demand for access to tourist destinations. I've written about this before, using the Great Walks as an example (see here, and here). Because the price for access to these tourist destinations is too low, the demand for access far exceeds the supply. The consequence is a much-degraded experience for everyone.

The solution, as I have noted before, is to let the price increase. Charge more for access to the Great Walks, and other tourist destinations. And, finally, that may be about to happen. As the New Zealand Herald reported last week:

A $20 access fee for Cathedral Cove, the Tongariro Alpine Crossing, Franz Josef Glacier, Milford Sound, and Aoraki Mount Cook National Park?

The Government is floating the idea of charging visitors – including New Zealanders – as part of two discussion documents, released today, which Conservation Minister Tama Potaka calls the biggest potential changes in conservation in more than three decades...

Charging $20 per New Zealander and $30 per non-New Zealander for accessing those places would bring in an estimated $71 million a year. Charging only international visitors would yield about half that.

Charging for access to these tourist destinations would go some way towards dealing with the excess demand. I'm totally ok with the differential price for New Zealanders and overseas travellers as well (which is something I have noted before, again in the context of the Great Walks). My main concern though is that the price of $20 for New Zealanders and $30 for non-New Zealanders may be too low. However, others have a different view:

But it has triggered a strong reaction from Forest and Bird, which said: “Connection to te Taiao (nature) is a fundamental part of being a New Zealander. All New Zealanders should be guaranteed the ability to connect with our natural environment regardless of how much money they earn.”

How easily can New Zealanders connect with their natural environment when it is thronged with tourists all visiting for free? Charging a price for access limits the numbers of tourists (including other New Zealanders), and makes it more likely, not less likely, that New Zealanders can get genuine access to these places. There is a meaningful difference between accessing a tourist location when there are hundreds of other tourists swarming all over it, and when few people are around and a peaceful engagement with nature is possible.

Quite aside from this being a way for the government to fund the Department of Conservation's operational costs, this proposal to charge a fee for access to these tourist locations is a sensible way to manage demand. Maybe we will finally have a working solution to the excess demand problem in these places.

Read more:

Wednesday, 20 November 2024

Natural capital and the problematic measurement of GDP

I've been thinking a bit about GDP this year, and in particular about the weirdness of its measurement. One of the key problems that has occupied me has been an asymmetry in how capital is accounted for within GDP. When new capital is created, the spending on the new capital adds to GDP. However, when capital is depleted, that depletion does not subtract from GDP. That is why, following a large natural disaster, GDP might actually increase due to rebuilding activity (and because any destruction of capital is ignored).

With that in mind, I was interested to run across this 2019 article by Colin Mayer (Oxford University), Published in the journal Oxford Review of Economic Policy (ungated earlier version here), deep down in my to-be-read pile of articles. Mayer was a member of the UK's Natural Capital Committee, which ran from 2012 to 2020, and this article considers how economists can, and should, approach accounting for natural capital. Mayer distinguishes between economists' traditional view of natural capital, and an approach more similar to how an accountant would approach natural capital:

To the economist, natural capital, like any other asset, is the plaything of humans, there to be treated as mankind sees fit. To the accountant, the firm is an entity of which the managers are the stewards. They are there to preserve the firm and to promote its flourishing. So, too, we should consider whether it is our right to employ nature in the way in which we see fit, or our obligation to act as its steward or trustee.

Mayer's solution is that we should revise how natural capital is treated, and should:

...incorporate a maintenance charge in the balance sheets and profit and loss statements of nations, municipalities, corporations, and landowners to reflect the liability associated with maintaining or restoring these assets.

I think that Mayer could have been much clearer in the explanation here. When natural capital is depleted, through pollution, or extractive industries, or carbon emissions, my view is that the cost of that depletion should directly reduce GDP (which is the equivalent of the 'profit and loss statements' that Mayer refers to). Instead, Mayer seems to be suggesting that this is a liability. Both of those approaches may be correct, given the simple accounting identity (Assets + Expenses = Liabilities + Proprietorship + Revenues). A liability on the right-hand side of that identity equation can arise because of an expense on the left-hand side. However, the labelling as a liability implies an obligation to repay, which may not be the case for all types of natural capital (how would one pay off the liability of mining extraction, for instance?).

Anyway, there is clearly more thinking to be done here. I don't think that economists' approach to natural capital is correct. I think that the approach to other forms of capital (physical, social, and human capital) is similarly flawed. For example, decreasing social capital over time (as accounted by Robert Putnam's 2000 book Bowling Alone (which I reviewed here) should also decrease GDP in my view. By correctly accounting for changes in capital (both upwards and downwards) GDP would better capture changes in societal-level wellbeing.

Saturday, 16 November 2024

This week in research #49

Another quiet blogging week for me, due to travel and the North American Regional Science conference in New Orleans (more on that in next week's post). However, I have been trying to keep up with research, and here's what caught my eye over the past week:

  • Mello (open access) finds that winning the FIFA World Cup increases a country's year-over-year GDP growth by at least 0.48 percentage points in the two subsequent quarters
  • Boyd et al. (open access) describe how an agent-based model could be used to evaluate the impact of minimum unit pricing of alcohol in Scotland (but they don't actually show the results of any such modelling, which is a bit disappointing)
  • Singleton et al. (open access) find that a university located in a town that loses an English Premier League team (due to relegation to the Championship) suffers a reduction in undergraduate year-to-year admissions growth by 4–8 percent
  • Ozkes et al. find that human players of the ultimatum game do not differentiate between human and algorithmic opponents, or between different types of algorithms, but they are more willing to forgo higher payoffs when the algorithm’s earnings benefit a human (this has interesting implications for how humans interact with AI)
  • Gjerdseth (with ungated version here) finds that the destruction of ivory does not reduce elephant poaching rates, using CITES data from 2003 to 2019 (for more on this topic, see this post and the links at the end of it)
  • Hagen-Zanker et al. (open access) use data from a large-scale survey conducted in 25 communities in ten countries across Asia, Africa and the Middle East, and show that there is little consistency in the individual-level and community-level factors that are associated with migration intentions, although women are less likely to have migration intentions, while those with access to transnational social networks are more likely to have migration intentions

Saturday, 9 November 2024

This week in research #48

It's been a quiet week in terms of my keeping up with research, as I've been travelling. However, here's what caught my eye in research over the past week:

  • Rasmussena, Borb, and Petersen merge Twitter data with Danish administrative data, and find that individuals with more aggressive dispositions (as proxied by having many more criminal verdicts) are more hostile in social media conversations, and that people from more resourceful childhood environments (those with better grades in primary school and higher parental socioeconomic status) are more hostile on average, as such people are more politically engaged

In other news, as I said above my wife and I have been travelling this week. We started in Texas, then Oklahoma, and now Arkansas (with Alabama, Mississippi, and Louisiana to come). While in Texas, I had the great pleasure of meeting Cyril Morong, The Dangerous Economist:

Next week may also be fairly quiet on the blog, as I'll be at the North American Regional Science Congress in New Orleans. And, New Orleans, of course.

Sunday, 3 November 2024

Book review: How Big Things Get Done

There are certain books that shouldn't need to be written. Inevitably, those are the books that, in reality, most need to be written. That is certainly the case for How Big Things Get Done, by Bent Flyvbjerg and Dan Gardner. This is a book about big projects, and importantly, how those projects succeed or, as is often the case, how they fail. As the authors note in the preface, it is a book that aims to answer a number of important questions:

Why is the track record of big projects so bad? Even more important, what about the rare, tantalizing exceptions? Why do they succeed where so many others fail?

The book draws on decades of Flyvbjerg's academic research on big projects, as well as his experience both consulting on, and being directly involved in, big projects. Through this work, Flyvbjerg has developed a massive database of projects, their cost and benefit estimates at the time the project began, and the cost over-runs and benefit shortfalls that so often resulted. The numbers do not make for easy reading, and the examples that Flyvbjerg uses range from transport infrastructure to It projects to nuclear power stations to the Olympic Games. On the latter, the book is a useful complement to Andrew Zimbalist's book Circus Maximus (which I reviewed here).

Flyvbjerg and Gardner spend a lot of time discussing failed projects, but devote substantial space to discussing successes, such as Terminal 5 at Heathrow. Many of us will remember the opening of Heathrow for the terrible problems associated with baggage handling in the first few days of opening, but the project itself delivered on time and on budget. Once you read this book, you'll realise just how extraordinary that accomplishment is.

Flyvbjerg and Gardner use the comparison between successful projects and failures to draw a number of lessons. Most of the lessons seem obvious, but clearly those lessons have not been learned well enough in the 'big projects' space, because they are so often not heeded. The biggest lesson of all is to 'think slow, act fast'. Thinking slow means spending substantial time planning before the project begins, ensuring that the risks are well known and have been planned for, before the first spade turns the first sod. Acting fast means completing the project as quickly as possible, to avoid the 'unknown unknowns' from impacting the project - the more delays, the more time there is for something unforeseen to happen.

The 'think slow, act fast' approach seems inconsistent with Silicon Valley's approach to development (as ably described in Jonathan Taplin's 2017 book Move Fast and Break Things, which I reviewed here). Flyvbjerg and Gardner anticipate that counterexample, and note that the two are not inconsistent at all, because:

Planning is doing: Try something, see if it works, and try something else in light of what you've learned. Planning is iteration and learning before you deliver at full scale, with careful, demanding, extensive testing producing a plan that increases the odds of the delivery going smoothly and swiftly.

That is, more or less, what the big tech firms do. Flyvbjerg and Gardner note that iteration is key to those firms' development process, and is generally successful (or where it isn't, the firm can rapidly iterate to something new). In contrast, most big projects are delivered using a 'think fast, act slow' approach that is doomed to failure. 

I really enjoyed this book, even though it does seem quite depressing at times, just how badly big projects are at delivering on their promises (both in terms of costs, and in terms of benefits). The book is not only well researched, but draws on many interviews that Flyvbjerg has completed with people in the industry. The writing did make me wonder what Gardner's contribution was - the whole book is written as if by Flyvbjerg alone (with lots of "I" and "my"), which seems an odd stylistic choice for a co-authored book. Nevertheless it is an enjoyable read, and definitely recommended.

Saturday, 2 November 2024

What does the Cantril Ladder really measure?

Imagine a ladder with steps numbered from 0 at the bottom to 10 at the top. The top of the ladder represents the best possible life for you and the bottom of the ladder represents the worst possible life for you. On which step of the ladder would you say you personally feel you stand at this time?

Now, consider the question you probably just answered. What factors played into your answer? What sorts of things contribute to the best possible life for you, compared with the worst possible life for you? If we used your answer to that question as a measure of life satisfaction, what is it really measuring?

That's not an unimportant question. The first paragraph of this post is a commonly used way of measuring life satisfaction, known as the Cantril ladder (see here). It is used in the Gallup World Poll, and is recommended by the OECD as a way of measuring subjective wellbeing. When researchers (or governments, or others) measure life satisfaction or happiness, it is often the Cantril ladder that is being used.

The question of what the Cantril ladder measures was explored in this recent article by August Nilsson (Lund University), Johannes Eichstaedt (Stanford University), Tim Lomas (Harvard University), Andrew Schwartz (Stony Brook University), and Oscar Kjell (Lund University), published in the journal Scientific Reports (open access, with non-technical summary on The Conversation). Nilsson et al. looked at the framing of the Cantril ladder, and investigated how nearly 1600 people responded to different framings of the question, and the words that they used to describe the top and the bottom of the scale in those different framings, and where they would 'prefer to be' on the scale. The first framing was the traditional Cantril ladder. The second framing essentially replaced the ladder metaphor with the word "scale" (but left the rest intact). The third framing removed references to the "bottom" and "top" (as well as the ladder metaphor). The fourth framing did all of that plus changed "best possible life" to "happiest possible life" (and "worst possible life" to "unhappiest possible life"). And the fifth and final framing instead replaced "best possible life" to "most harmonious life" (and "worst possible life" to "least harmonious life").

Nilsson et al. found that:

The ladder and bottom-to-top scale anchor descriptions influenced respondents to use significantly more words from the LIWC dictionaries Power and Money when interpreting the Cantril Ladder... compared to when these anchors were removed. Of all the words respondents used to describe the top of the Cantril Ladder, 17.3% fell into the Power and Money dictionaries. This language was reduced by more than a third when the ladder was removed in the no-ladder condition (absolute difference of 6.0%, d = 0.35, p < 0.001), and more than halved when the bottom-to-top scale descriptions were removed too (absolute difference of 10.3%, d = 0.64, p < 0.001). Further, for the Cantril Ladder, words in the Power and Money dictionaries occurred 3.3 times as frequently compared to the alternative Harmony anchor condition (absolute difference of 12%, d = 0.77, p < 0.001).

They interpret those results as meaning that:

...the original Cantril Ladder influenced respondents to focus more on money in terms of wealth (whereas when the ladder framing was excluded, they focused more on financial security) than the other conditions.

Were you thinking about the financial aspects of life when you answered the question above? The results seem to suggest that is more common than thinking about social relationships or the various other contributors to our subjective wellbeing. Nilsson et al. don't explore the use of words other than in the 'Power' and 'Money' domains, but it would have been interesting to see some others to compare with.

It's not surprising that financial security, income, or wealth are important contributors to subjective wellbeing or life satisfaction. We should expect people to be better able to satisfy their needs when they have greater financial resources available to them. However, the results on research participants' preferred level on the ladder are genuinely surprising, because:

...over 50% did not prefer the highest level (of 10) in any of the study conditions, and less than a third preferred the top of the Cantril Ladder, which had a significantly lower average preferred level than all the other study conditions.

In other words, even though the top of the Cantril ladder is framed as the 'best possible life', around two-thirds of research participants said that they would prefer not to be at the top of the ladder. This proportion was lower (but still not zero) for other framings, as shown in Figure 4 from the article (where the dark blue part of the bar shows the proportion of research participants who responded that 10 was their preference):

What was your preferred level on the ladder? Did you want to have the best possible life (that is, 10 on the scale)? Or would you prefer to be somewhere just below the best possible life? What do you think about in answering the question on your preferred level? Maybe research participants want 'room to grow' and become even happier or more satisfied with their lives? I have no idea. Nilsson et al. have given us something to really think about here, but unfortunately the article doesn't go far enough in exploring why people don't prefer the top of the ladder. There is definitely scope for further follow-up research on this point.

In addition to being surprising, that last result may call into question how the Cantril ladder is interpreted (on top of the arguments about the validity of happiness data generally - see here, and here, and here). If the top of the scale is not the top of the scale, or if it is different for different research participants, then how do we interpret an average across all people responding to the question? That should make researchers worry, and makes follow-up research even more important.

[HT: New Zealand Herald, back in April]

Read more:

Friday, 1 November 2024

This week in research #47

Here's what caught my eye in research over the past week:

  • Geerling, Mateer, and Wooten (open access working paper) identify a group of “rising stars” in the economics teaching field (where I'm ranked #27 in the world according to their ranking, and #5 outside of the US)
  • Li and Xia find that students just above a letter-grade cutoff in an introductory course are 3.6% more likely to major in the same field as that course, using data from the National University of Singapore
  • Divle, Ertac and Gumren find in an online experiment that although working in a team is more profitable and participants also expect this, a large proportion shy away from teamwork, and that research participants primed with COVID-19 are less likely to self-select into teamwork
  • Dickinson and Waddell find, using data from GitHub, that the transition to Daylight Saving Time reduces worker activity, but that the effects are relatively short-lived, although when using more detailed hourly data losses appear in the early working hours of work days into a second week following the initiation of Daylight Saving Time
  • Naidenova et al. look at twelve years of data from professional Counter-Strike: Global Offensive games and find that there is a substantial decrease in the performance of esports players during overtime, which they attribute to 'choking under pressure', although the impact is less in online competitions compared to live events
  • Martínez-Alfaro, Silverio-Murillo, and Balmori-de-la-Miyar (open access) find in an audit study that job applications from transgender candidates received 36% fewer positive responses than those from cisgender candidates in Mexico

Thursday, 31 October 2024

Book review: Wonderland (Steven Johnson)

When I think about the dramatic changes in society that have occurred since the end of the Industrial Revolution, one of the trends that stands out (to me) is the massive increase in leisure time. In the 19th Century, most people worked far more hours than they do today. The recent decades of that trend were well-described in Daniel Hamermesh's book Spending Time (which I reviewed here). What was left unexplored in that book was the way that leisure pursuits have affected the economy and society.

That is the purpose of Steven Johnson's book Wonderland, which is subtitled "How play made the modern world". Johnson describes the book as:

...a history of play, a history of the pastimes that human beings have concocted to amuse themselves as an escape from the daily grind of subsistence. This is a history of what we do for fun.

The book is comprised of chapters devoted to fashion and shopping, music, food, entertainment, games, and our use of public space. Each chapter is well written and well resourced, and a pleasure to read. Johnson is a great storyteller and the stories he presents are interesting and engaging.

However, from the first chapter, I struggled with the overall thesis of the book, which is that changes in leisure pursuits drove broader societal changes and economic changes. This is most glaringly demonstrated in the first chapter, where Johnson contends that it was the desire for fashion that drove the Industrial Revolution:

When historians have gone back to wrestle with the question of why the industrial revolution happened, when they have tried to define the forces that made it possible, their eyes have been drawn to more familiar culprits on the supply side: technological innovations that increased industrial productivity, the expansion of credit networks and financing structures; insurance markets that took significant risk out of global shipping channels. But the frivolities of shopping have long been considered a secondary effect of the industrial revolution itself, and effect, not a cause... But the Calico Madams suggest that the standard theory is, at the very least, more complicated than that: the "agreeable amusements" of shopping most likely came first, and set the thunderous chain of industrialization into motion with their seemingly trivial pursuits.

In spite of the excellent prose, I'm not persuaded by the demand-side argument for the Industrial Revolution, which flies in the face of lots of scholarship in economic history (as well as in history). Now, it may be that the first chapter just made me grumpy. But Johnson draws several conclusions which are, at best, a selective interpretation of the evidence. And at times, he makes comparisons that are somewhat odd, such as a comparison between the tools and technologies available to artists and scientists and those available to musicians in the 17th Century, concluding that there were fewer and less advanced tools available to artists and scientists than for musicians. There doesn't seem to be any firm basis to make such a comparison (how does one measure how advanced technologies in different disciplines are, in order to compare them?).

The final chapter, though, was a highlight to me. There was a really good discussion of the role of taverns in the American Revolution. And in that discussion, Johnson acknowledges that it is difficult to establish a causal relationship (which made me again wonder why he was unconcerned about the challenges of causality between shopping and the industrial revolution earlier in the book). I really appreciated the discussion of the work of Jürgen Habermas, Ray Oldenburg, and the "third places" (places of gathering that are neither work, nor home). It reminded me of my wife's excellent PhD thesis on cafés.

Overall, I did enjoy the book in spite of my griping about the overall thesis and the way that Johnson sometimes draws conclusions from slim evidence. If you are interested in the history of leisure pursuits, I recommend it to you.