A qualitative assessment of the implications of AI for New Zealand

Back in July, Treasury released an analytical note written by Harry Nicholls and Udayan Mukherjee on the implications of artificial intelligence for New Zealand. I read it last month, but didn't have time to write up my comments until now due to travelling. The analytical note is not a quantitative assessment on the economic or labour market or other impacts, but really just testing the waters and providing some broad overview of some of the issues. Fortunately (or unfortunately, depending on your perspective), this hasn't been superseded by subsequent quantitative analysis of the impacts, so it still provides a starting point for some discussion.

Nicholls and Mukherjee see three main issues: (1) the impacts of AI on productivity and investment; (2) the impacts on employment and the labour market; and (3) the development of regulatory approaches for AI. It may be my bias showing, but I think they miss the impact of AI on education and human capital development as a fourth main issue, but perhaps they see it as being subsumed under their first main issue (although they don't mention education at all, and mention human capital only once in the note).

The note discusses each of the three main issues in turn, but first it lays out frameworks for thinking through each issue. There are two (competing) approaches. The first is based on macroeconomic growth accounting framework, where:

AI can be thought of as a form of capital deepening, improving the quantity and quality of capital inputs. It can also be thought of as labour-augmenting technology that improves the quality of labour inputs. And it might be a form of technology that more effectively combines capital and labour, improving multi-factor productivity.

The second is a microeconomic approach, which sees:

generative AI as a form of workplace automation...

There are three types of effects [Acemoglu and Restrepo] identify in this framework:

• Displacement Effect: Automation makes it more efficient to produce some tasks by capital that were previously done by labour, so it reduces the share of labour in production.

• Reinstatement Effect: Automation can allow a more flexible allocation of tasks in production, and so can create a range of new tasks in which labour has a comparative advantage.

• Productivity Effect: Automation allows some of the tasks previously performed by labour to now be performed more cheaply by capital, and so increases the value-added in production and so potentially the overall demand for labour.

The benefit of this framework is that it helps to think through how the net impact of new automation technologies can be thought about as the result of the combination of the strength of these effects.

For example, the total effect of AI on labour demand is a combination of the Productivity Effect increasing the demand for labour and the Displacement Effect replacing labour from tasks it previously performed. Similarly, the impact of AI on wages could be thought of as the net change on marginal productivity from the Reinstatement Effect creating new tasks for labour demand and the Displacement Effect create new labour supply from replaced tasks.

I found the frameworks quite useful, but it is difficult to see how they can be reconciled (which, to be fair, is a common problem comparing macroeconomic and microeconomic frameworks). Nicholls and Mukherjee then turn to thinking through the three issues, but the frameworks seem to be a bit lost in those sections. However, they conclude by offering some potential areas for future research, which are:

• A deeper exploration of the policy levers that might accelerate the diffusion of AI, reducing the lag that characterises New Zealand’s diffusion of new technology.

• Investigating the likely impacts of AI on the productivity and competitiveness of particular sectors, or small-to-medium sized enterprises (SMEs), given their significance to the New Zealand economy.

• Taking a closer look at the implications of AI for employment and labour markets, with a particular focus on how these impacts should inform our policy settings around skills, immigration, and labour markets.

• Examining how AI intersects with our economic security, particularly if AI development is concentrated in a small number of large multinational overseas-based technology companies.

• Considering how AI could lift the productivity of New Zealand’s public sector, in order to maintain or enhance service levels in the face of pressures like an aging population.

Again, I would add the impacts on education and human capital development here, as well as research on the distributional consequences of AI. Those questions are important to understanding the future labour market impacts, as well as the need for changes to policy settings on taxes and transfers. Anyway, this is a good starting point as a think piece, and it is good to see that Treasury are actively thinking in this space, and hopefully there is more careful analytical work to come from them.

[HT: Inside Government, back in July]

Enforced class attendance might make better students worse off

Each trimester, I make it very clear to students that regular attendance at classes offers greater chances at success for them. Those statements are based on years of observational data on student performance in my papers - students who attend more regularly have higher pass rates, and better grades. Part of that is clearly selection bias - students who attend less often may have lives that are more complicated, making it difficult for them to keep up with studying. Alternatively, students who attend less often may be less motivated, and that affects both attendance and willingness to study. However, just before the pandemic, we ran an experiment in the ECONS101 class that provided some evidence that incentivised non-attendees to attend class, and that did appear to improve performance, suggesting that at least some of the effect of attendance on performance is causal (that study has still not been written up for publication, sadly).

On the other hand, I am very aware of the large (and still growing) literature comparing online, blended, and in-person teaching and learning modes (see this post, and the long list of links at the end of it). My takeaway from that literature is that teaching mode has no effect on student performance (or student learning) on average, but has important heterogeneous effects. More able (or more self-directed) students tend to do better in online and blended learning environments, whereas less able (or less self-directed) students tend to do worse in those environments. Based on that evidence, I encourage in-person attendance as I am somewhat more focused on ensuring that the less able students have the best opportunity to succeed. However, in the back of my mind, I do harbour some reservations that the top students are not necessarily getting the best learning experience as a result.

Those reservations seem to now have some support, from this 2023 article by Sofoklis Goulas (Brookings Institution), Silvia Griselda (Bocconi University), and Rigissa Megalokonomou (Monash University), published in the Journal of Economic Behavior and Organization (ungated earlier version here). They look at a relaxation of attendance rules at Greek high schools, and the effects on attendance and student performance. Specifically:

The education system in Greece has a very strict attendance policy. All high- school students attend classes back to back with short recesses in between from 8 am to 2 pm, Monday through Friday...

Until the end of the 2005-06 school year, every student was allowed a maximum of 50 unexcused and 64 hours (about 10 days) of excused class absences in a year. One absence is equal to one missed school period...

Before the beginning of the 2006-07 school year, the Ministry of Education announced a nationwide policy to encourage students’ autonomy (Gov. Gazette 65/A/30-3-2006). The policy provided eligible students with 50 additional excused class absences. Every student who had received a raw GPA higher than 75% the year before was eligible to take up more parent- approved absences in the current year.

Goulas et al. then use a difference-in-differences-in-differences approach, which involves comparing the difference between students who were eligible (greater than 75% GPA the previous year) and those that were not eligible, between Grade 11 (no relaxed policy) and Grade 12 (relaxed policy), between the 2006 and 2007 years (when the policy was implemented). That allows them to estimate the causal impact of the change in attendance policy on student outcomes, including high-stakes exam performance (in subjects that count towards university entrance) and low-stakes exam performance (in subjects that do not count towards university entrance). Goulas et al. find that:

Targeted students in the treated cohort took more absences and improved their high-stakes exam performance as a result of the increased autonomy policy. In particular, targeted students increased their total (excused) absences by four (three) additional class hours—roughly 0.09 (0.13) standard deviations—relative to non-targeted students during the year the increased autonomy policy was in effect. Targeted students' high-stakes exam performance increased by 0.07 standard deviations—or 0.019 standard deviations per missed period—due to the increased autonomy policy. Targeted students' low- stakes exam performance remained unaffected by the increased autonomy policy.

So, the policy, which was targeted at top students, appeared to increase their class absences (which was what it was designed to do), and improved those students' performance. Goulas et al. also show that: 

...the increased autonomy policy is found to be associated with an increase in university admission score of 0.13 standard deviations... also... increased autonomy is associated with being admitted to university degree programs of higher quality/selectiveness—an improvement roughly equivalent to two percentiles in the distribution of degree quality.

So, the relaxation of attendance for top students also affects their long-term prospects in a positive way. However:

The policy is also associated with an increase of roughly 0.12 standard deviations in non-targeted students’ excused absences (roughly three additional excused class absences) in the year the increased autonomy policy was introduced. The effect of the policy on non-targeted students’ total absences or on school performance is not statistically significant. The unexcused absences of targeted and non-targeted students are unaffected by the increased autonomy policy.

So, other students also had more absences, but their performance was not affected. At least they weren't made worse off. Goulas et al. then dig a little deeper, showing that the change in absences was greatest among the top quartile of eligible students (in terms of GPA). However, that analysis doesn't show that those top-quartile students perform any better than other eligible students (in fact, the effects seem to be largest for the lowest quartile of eligible students).

What might be causing these results? Goulas et al. identify five potential mechanisms:

A first mechanism through which increased autonomy could influence performance is because of the attendance-performance association, which may also differ across students... A second mechanism through which increased autonomy could affect performance is related to the effective class size. As students take more class absences due to the increased autonomy policy, the class size decreases and student performance increases. A third mechanism may be related to changes in peer characteristics due to the increased autonomy policy. As targeted students skip class more often, the effective peer characteristics may change, impacting student performance... A fourth channel of influence of increased autonomy on performance is through teachers. As targeted students skip class more often because of the increased autonomy policy, instruction might become more effective... A fifth channel of influence of the increased autonomy policy on performance is related to student motivation from agency as suggested by the Self Determination Theory... Students targeted by the increased autonomy policy might feel they are being treated as responsible individuals, capable of making optimal decisions on their own, potentially feeling motivated to perform higher...

Goulas et al. then show that there is little evidence to support the effective class size, effective peer characteristics, and the instruction quality mechanisms, because there are similar impacts on eligible students in classes with higher or lower proportions of eligible students. That leaves the attendance-performance mechanism and greater agency. Goulas et al. show that controlling for decreased attendance explains most of the effect of the policy change on student outcomes, which suggests that it is the change in attendance that most explains the change in student performance.

What can we take away from this study? Class attendance may well make the top students worse off in terms of their academic performance. However, it is likely that what students do with the spare time does need to also be considered. This policy change was directed at the top 25 percent of Greek high school students, and they were well aware that they had a high-stakes university entrance examination at the end of the year. And, the additional absences needed to be approved by parents, who are likely to maintain at least some control over their children's activities during their absences. In other contexts, and with less motivated (or less closely supervised) students, the impacts are unlikely to be the same.

Also, I'd be cautious before extending these results to the university context. In my experience, students who are not attending are likely to be working (or playing video games) rather than studying during that freed up time. And past studies have shown that working impacts negatively on student performance (see this post, and the links at the bottom of it). However, the study does give some food for thought - maybe strict attendance norms can be relaxed for top students who are clearly engaged in other outside learning activities?

This week in research #52

I started this week at the Australia New Zealand Regional Science International (ANZRSAI) Conference in Canberra. Unfortunately, I left my notebook in one of the conference rooms at the end of the second day, so most of my notes were lost. However, from memory (and the camera roll on my phone) here are some of the highlights I found from the conference:

• Andrew Leigh (economist and Australian MP) gave a great keynote on inequality in Australia based on his revised book Battlers and Billionaires, although the most interesting bit was a graph on mentions of 'inequality' in Australian parliamentary debates, which is fairly low until a massive spike from 2011 onwards (I guess politicians didn't really care about inequality before then?)
• Putu Widyastaman presented on the relationship between gentrification and prostitution in Jakarta, showing that gentrification reduces prostitution in an urban village (roughly equivalent to a neighbourhood), but increases prostitution in surrounding urban villages
• John Madden presented on the economic impact of Victoria University's west Melbourne campus (although the counterfactual of 'what would have happened if Victoria University didn't have a campus in west Melbourne' is always going to be a challenge)
• Robert Tanton presented on a geocoded synthetic Census that is being developed for Australia, allowing researchers to conduct research on Census data without access to the underlying Census data (which is not a problem in New Zealand, where access to the Census data is available through the secure Integrated Data Infrastructure)

Aside from the conference, here's what caught my eye in research over the past week:

• Duan, Yuan, and Snyder find that a one standard deviation increase in the local sex ratio raises rural unmarried males’ likelihood of temporary migration in China by 3.6 percentage points
• Vatsa and Pino (open access) find that petrol price shocks had a slightly delayed but persistent effect on one-year inflation expectations in New Zealand, whereas five-year inflation expectations were largely insensitive to these shocks
• Meng et al. (with ungated earlier version here) find that a one-standard-deviation increase in wind turbines reduces bird abundance by 9.75% and leads to a 12.2% reduction in bird species richness at the county level in China
• Arora and Roy (with ungated earlier version here) do not find any significant differences in student evaluations of teaching received by female and male professors, using an experimental approach in India
• Abel et al. find that, in an experiment, investors are more likely to continue to follow financial advice from male professionals following advice that proves incorrect, but 47 percent less likely to follow financial advice from female professionals in the same circumstances
• Cordes, Dertwinkel-Kalt, and Werner (open access) investigate the economics of 'loot boxes' in online and mobile games using an experimental study design, and find that common design features of loot boxes (such as opaque odds and positively selected feedback) double the average willingness-to-pay for lotteries
• Klöcker and Daumann (open access) develop a theoretical model to explain success and dominance in international sports
• De Acutis, Weber, and Wurm (open access) conduct a meta-analysis of 51 studies on the effect of gender quotas on company boards, and find that (among other results) stock market returns are negatively affected by quota policies (I feel like there is more to this than meets the eye on a quick skim-read though)
• König et al. (open access) find that there is a statistically significant, but very small, positive association between short-term ambient temperature changes and individuals’ general willingness to take risks, using a large survey in Augsburg, Germany
• Lin, Churchill, and Ackermann (open access) find using 14 waves of HILDA data in Australia that a 1 percent increase in the proportion of a postcode that has access to the national broadband network (NBN) is associated with a 1.573 increase in Body Mass Index and a 6.6 percentage point increase in the probability of being obese
• Oprea challenges the existence of loss aversion by showing using experimental data that it may simply arise because of complexity of experimental tasks that are used to demonstrate loss aversion (which may also explain these results, I guess?)

This week in research #51

It's been a slow week again this week, as I've been travelling for the Australia New Zealand Regional Science International (ANZRSAI) Conference in Canberra (more on that next week). Nevertheless, here's what caught my eye in research over the past week:

• Borbely et al. (open access) find that a universal free school meal policy covering the first three grades of primary schools in Scotland had a small positive effect on attendance and a small negative effect on health-related absences (potentially important results given New Zealand's free school lunch policy and mooted changes)
• Asanov, Schirmacher, and Bühren (open access) conduct a meta-analysis of 'lost letter' experiments (78 studies with an overall sample size of 53,504 letters from 18 countries), and find that the return rate is lower for political or deviant issues, stamped letters are more likely to be returned, but letters with money are not more likely to be returned, and a high socio-economic environment increases the chances of the return