Generative AI and grade inflation

If generative AI can produce assessed work that earns higher marks than a student would earn by themselves, widespread use of generative AI should increase measured grades even in the absence of equivalent learning gains. In other words, we might expect grade inflation to occur as a result of students using generative AI, and that grade inflation to not reflect improved learning. To what extent is this generative AI-induced grade inflation occurring? That is the question addressed by this recent working paper by Igor Chirikov (UC Berkeley). 

Specifically, Chirikov looks at the change in the grade distribution across 319 courses (and over 500,000 enrolments) at "a large, selective public research university in Texas", covering the period from 2018 to 2025. He follows the labour economics literature by measuring 'task exposure' to AI, using the share of required tasks in each course's Fall 2022 syllabus that involved writing or coding - areas in which generative AI is particularly capable and could be used as a substitute for students' own efforts. Using a difference-in-differences (DID) approach, Chirikov compares the difference in the share of A grades and GPA between years before and including 2022 (when ChatGPT was released) and more recent years up to 2025, between courses more or less exposed to generative AI. He finds that:

...grades rose substantially in high-exposure courses after 2022: the share of A grades increased by 13 percentage points (about 30% relative to the 2022 baseline) and GPA by 0.12 points, accompanied by compression of the grade distribution.

Looking at the shares of other grades, Chirikov finds that:

The share of A- grades fell by 4 percentage points, the share of B+ grades by 3 percentage points, and the shares of B and below show smaller and mostly insignificant changes.

So, courses that were more exposed to generative AI have seen a greater increase in the share of A grades and a greater decrease in the share of lower grades than courses less exposed to generative AI. Chirikov then turns to exploring the mechanism that underlies the change, by extending the DID approach to also compare courses that place greater or lesser assessment weight on homework tasks (in what is called a 'triple differences' approach). In that analysis, he finds that:

...above-median homework courses show an additional 16 percentage point increase in the share of A grades relative to below-median courses with the same level of AI exposure. The effect on GPA follows a similar pattern, with an additional 0.13 point increase in high-homework courses, though this estimate is less precisely estimated...

In above-median homework courses, the share of other grades declines significantly with AI exposure relative to below-median homework courses.

So, the observed pattern of grade inflation, with grades near the top of the distribution shifting towards As to a greater extent in courses that have greater homework weight, provides strong evidence consistent with generative AI contributing to the observed grade inflation, principally by substituting for student effort on unsupervised assessment tasks.

We should be cautious about over-interpreting the results from this study. It is based on results from a single university, and it measures course-level task exposure to generative AI, rather than students' actual use of generative AI. Nevertheless, the results are consistent with what we would expect, and are likely to hold in other contexts.

Given that, how should universities adapt to solve this issue? The obvious response is to move to more invigilated, in-person assessment. However, Chirikov cautions that:

Not all skills can be meaningfully evaluated under exam conditions: the ability to produce a well-researched essay, develop a software project, or conduct an empirical analysis requires sustained engagement that timed in-person assessments cannot capture. Restricting assessment to formats that are AI-proof risks measuring a narrower set of capabilities than the ones courses are designed to develop, potentially undermining the learning goals that graded work is meant to serve. A more promising direction is to redesign assessments so that AI use is either structurally constrained by the task or purposefully incorporated into it, for example, by requiring students to document their process, justify their choices, or demonstrate understanding through follow-up interaction.

In my view, the optimal response for universities is a combination of invigilated in-person assessment and authentic assessments in which AI use is permitted (or even encouraged) and evaluated. The appropriate balance depends on the specific learning outcomes for each course. However, as I have argued before, one approach is to scaffold students through their studies, with lower-level courses that rely more on developing core knowledge, relying less on generative AI and therefore using more secure assessment, while higher-level courses increasingly incorporate generative AI use explicitly into the assessment. This also requires scaffolding students through learning how best to apply generative AI at each level.

It almost goes without saying that we cannot simply continue to assess students as we always have done. Generative AI has broken key elements of the assessment toolkit that we previously used. This is not a new observation. However, evidence that generative AI may be accelerating grade inflation makes it even more imperative that assessment practices are updated to better reflect the availability of generative AI.

Read more:

• Teaching evaluations and grade inflation
• Grade inflation is harming students' learning
• A few papers on grade inflation at universities
• Grade inflation and college completion in the U.S.
• More on teaching evaluations and grade inflation
• Grade inflation at New Zealand universities, and what can be done about it
• Who is morally responsible for grade inflation?

Book review: Econometrics for Dummies

I just finished reading Roberto Pedace's 2013 book Econometrics for Dummies. As you might expect from a book written as part of Wiley's 'Dummies' series, the book is written as a basic introduction to its topic, starting from the basics of probability distributions, and ending with a brief primer on how to conduct an econometrics research project, as well as common mistakes to avoid in applied econometrics.

The book mostly hits the mark as a good introduction. However, Pedace clearly has a high opinion of dummies, because he assumes a great deal of statistical understanding. The book also has a lot of mathematical formulae to negotiate. To be fair to Pedace, it would be difficult to teach econometrics without the formulae without turning it into a 'recipe book' of steps to follow that would not help readers to understand. Nevertheless, I feel like the book could have been pitched perhaps a little lower, as more of a stepping stone between basic statistics and a full introduction to econometrics.

Nevertheless, the book is well written and easy to follow. Pedace does use some unusual terminology though. I struggled with his reference to categorical variables as "qualitative variables". In my mind, qualitative is something quite different. The book is also a little repetitive at times, in part because Pedace has written it in a way where the reader need not necessarily linearly follow through each chapter, but instead can jump directly to the bits of most relevance without missing out on important details that are hidden earlier in the text.

The book may be an introduction for dummies, but Pedace certainly stretches the dummies. I really appreciated that it included a discussion of difference-in-differences (and regular readers of this blog will recognise that this is a research approach that is applied quite commonly in research papers). I also thought that Pedace gave the clearest description of the difference between fixed effects and random effects models for panel data, as well as the Hausman test. Although applications of econometrics to panel data are a feature of every econometrics textbook, in my mind most do not clearly explain these models. At least, not as well as Pedace does.

Finally, the book offers an online 'cheat sheet', although I was disappointed that this seemed to simply be a static webpage, and not really a sheet that can be downloaded or printed.

Overall, this book is a good accompaniment to a full econometrics textbook, or as a memory aid for those who did econometrics some years ago and want the simple details quickly. Or, for those who want an intermediate step between introductory statistics and a full econometrics book such as Mastering 'Metrics (which I reviewed here) or Mostly Harmless Econometrics (which I reviewed here).

Broadband coverage and rural fertility

I recently expressed some scepticism at a paper showing that the release of the iPhone explained a third or more of the decline in US fertility. So, I was interested to read this new working paper by Gokhan Kumpas (California State University, Los Angeles), which looks at the effect of broadband coverage on fertility. 

Specifically, Kumpas studies broadband expansion to rural areas through the USDA's Broadband Initiatives Program (BIP) and the National Telecommunications and Information Administration’s Broadband Technology Opportunities Program (BTOP), which accepted applications in 2009-10 for broadband expansion. Kumpas identifies counties where applications for the programmes were rejected, using a subset of those counties as a control group to compare with counties where applications were successful. Moreover, Kumpas limits the control group counties to those that most closely matched the pre-treatment trajectory of the treated counties in terms of population growth, in order to deal with any problems of mean reversion.

Kumpas compares fertility among teenage women (aged 15 to 29 years) between 2010-13 and 2018-19, leaving out the intervening years where broadband was being expanded. In his main specification, he finds that broadband rollout:

...reduced the rural teen birth rate by approximately 1.6 per 1,000 in the pre-COVID post period (CHR release years 2018–2019), or about 3 percent of the pre-period baseline of 48.6 per 1,000.

The teenage birth rate for the treated counties fell from 48.6 to 34.0 per 1,000, so based on these results the effect is equivalent to about 11 percent of the 14.6-point decline in the teenage birth rate. Notice that this is a much more modest estimated contribution to fertility decline than that in the iPhone and fertility paper. Moreover, Kumpas has a good theoretical basis for believing that broadband would affect fertility, based on the opportunity cost of fertility work of Kearney and Levine (see here, for example), which:

...predicts that any local shock that meaningfully expands the perceived economic or informational opportunity set young women face should depress teen fertility. Broadband Internet expands the perceived opportunity set through several channels.

First, broadband expands informational access to contraceptive methods, to family-planning service locations and scheduling, and to the comparative costs and consequences of different reproductive choices...

Second, broadband expands access to schooling and credentialing options beyond what is locally available, including online community-college coursework, remote tutoring and test preparation, financial-aid information, and credential-program advertising. The downstream consequence is to raise expected returns to schooling and to deferring family formation.

Third, broadband expands the labor-market opportunity set by making non-local jobs visible and (with telework) accessible.

Kumpas finds results consistent with the first two of these channels, with the effects concentrated in counties that had at least one Title X family planning clinic (with no statistically significant effect in counties without a clinic), and individual-level evidence that high school completion and college attendance increased in treated counties. However, there was no evidence for changes in the adult labour market. These results are suggestive that the contraception-access and education channels explain the results, although they are not definitive.

Now, the analysis relies critically on comparing counties covered by successful applications with those covered by unsuccessful applications. This rejected-applicant comparison is appealing, but it relies on the assumption that the treated and control counties would have followed similar trends in the absence of funding. Kumpas provides considerable evidence in support of that assumption, including by selecting control counties with similar pre-treatment population trends, but it cannot be tested conclusively. Funding decisions were based partly on project benefits, viability, and sustainability, and the applications would have included information about subscriber projections and local demographics. It is therefore possible that factors related to anticipated demographic change influenced both the likelihood of receiving funding and subsequent fertility trends. This is similar to the concern I raised about the iPhone and fertility paper.

Nevertheless, if we take the results at face value it appears that broadband access contributed to reduced teenage births in rural US counties. Let's not get carried away though - broadband explains only around 11 percent of the decline in the teenage birth rate. That is a meaningful proportion, but most of the decline in teenage births happened for other reasons. In other words, fertility would have declined substantially even without the contribution of broadband access.

Read more:

• The pandemic 'baby bust' in other countries
• Cohort effects and the downturn in US fertility since the Great Recession
• The new economics of fertility
• The economics of the falling total fertility rate in New Zealand
• The economics of fertility in high-income countries
• Can fertility return to replacement levels?
• You can make future population decline disappear just by changing the way you categorise people and fertility
• The modest impact of Australia's baby bonus on fertility timing
• My take on that iPhone-fertility paper

This week in research #134

Here's what caught my eye in research over the past week:

• Wu and Lee use a theoretical model to show that higher military spending stimulates aggregate demand, thereby promoting employment and income growth, particularly among low-income groups, and consequently reducing income inequality
• Wang (open access) finds using data from British Columbia that having larger proportions of female peers has a large positive effect on students' choice of a STEM major
• Clark and Nielsen (open access) conduct a meta-analysis on the returns to education, including 79 studies that use changes in the minimum school leaving age to identify effects, finding that the average return to an additional year of education is 8.2 percent
• Giuranno and Manni (open access) review the literature on the impact of TikTok on elections, concluding that TikTok functions as a fast-moving marketplace for political ideas in which algorithmic incentives may shape conditions relevant to electoral integrity
• Collischon and Zimmermann (open access) unsurprisingly find zero effects of western Zodiac signs on wages, education, and managerial status among workers in Germany
• Garcia et al. find that both social trust and civic pride contribute to the willingness to pay to prevent the relegation of a professional football (soccer) team, with social trust more relevant for those who attend games

Also, I am delighted to report that my Master's student, Josh McNamara, won the Jan Whitwell Prize for the best student research paper at the New Zealand Association of Economists conference last week. Josh has blossomed into a standout research student over the last couple of years, and will no doubt do great things in a PhD programme in the near future. Congratulations Josh!

Finally, I can also report a marker of my own mild fame. While in Scotland last week, I visited my ancestral family lands at Achnacarry in the western highlands, and the Clan Cameron Museum. My wife asked if they would add me to their list of famous Camerons, and after looking me up online (including my blog!), they agreed! So, I join the likes of former UK prime minister David Cameron, Pearl Jam drummer Matt Cameron, and others on the list.