PhD students' growing use of generative AI in STEM subjects

I see a large number of initial research proposals, sent to me as part of a cold-emailing strategy by prospective PhD students. The vast majority of those proposals are now obviously written by generative AI. For the most part, that makes the proposal largely worthless in terms of judging the quality of potential PhD students. If anything, it is a negative signal about their ability to conduct primary research. If they can't write a short initial proposal without significant help from generative AI, how will they cope with writing research papers and a thesis?

No doubt I am being too harsh in my approach. However, when you receive three or four cold emails per day from prospective PhD students, as I do about this time of year, you have to be a bit ruthless. I know some colleagues that delete those emails unread. I at least want to give these prospective students a chance to capture my attention.

My own experience is based on a sample of one. What does the evidence tell us about the growing use of generative AI by PhD students more generally? We receive an answer to that question in the context of STEM PhD students in this new NBER Working Paper (sorry, I don't see an ungated version online) by Daniel Gross (Duke University), Dror Shvadron (University of Toronto), and Hansen Zhang (Duke University). They collected data on US-trained PhD students from 2019 to May 2026, ran their dissertation abstracts through the AI-detection tool Pangram, and then collected data from each student's LinkedIn profile to determine their post-graduation outcomes. The growth in generative AI writing in dissertation abstracts is striking, as shown in Figure 1 from the paper:

Prior to 2023, almost no dissertation abstracts are flagged as having AI content. By the first five months of 2026, 29.4 percent of abstracts contain AI writing (at least, according to Pangram). Looking across fields, AI adoption has been most rapid in civil engineering and computer science (both around 50 percent in 2026), and lowest in mathematics and statistics (17 percent in 2026).

Gross et al. then dig into the characteristics of the PhD students using generative AI in their dissertation abstracts, finding that:

Non-native English-speaking students use AI far more intensively than native speakers... Among 2026 graduates, 41% of dissertations by students educated in non-English-speaking countries have AI-generated abstracts, against 23% for students from English-speaking countries...

AI use also increases at institutions with lower research intensity and programs of lower rank. In the 2026 cohort, 28% of dissertations at R1 universities have AI writing, compared with 31% at R2 universities and 37% at other doctoral institutions, and this gradient is particularly strong among US-origin students (foreign-origin students write with AI at high rates at all institutions...).

Gross et al. then look at where students go after receiving their PhD, finding that:

Among US-origin graduates, those with AI-written dissertations are 7.4 p.p. less likely to be employed in academia in 2026 (17% over the 44% base rate) and 8.0 p.p. more likely to be employed in industry (16% over a 50% base), with no difference in government employment. These differences attenuate substantially for likely foreign-origin graduates. We also find differences within academia: conditional on academic employment, graduates with AI-written dissertations are 5.4 p.p. more likely to hold non-tenure-track positions (on a base of 12%), with no significant difference in the likelihood of tenure-track or postdoctoral positions.

Gross et al. explicitly caution against taking a causal interpretation of the effect of generative AI use on student outcomes. I didn't interpret the results that way, instead seeing them as perhaps indicating that students with dissertation abstracts written by generative AI may be lower-quality students, or those with less of an academic focus for their future employment. Alternatively, some students may simply be using AI to overcome language barriers, or to polish their writing, rather than to substitute for their own research skills. However, the results don't definitively establish either of those interpretations either. 

What the results do show is more banal - PhD students are increasingly using generative AI. Of course, it is important to keep in mind that these results tell us about the prevalence of AI-generated writing in dissertation abstracts, rather than the dissertation more generally. Since generative AI is quite good at summarising, and that is essentially the purpose of an abstract, I would expect AI-generated writing to be more prevalent in abstracts than in the main text of the dissertations.

Finally, while these results apply to STEM subjects, it seems likely that this is indicative of a more general trend across all fields. If this trend continues, it will not be long before AI-assisted dissertation writing becomes the norm rather than the exception. What we should be wondering now is whether that trend will hamper PhD students in their development as researchers, or not.

[HT: Marginal Revolution]

Read more:

• Lessons from Joshua Gans on AI for economics research
• This is not how generative AI should be used in research

Generative AI and the future of the university

Universities and other educational institutions have had a few years now to come to grips with generative AI and its implications. For universities, what I have seen is a lot of angst about student cheating and the impacts on learning, courses set up to take advantage of the demand for students with AI skills, but mostly a lot of tinkering around the edges and hoping for the best.

A new working paper by Jason Potts (Alfaisal University) outlines the fundamental challenge to the entire business model of universities. Potts notes that universities are a very complex multi-sided platform market, matching at least fifteen disparate groups including, but not limited to, students, employers, and government.

Potts also notes that what universities offer to students is not content at all, but a 'warrant' - a signal to employers of the student's quality (see my post here for more on that point). This overstates the case of course. Universities do offer students the opportunity to develop important knowledge and skills. Even Bryan Caplan, in his book The Case Against Education (which I reviewed here), acknowledges that education develops human capital, despite arguing that most of the financial return to education comes from signalling.

Now, the challenge is that generative AI makes content delivery a commodity, so Potts argues that universities should concentrate more on providing the warrant of quality, essentially becoming a certifying organisation that provides a signal of the quality of students to future employers. He notes that:

The strategic form this implies is a university that examines and certifies as its primary business, and that treats instruction as a service which may be produced internally, purchased, or left to the student. Almost no one working inside a university right now wants this. But it is the logical direction this new wave of technologies is pushing the university.

I think Potts is right that no one wants this outcome. But he is also right that the value that universities provide to students will be increasingly embedded within the quality of the signal. And it is there that generative AI presents another challenge. Effective signals need to be able to separate high-quality and low-quality students, and one of the ways that occurs is by education being more costly (more difficult or time-intensive, rather than necessarily more financially costly) for lower quality students. Generative AI lowers the cost of producing assessment work that is of high quality, meaning that low-quality students are more difficult to distinguish from high-quality students. The signal value of education is reduced, unless universities make changes to preserve the value of the signal. On that point, Potts argues to:

...move the assessment to a modality where the production cost remains type-dependent. At this stage, the best candidate is probably a return to the viva, common in earlier times as an observed, embodied, high-bandwidth, unscripted examination conducted as an oral defense, or as live problem-solving before an examiner... This is expensive per student and is complementary with cheap AI-delivered instruction.

I'm not convinced that oral examination is the only way to achieve this aim. Properly invigilated in-person written exams are similarly resistant to unauthorised AI use, and are considerably more efficient than oral examinations. However, they are not appropriate to all courses, and may best suit courses where basic foundational knowledge needs to be assessed (although noting that it is possible to assess higher-level skills in a written exam as well). Potts notes that, in terms of university revenue, this means:

...open entry, examination-only pathways to a degree, and pricing on the assessment rather than on tuition or contact hours.

So, in Potts' view, universities wouldn't necessarily do any teaching, only assessing. Or rather, they might do teaching, but it would be complementary to their main activity of assessing students and certifying their knowledge and skills. 

There is a positive element to this for some university academic staff though. If you think through the incentives here, it becomes clear that the endpoint of this transition could easily be for universities to become diploma mills, offering certifications to paying students that bear no meaningful relation to students' actual learning, knowledge, or skills. Universities can only avoid that outcome if they can credibly demonstrate to employers that the credentialling process is not influenced by student demands. And so, academic staff retain some academic freedom:

Tenure exists so that an examiner can fail a candidate, reject a result, or refuse a certification without personal consequence. It is therefore the certifying body that must inherit the collegiate form (i.e. peer judgement, protected tenure, academic freedom) while the instructional arm can be run on ordinary managerial lines.

I'm not convinced that the market will discipline universities to the extent that Potts believes. Diploma mills already coexist alongside credible institutions, demonstrating that there is demand for credentials of questionable value, although some employers may already recognise those differences. The more serious problem is that employers may not recognise gradual changes in assessment standards and the quality of certification. Universities might slowly drift towards satisfying student demand for credentialling, rather than learning, while continuing to benefit from reputations built up over many years. By the time employers recognise that the qualifications no longer signal what they once did, their value may already be substantially less than it was.

Complementary to credentialling, Potts notes that the decline in junior positions at many employers opens an opportunity for universities. If employers are unwilling to invest in students to train them in what were previously junior roles, universities could step into that gap. In part, universities have already started moving in that direction - witness the rise of 'work-integrated learning', which is now a feature of many universities' offerings, and increasingly compulsory for students (including at Waikato). 

Potts notes that the challenge here will be to find a financing mechanism. However, to me there is an obvious solution. Traditionally, junior roles have bridged the gap between university study and the point at which employers see graduates as sufficiently experienced and skilled to be fully productive. Employers have effectively funded that transition through on-the-job training. If employers are increasingly unwilling to employ graduates in these junior roles, universities could provide that bridge instead, through postgraduate qualifications involving work-integrated learning, either with an employer or in a 'simulated' work environment. Employers may not be willing to pay for this (after all, they are increasingly unwilling to employ juniors), but students may, especially if it smooths their pathway into employment. Again, this might not be something that we like, but it may be the reality in the future.

What about research? Potts has some interesting things to say about that as well, in particular noting that the minimum efficient scale for research is declining and fast approaching one. What would previously have required a large lab team or research group, can increasingly be undertaken by a sufficiently motivated researcher working with agentic AI. Of course, Potts is overstating the case here, because there remains a lot of research that relies on expensive infrastructure, specialised equipment, labour-intensive fieldwork, or collaboration among researchers. However, Potts' view suggests a fundamentally different relationship between research-intensive academic staff and universities. Specifically, Potts proposes that universities:

...unbundle affiliation from employment, making the university more like a club... Rather than the binary of tenured staff or nothing, universities can offer graded affiliation over institutional standing, grant-holding capacity, facility access, and the right to publish under institutional imprimatur, priced as a service and subject to quality review. This converts a fixed cost into a revenue line, retains scholars the institution could not otherwise afford to employ, and extends the warrant into the market for independent research.

This may work for high-profile researchers who can maintain an ongoing flow of external research grant money. However, most researchers are not in that privileged position. Also, Potts ignores that research funding is under pressure, and his proposal would make researchers even more beholden to grant funders than they are now. And it would create an even starker division among academic staff, between those who are highly-research active, bringing in substantial external research funding, and maintaining independence from the institution they are currently affiliated with, and those who are reliant on a role that simply certifies students. This problem is going to affect disciplines to different extents. For example, in a research environment like New Zealand at the moment, where funding changes have severely undercut research funding in humanities and the social sciences, this would be particularly problematic.

Potts also notes that the type of research that universities should specialise in should change:

AI has commoditized knowledge and particularly its search and discovery, including in domains that were previously high in expertise and difficulty such as literature review, statistical analysis and theorem proving. The implication is that the comparative advantage of universities has moved further afield, into the harder to conceptualize and reach corners of new ideas and possibility.

However, that proposal really relies on the willingness of research funders to fund frontier research. That might work in the US, but New Zealand's current research funding environment is increasingly emphasising economic benefits rather than fundamental research. That may make it more difficult to obtain funding for speculative frontier research, where economic benefits are uncertain.

Potts' working paper has a lot of other proposals for universities. However, I think the most important ideas are those I have outlined above. Potts is clear that not all universities are equally affected by generative AI. In particular, he notes that the elite universities are somewhat insulated from changes by their reputations. However, the vast majority of universities are not elite institutions. Potts' vision of universities is radically different from the status quo. We might buy into his vision, or not. But regardless, the economics of higher education are changing, and universities will need to adapt. Whether that requires moving away from the traditional model that combines teaching, credentialling, and research, remains to be seen.

Read more:

• Some notes on generative AI in higher education
• Some notes on generative AI and assessment (in higher education)
• More notes on generative AI and teaching (in higher education)
• Universities' (and teachers') cheap talk on generative AI and assessment
• Simas Kucinskas on AI, university education, and the 'mushy middle'
• The wicked problem of generative AI and assessment
• Matthew Kahn on generative AI and the economics major
• David Deming on generative AI and commitment to learning, and the impact of generative AI on signalling in education

Will AI make personalised pricing a reality, and is that really a bad thing?

Personalised pricing (or first-degree price discrimination) occurs when the seller sells their good or service to every consumer at a different price, as I described in this 2023 post. If executed perfectly, the seller could extract all of the consumer surplus as profits, by charging a price to every consumer that is exactly equal to the maximum the consumer is willing to pay. Fortunately for consumers, such perfect personalised pricing has remained a theoretical possibility.

But technological tools are increasingly helping firms to learn more detailed information about consumer preferences, and that allows firms to home in on consumers' maximum willingness-to-pay. The latest worry for consumers is AI, as this article in The Conversation by Patrick Dodd and Hanoku Bathula (both University of Auckland) notes:

Digital platforms can observe thousands of individual decisions. A ride-hailing platform can see which jobs a driver accepts, when they work and which incentives bring them online. A retailer can see purchases, abandoned carts and responses to discounts.

There is no strong evidence major companies already know everyone’s precise financial breaking point. But algorithmically mediated pay, personalised worker incentives, discounts and consumer offers are already real.

Notice that Dodd and Bathula also take the logic of personalised pricing for consumers, and apply it to gig-economy workers as well. Platforms such as Uber or Lyft or Doordash can increasingly use what they know about their delivery workers' preferences to determine their minimum willingness-to-accept for each delivery. The target is different (minimising how much they pay to the delivery worker, rather than maximising the price they charge the consumer), but the underlying premise of personalised pricing is the same.

Algorithms have been around for a while, though. Dodd and Bathula do not clearly lay out why they think that recent developments in AI make personalised pricing more of a reality than before. Most of what they say about 'algorithms' applies equally to statistical algorithms that have been around for years (decades, even) as to more recent developments in AI and machine learning (AI/ML). So, let me extend their argument more explicitly.

AI/ML dramatically lowers the cost of estimating individual willingness-to-pay. It can combine huge numbers of relatively weak signals about a particular consumer, learn complex patterns from the behaviour of millions of other similar consumers, experiment continually with prices and discounts, and update its estimate each time that circumstances change. That gives firms far richer models from which to estimate each particular customer's maximum willingness-to-pay (or, for their workers, to estimate their minimum willingness-to-accept). Moreover, while older statistical algorithms allowed firms to segment customers into fairly coarse categories, AI/ML allows firms to make predictions for each individual, and in real time. The better estimates from these newer models therefore allow firms to price much closer to the perfectly price-discriminating ideal. It's still not completely perfect, but it is a further improvement on what they were previously able to achieve.

Dodd and Bathula finish their article by noting the unfairness of personalised pricing. Their argument is essentially that there is asymmetry in the relationship between consumers and firms. Firms using algorithms (and now AI/ML) know increasingly more about what consumers are willing to pay, but consumers know very little about what firms are willing to accept.

However, it is worth unpacking that a bit more. Firms that don't know consumer willingness-to-pay can't raise their prices without limit, as consumers with low willingness-to-pay would stop buying from them. In practice though, personalised pricing will never be perfect. Firms may charge higher prices to consumers that they estimate have high willingness-to-pay, while offering lower prices or discounts to consumers with lower willingness-to-pay. So, relative to offering the same price to everyone, personalised pricing need not make every consumer worse off. The high-willingness-to-pay consumers are likely to be worse off, but some low-willingness-to-pay consumers may actually be better off.

Now consider which types of consumers tend to have high willingness-to-pay, and which types tend to have low willingness-to-pay. For many goods, lower-income consumers are likely, on average, to have lower willingness-to-pay, so personalised pricing could result in some of them being offered lower prices. That won't always be true though. Some lower-income consumers with few alternatives or an urgent need may have high willingness-to-pay despite having a low income. Taken together, this means that the distributional effects of personalised pricing are not necessarily straightforward. However, in some instances preventing firms from price discriminating could be making low-income consumers worse off. With that in mind, is it really fairer that firms are not allowed to offer lower prices to consumers with low willingness-to-pay?

I'm not really trying to defend price discrimination here. I'm not keen on personalised pricing for very selfish reasons - I don't want to pay more, even if I am willing to pay more! And like me, most consumers should probably not be keen on personalised pricing. But before we rail against the evils of firms price discriminating, we need to properly consider its distributional consequences. And that means thinking about which groups may be made better off by price discrimination, not just which groups are made worse off.

Read more:

• Surge pricing is coming to a supermarket near you
• What would happen if perfect personalised pricing was possible?
• Personalised pricing is now a step closer with electronic shelf labels

How important is the apprenticeship model to entering a research career?

Those of us working in research careers can invariably share stories about working as a research assistant, cleaning datasets, coding, running models, doing literature reviews, and many of the other less-glamorous tasks that make up the research process. That was a key aspect of our apprenticeship into the world of research, and a gateway into a research career. But how important really is research assistance as an entry point into a research career?

That is the question addressed in this 2025 NBER Working Paper (ungated version here) by Ina Ganguli (University of Massachusetts, Amherst) and Raviv Murciano-Goroff (Boston University). They look at the impact of working in a university lab (an important subset of research assistance work) on subsequently pursuing a scientific career. Interestingly, Ganguli and Murciano-Goroff use changes in the local minimum wage as an exogenous source of variation in lab employment, so this paper also indirectly contributes to the literature on the employment effects of the minimum wage, in a context (research labs) that is not often the focus of that literature.

Their data comes from UMETRICS, which collates data on research grants across universities, and their dataset covers 32 universities over the period from 2000 to 2019 (although the dataset has coverage up to 2022, including those additional years would mean having to account for the COVID-19 pandemic).

First, using a dataset collated at the lab level, Ganguli and Murciano-Goroff use a staggered difference-in-differences approach to look at the impact of minimum wage changes on employment of undergraduates in the labs. This analysis essentially compares the change in undergraduate employment between the time before and the time after an increase in the minimum wage, between university labs that were affected by the minimum wage increase and those that were not. In that analysis, they find that:

...following minimum wage increases, labs decrease the employment of undergraduates by 7.4% on average...

So, not dissimilar to the literature on minimum wage effects on employment, when focused on young people in exposed occupations. Ganguli and Murciano-Goroff then use the minimum wage change as an instrument for students' exposure to laboratory research as an undergraduate. The key assumption is that minimum wages while an undergraduate affect later scientific careers only through their effect on lab employment opportunities.

Using a dataset of over 28,000 undergraduates and their subsequent career paths, Ganguli and Murciano-Goroff find that:

...decreased exposure to scientific work translates into significantly lower rates of undergraduate research assistants pursuing doctoral degrees or working in the life sciences sector after graduation. We find that working one fewer quarter in a lab during an undergrad student’s college years translates into between a 7.0 and 10.3 percentage point decrease in the rate of enrolling in a doctoral-level program. Given our sample of 28,283 students, this implies that if all students had experienced a minimum wage increase, roughly 500 fewer undergraduates in our sample would have pursued these advanced degrees.

So, the results imply that one fewer quarter of lab experience as an undergraduate reduces enrolment in a doctoral programme by 7.0 to 10.3 percentage points. That is a fairly large effect and should make graduate research programmes take notice of the importance of undergraduate research assistance opportunities for the pipeline into graduate research.

So, working in a lab as an undergraduate is a key pathway towards a career in the life sciences, and when those opportunities are restricted, students are less likely to embark on such a career. These results won't be too surprising to those of us who have been through an apprenticeship as a researcher. It is likely that I would be doing something very different right now if I hadn't been pulled into a lot of research projects towards the end of my undergraduate studies. I almost certainly wouldn't have pursued a PhD, or become an academic. Research assistant jobs are more than just jobs. They provide mentoring, information, skills, networks, references, and a chance to try out being a researcher in a safe setting.

These results also highlight two other things for me. First, they suggest that minimum wage increases may have a negative impact on the career pipeline into science. I'm unsure that this negative impact of the minimum wage has been identified before. However, we should be cautious because Ganguli and Murciano-Goroff are primarily interested in using minimum wage changes to identify the effect of undergraduate research experience on later careers, rather than estimating the overall long-run consequences of minimum wage increases for the scientific workforce. Nevertheless, their results suggest that this may be an important unintended consequence of higher minimum wages, and one that would be worth further research.

Second, given that research assistants do a lot of the 'drudge work' in research, if generative AI is also able to do a lot of that work, that further suggests a negative impact on the career pipeline into science through the rise of generative AI. I know others have written on this before (see here), so the challenge here is not a new idea.

On the plus side, that suggests another method that could be used to further test for the impacts of undergraduate (and graduate) research assistance on future academic careers, beyond focusing on life sciences (as Ganguli and Murciano-Goroff do). By comparing academic fields that are more (or less) exposed to generative AI (especially in the early days of generative AI), we might be able to tease out how important research assistance is to future academic careers across many fields.

Research apprenticeship is important, and these results give us a clear indication of how important it can be. Cleaning data, running models, searching the literature, and doing all the other seemingly mundane tasks of a research assistant are not just cheap ways for senior researchers to get research done. They are also how the next generation of researchers learns what research is, discovers whether they enjoy doing it, and gets started on a research career. If these opportunities are reduced, whether through higher minimum wages or through substitution by generative AI, we may save on some of the drudge work today, but at the cost of having fewer researchers tomorrow.

[HT: Marginal Revolution]

Will generative AI mean the end of rational ignorance?

In this Substack post back in March, Andy Hall made the case for generative AI to create 'political superintelligence':

The more I work with and study AI, the more I believe it can give every human being on the planet access to a sort of political superintelligence, if we shape it right. And that intelligence, in turn, can make governments smarter and more effective, representatives more faithful, and institutions more responsive than anything we’ve built in over 2,000 years of experimenting with democracy.

Hall's post is worth reading in its entirety, but I want to explore a related point - will generative AI mean the end of rational ignorance for voters? Rational ignorance is the idea that it may be better for voters to not know what decisions policymakers are making on their behalf. That's because it's costly (in terms of time and effort) for voters to keep track of how the decisions that policymakers (and politicians) make on their behalf will affect them (economists call those monitoring costs). The benefit that a voter would receive by becoming informed of what policymakers (and politicians) are doing is relatively small, because their ability to change an election (and therefore policy) is very small. When the monitoring costs are greater than the benefits of being better informed, then voters would be better off not paying the monitoring costs. That is, voters would be better off not paying attention to what the policymakers (and politicians) are doing - the voters would be better off remaining rationally ignorant. This theory of rational ignorance was introduced in the 1950s by the late economist Anthony Downs.

Where does generative AI fit into this? Generative AI could meaningfully lower the monitoring costs for voters, as it gives the opportunity for voters to ask for quick summaries of policy proposals that may affect them. This will be even more effective as generative AI understands more about users' preferences. Moreover, agentic AI offers voters even greater opportunity to investigate what policymakers (and politicians) are doing, at relatively low cost.

When the monitoring costs decrease, then the rationale for voters to remain rationally ignorant weakens. We might expect voters to become more engaged with what the government is doing on their behalf, and to be more active in engaging with government to make their preferences known. Or, at least, maybe voters will delegate these activities to their favourite agentic AI model.

There are, of course, some reasons for caution. Generative AI might reduce the cost of obtaining political information without reducing the cost of checking whether that information is accurate or unbiased. Moreover, an overly sycophantic generative AI that knows the voter's preferences might reinforce the voter's existing views rather than challenging them. So, perhaps generative AI simply moves the monitoring costs from monitoring the government to monitoring the generative AI?

Hall makes the point that political superintelligence has the potential to increase the quality of governance. If generative AI enables voters to become better informed at low cost, it could strengthen political accountability. Policymakers (and politicians) who know that voters can easily scrutinise their decisions may be less willing to act against voters’ interests, or may face greater consequences when they do.

We may not have political superintelligence yet, and large numbers of voters may still be rationally ignorant. However, it may not be long before we start to see some substantive changes in the political process, driven in part by the emergence of generative AI.

[HT: Marginal Revolution for the Andy Hall post]

Generative AI and grade inflation

If generative AI can produce assessed work that earns higher marks than a student would earn by themselves, widespread use of generative AI should increase measured grades even in the absence of equivalent learning gains. In other words, we might expect grade inflation to occur as a result of students using generative AI, and that grade inflation to not reflect improved learning. To what extent is this generative AI-induced grade inflation occurring? That is the question addressed by this recent working paper by Igor Chirikov (UC Berkeley). 

Specifically, Chirikov looks at the change in the grade distribution across 319 courses (and over 500,000 enrolments) at "a large, selective public research university in Texas", covering the period from 2018 to 2025. He follows the labour economics literature by measuring 'task exposure' to AI, using the share of required tasks in each course's Fall 2022 syllabus that involved writing or coding - areas in which generative AI is particularly capable and could be used as a substitute for students' own efforts. Using a difference-in-differences (DID) approach, Chirikov compares the difference in the share of A grades and GPA between years before and including 2022 (when ChatGPT was released) and more recent years up to 2025, between courses more or less exposed to generative AI. He finds that:

...grades rose substantially in high-exposure courses after 2022: the share of A grades increased by 13 percentage points (about 30% relative to the 2022 baseline) and GPA by 0.12 points, accompanied by compression of the grade distribution.

Looking at the shares of other grades, Chirikov finds that:

The share of A- grades fell by 4 percentage points, the share of B+ grades by 3 percentage points, and the shares of B and below show smaller and mostly insignificant changes.

So, courses that were more exposed to generative AI have seen a greater increase in the share of A grades and a greater decrease in the share of lower grades than courses less exposed to generative AI. Chirikov then turns to exploring the mechanism that underlies the change, by extending the DID approach to also compare courses that place greater or lesser assessment weight on homework tasks (in what is called a 'triple differences' approach). In that analysis, he finds that:

...above-median homework courses show an additional 16 percentage point increase in the share of A grades relative to below-median courses with the same level of AI exposure. The effect on GPA follows a similar pattern, with an additional 0.13 point increase in high-homework courses, though this estimate is less precisely estimated...

In above-median homework courses, the share of other grades declines significantly with AI exposure relative to below-median homework courses.

So, the observed pattern of grade inflation, with grades near the top of the distribution shifting towards As to a greater extent in courses that have greater homework weight, provides strong evidence consistent with generative AI contributing to the observed grade inflation, principally by substituting for student effort on unsupervised assessment tasks.

We should be cautious about over-interpreting the results from this study. It is based on results from a single university, and it measures course-level task exposure to generative AI, rather than students' actual use of generative AI. Nevertheless, the results are consistent with what we would expect, and are likely to hold in other contexts.

Given that, how should universities adapt to solve this issue? The obvious response is to move to more invigilated, in-person assessment. However, Chirikov cautions that:

Not all skills can be meaningfully evaluated under exam conditions: the ability to produce a well-researched essay, develop a software project, or conduct an empirical analysis requires sustained engagement that timed in-person assessments cannot capture. Restricting assessment to formats that are AI-proof risks measuring a narrower set of capabilities than the ones courses are designed to develop, potentially undermining the learning goals that graded work is meant to serve. A more promising direction is to redesign assessments so that AI use is either structurally constrained by the task or purposefully incorporated into it, for example, by requiring students to document their process, justify their choices, or demonstrate understanding through follow-up interaction.

In my view, the optimal response for universities is a combination of invigilated in-person assessment and authentic assessments in which AI use is permitted (or even encouraged) and evaluated. The appropriate balance depends on the specific learning outcomes for each course. However, as I have argued before, one approach is to scaffold students through their studies, with lower-level courses that rely more on developing core knowledge, relying less on generative AI and therefore using more secure assessment, while higher-level courses increasingly incorporate generative AI use explicitly into the assessment. This also requires scaffolding students through learning how best to apply generative AI at each level.

It almost goes without saying that we cannot simply continue to assess students as we always have done. Generative AI has broken key elements of the assessment toolkit that we previously used. This is not a new observation. However, evidence that generative AI may be accelerating grade inflation makes it even more imperative that assessment practices are updated to better reflect the availability of generative AI.

[HT: The Dangerous Economist]

Read more:

• Teaching evaluations and grade inflation
• Grade inflation is harming students' learning
• A few papers on grade inflation at universities
• Grade inflation and college completion in the U.S.
• More on teaching evaluations and grade inflation
• Grade inflation at New Zealand universities, and what can be done about it
• Who is morally responsible for grade inflation?

Is it working from home, and not generative AI, that is harming the prospects of young workers?

There is growing evidence that the labour market for young workers is challenging. Graduates are finding it more difficult to get jobs after graduation. Several research papers have noted that generative AI may be to blame (see this post, for example), with one research paper referring to the changes in the labour market as seniority-biased technological change (see this post).

But the challenge with trying to attribute changes in the labour market to the rise of generative AI is that there are other contemporaneous changes affecting the labour market as well. One of those changes is the rise of working from home (as I noted in yesterday's post). Working from home may reduce the prospects for junior workers in part because it costs more to supervise and monitor them when they are working from home. Junior workers also benefit from on-the-job learning when they work with other people, and that on-the-job learning is less effective when they work from home. Combining those two effects, working from home reduces the incentive for employers to hire junior workers.

This new working paper by Peter Lambert (University of Warwick) and Yannick Schindler (Ellison Institute of Technology, Oxford) tries to disentangle the effects of generative AI and working from home on employment of younger workers. They use data from Revelio Labs that is made up of monthly matched employer-employee records collected from résumés (predominantly from LinkedIn) to construct a measure of the junior share of all new hires. They also use data from Lightcast on the near-universe of online job postings across thousands of online job sites and other websites. They use the Lightcast data to construct a measure of the share of job postings that require three or fewer years of experience. Their data from both sources covers the period from 2017 to 2025, and includes four countries: the US, the UK, Canada, and Australia.

Lambert and Schindler then use that data, along with measures of 'exposure to generative AI' and 'exposure to working from home' at the occupation level, in a difference-in-differences strategy. That means that they essentially compare the change in the share of junior job hires (or job postings) between occupations that are more or less exposed to generative AI (or working from home). Their main results are neatly summarised in Figure 3 from the paper:

Panel (a) shows that the junior share of new hires decreases significantly in jobs that are more exposed to working from home, from 2023 onwards (the black line). When they also control for exposure to generative AI (the red line), the effect of working from home barely changes. In contrast, Panel (b) shows that the junior share of new hires also decreases significantly in jobs that are more exposed to generative AI, from 2023 onwards (the black line). However, when they also control for exposure to working from home (the blue line), the effect of generative AI becomes much smaller and statistically insignificant. The results are similar for the share of job postings requiring three or fewer years' experience, as shown in Panels (c) and (d) of the figure.

The size of the effects are quite large too. A one-standard-deviation increase in exposure to working from home reduces the junior share of new hires by about two percentage points, and the share of job postings requiring three or fewer years' experience by 1.5 percentage points.

Lambert and Schindler conclude that, based on their results, working from home is a better predictor of the decline in junior hiring than generative AI. Given potential benefits of working from home, they are reluctant to recommend policies against working from home, instead noting that:

...micro-level adjustments may be required to help firms adapt their organizational practices, so as to enjoy the benefits of WFH [work from home] arrangements while simultaneously managing the development of early-career talent.

Seen alongside the negative mental health impacts of working from home (as noted in yesterday's post), this should give us further pause for thought. However, it is worth noting that even if working from home is a better predictor of reductions in junior hiring than generative AI within their model, that doesn't let generative AI off the hook entirely. Since both trends are happening at the same time, reducing working from home might not eliminate the negative impacts on junior hiring, but instead make generative AI appear more important as an explanation. Lambert and Schindler note early in their paper that it is often the same occupations (white-collar occupations) that are most exposed to both working from home and generative AI. Given that, perhaps Lambert and Schindler's recommendation for micro-level changes in organisational practice may be the best mitigation strategy available to us.

[HT: Marginal Revolution]

Read more:

• ChatGPT and the labour market
• More on ChatGPT and the labour market
• The impact of generative AI on contact centre work
• Some good news for human accountants in the face of generative AI
• Good news, bad news, and students' views about the impact of ChatGPT on their labour market outcomes
• Swiss workers are worried about the risk of automation
• Generative AI and entry-level employment
• Generative AI and seniority-biased technological change

Does the future of higher education look more like a mentoring pyramid scheme?

In response to my recent post about the future of higher education and one-on-one mentoring, one of my students from last year, Yunze, got in touch via email to offer a potential solution:

...I wonder whether it is possible to set a clear academic threshold within each discipline. If students who reach this threshold could mentor upper‑middle‑level students, while professors spend only a small amount of time supervising the overall direction, the system might become more sustainable. However, I suspect this could harm the interests of the top students, since they might otherwise use that time to further advance their own academic achievements, and If [sic] they fail to successfully train students with real research ability, it would likely damage both the university’s reputation and the professor’s own reputation.

You know, I think Yunze is right on the money here. Consider the problems I outlined in the earlier post: (1) the signalling value of education is falling due to generative AI; (2) a one-on-one mentoring approach may be a solution; but (3) one-on-one mentoring doesn't scale due to limited faculty time. If one-on-one mentoring is not conducted between faculty and students, but works more like a pyramid mentoring model, then this might actually work, not just for students, but for faculty and for universities as well.

So, let's think it through. But first, remember that the mentoring model I introduced in the earlier post is not simply a model of small classes, where senior students perform limited teaching roles, such as tutoring. This is a model of genuine mentoring, where the mentor encourages the mentee to become a builder, in the words of Auren Hoffman. A builder creates things, and it is the act of building, and the learning alongside that, which will be a durable signal to future employers. In relation to mentoring, I said in that post that mentors should do the following for their mentees:

Teach them to be builders. Encourage them to create things. Work with them and chart a path forward for their success.

If faculty provide one-on-one mentoring to a small number of senior students, then that makes better use of faculty time than them mentoring hundreds of first-year students. The senior students can then each mentor several second-year students, who in turn can then mentor several first-year students. [*] In this model, faculty time is targeted at the senior students, where the impact of faculty on student employment outcomes may be greatest.

Students benefit from helping junior colleagues to become builders, where the signalling value may remain even in the face of generative AI. Even better, mentoring provides student mentors with an opportunity to build - they may be able to point employers to the success of their mentees as an example of their building, talking also about what went wrong in the mentoring relationship, and what they learned from the experience.

In this mentoring pyramid model, universities retain a key role, but that role becomes very different. Universities essentially become a platform, connecting students with mentors - first-year students with second-year mentors, second-year students with senior student mentors, and senior students with faculty mentors. In the terms of my earlier post, the university runs their own OnlyStudents platform.

Of course, this platform role creates a new problem for universities. If mentoring works mainly as a way of matching students with mentors, then the market may not need eight OnlyStudents platforms in New Zealand, or thousands of OnlyStudents platforms worldwide. A small number of large platforms could have a big advantage in that case - more students attract more mentors, more mentors improve the quality of matching, and better matching attracts still more students. Those network effects could create a winner-take-all dynamic, in which universities would struggle to differentiate themselves simply by running their own mentoring platforms, and where a single surviving OnlyStudents platform might be the ultimate outcome. However, that conclusion depends on the strength of the network effects. If effective mentoring also depends on institutional trust, disciplinary reputation, local employer connections, pastoral care, or an in-person community, then universities may retain some defensible advantages. Geography alone probably won’t be enough, especially if online mentoring is close to being as effective as in-person mentoring, but local connections might still matter. So the question for universities is not just whether they can build their own OnlyStudents, but whether they can attach that platform to something that a larger, more generic OnlyStudents cannot easily replicate.

Universities may also retain a role in the initial and ongoing training of mentors. Since each student, and each faculty member, will need to be a mentor to one or more others lower down in the pyramid, they will need to understand how to mentor. That means universities would not simply be matching students with mentors. They would also need to train mentors, monitor the quality of mentoring relationships, and intervene when mentor-mentee relationships are not working well. Moreover, the adoption of a mentoring pyramid model is likely going to change who the most successful students (and faculty members) are. The top students do not necessarily make the best mentors (or the best tutors, as I have learnt across years of coordinating tutors in my first-year papers). Good mentoring requires a specific skill set, but it is those skills that may also demonstrate the quality of the student as a builder - a signal of high quality for employers.

A further point about the pyramid mentoring model is that it likely requires a strong filtering effect to be financially viable. Since each faculty member can only mentor a limited number of senior students, and each of those senior students can only mentor a limited number of second-year students, who in turn can only mentor a limited number of first-year students, each level of the pyramid probably needs to be somewhat wider than the levels above it. To achieve that, student progression needs a strong filter, limiting the number of students who progress from first-year to second-year, and from second-year to senior.

Let's consider some simple numerical examples that illustrate why filtering is needed. If each faculty member mentors five senior students, and each senior student mentors five second-year students, who each mentor five first-year students, then the pyramid contains 155 students per faculty member - five senior students, 25 second-year students, and 125 first-year students. A model where each faculty member’s salary is covered by fees from 155 students, setting aside any contribution to central university costs, seems likely to be financially viable to me. However, in this model only one-fifth of students could be allowed to progress each year. That means the model would also need some form of orderly exit for students who are filtered out - perhaps an exit qualification, or a pathway into a non-mentored track. The problem is that both options may provide negative signals about the student who is filtered out.

If all students were to progress, then that would require each student to mentor at most one student at the level below. Keeping five senior students mentored by faculty, then the pyramid would contain 15 students per faculty member - five senior students, five second-year students, and five first-year students. That system would be much less likely to cover the cost of faculty time. So, it's unlikely that the pyramid mentoring model would be viable to run without some form of filtering - perhaps not as extreme as only one-fifth of students progressing each year, but clearly not all students could progress every year.

So, to return to my conclusion from the previous post, the current mass higher education model still looks increasingly fragile, but perhaps one or a few universities might be able to navigate their way through. However, the survivors are likely to be first-movers or fast followers in developing a platform market strategy that leverages a pyramid mentoring model. This model is still going to cost students a lot, and the filtering effect would make higher education more elitist as well.

And thanks to Yunze for inspiring this post with his perceptive email comments.

*****

[*] For simplicity, I'm assuming a three-year higher education degree structure, as we have in New Zealand. For a four-year degree structure, you would of course need to add an additional level.

Read more:

• Why all students need to understand adverse selection and signalling
• Signalling theory suggests that we may need two categories of university degree
• Does the future of higher education look more like one-on-one mentoring?

Does the future of higher education look more like one-on-one mentoring?

It almost seems a cliche to say that generative AI is both the greatest threat, and the greatest opportunity, for higher education. That doesn't make the statement any less true. And there are many commentators who are trying to work out what happens next for higher education. I think one of the best is Hollis Robbins, whose Substack Anecdotal Value is well worth subscribing to.

Robbins was recently interviewed by Jay Caspian Kang of The New Yorker (paywalled, but you can find an ungated version here) about the future of higher education. The whole interview is worth reading, but I want to highlight this bit in particular, where Robbins says:

I was in Austin, Texas, a couple of times in March with a bunch of twenty-five-year-old billionaires. This is what they’re looking at. Instead of having the credential from the institution, why not have the credential from the professor? If you have a Hollis Robbins education, what would that signal? What would that credential mean as opposed to a degree from a university? There was some conversation about what that would look like, and one guy at the end of the dinner said, “Instead of OnlyFans, it’s like OnlyProfessors.”

The correct analogy here would be that it would be OnlyStudents, not OnlyProfessors (since the name contains the audience, not the performer). However, Robbins makes a good point. Higher education is, in part, an exercise in signalling (as I've noted before here and here). In fact, Bryan Caplan argued in his book The Case Against Education (which I reviewed here) that one third of the benefit of higher education is signalling (the other two-thirds is made up of genuine learning, socialisation, and transferable skills). 

The diploma that a student receives at the end of their higher education journey is a signal to employers of the student's quality as a future employee. The signal is credible to the employer because it is costly for the student to obtain, and costly in such a way that low-quality students wouldn't attempt the signal. University reputation matters here, because it is an assurance of the second of those conditions - low-quality students wouldn't attempt the signal of a Harvard degree, firstly because they wouldn't gain admission to Harvard in the first place. So, part of the signal comes from getting into Harvard. Second, low-quality students wouldn't attempt the signal of a Harvard degree because passing courses at Harvard is hard (or, at least, harder than at many other universities). The quality of the education at Harvard has traditionally been higher than elsewhere.

In the interview, Robbins makes a further important point, which is that we've spent the last several decades making higher education a commodity. Students studying a degree in a particular subject learn the same things, often using the same teaching materials, the same textbook, and the same style of assessment, regardless of which university they go to. That means that the quality of signal that arises from the education itself has reduced over time, meaning that most of the value of the signal arises from the admission process. Once a student has been admitted to Harvard, their signal is in place, and the further signalling from their education is lower than for comparable students in years past.

Robbins argues that generative AI is accelerating and expanding this commodification of education. Since generative AI has access, through its training corpus, to a large store of human knowledge, to add value over and above generative AI a professor must be a true expert in a very specific subfield. For students, learning a subject from a true expert still retains value, because generative AI cannot as easily replicate the learning that would occur from the expert. At that point, the university becomes less important as a mediator of education.

In other words, Robbins is arguing that the university could be disintermediated, with individual professors issuing credentials instead of universities. Who needs a Harvard degree, when you could have a Hollis Robbins degree? And since Robbins is an expert (in African American sonnet tradition, she says), the signal retains high value. However, that is true only to the extent that employers find the Hollis Robbins degree a credible signal.

That brings me to this Substack post by Auren Hoffman. Hoffman also argues that generative AI has diminished the value of the higher education signal. However, Hoffman argues for a different solution, this time from the perspective of the graduating student:

you have to show you can add value. that is it. that is the only thing.

the test the smart hiring manager applies in 2026 is simple. can you learn something on your own? can you finish what you started? can you do what you said you would do? these were always the skills that mattered. the difference is they are now the ONLY skills that matter, because the credential stopped doing the screening.

if you have a few years of experience, your resume can show you have these skills. if you are a new grad, you have to show what you have created and built.

Hoffman is overstating things a little, as university degrees are unlikely to disappear overnight. However, his critique is still important, as is the implication he draws. Hoffman argues that graduates (or young people, generally, since the solution doesn't depend on a student attending a university or completing a degree) need to become builders:

the most valuable thing a 22 year old can do in 2026 is create something. an app. a screenplay. a side business. an internal tool you wrote for a club you were in. a dinner series. a script that automates something annoying. a website. a chrome extension. a sculpture. a dance party. a discord bot. a substack with 32 readers and a real point of view. anything that moved from idea to working.

will the thing make money? probably not. that is not the point.

the point is that you taught yourself something. you finished it. you can describe what you learned, what broke, what you fixed, why you made the calls you made. that story is the new resume.

every hiring manager would rather interview a 22 year old with a launched app and a github full of weird side projects than a 22 year old with a 3.9 GPA from a top 50 school. it is not close. when one candidate has tangible evidence of what they can ship and the other has a transcript, the transcript will lose every time.

According to Hoffman, the best signal for young people to be sending in future is that they can build. Our students will ultimately be more successful if they can show off their skills (both technical and transferable skills) by building something. That is a signal that is costly, and costly in such a way that low-quality students will not attempt it. The signal is credible, and for the most part it retains currency even in the face of generative AI. Indeed, if the student builds something while effectively leveraging generative AI, then the signalling value to employers may be even greater. The key distinction is between the student using generative AI as a tool and the student using it as a substitute for doing the work. A student who can explain what they built, what broke, what they learned, and why they made the choices they made, is still sending a costly signal. A student who simply lets generative AI build for them is not. Employers would likely see through the latter pretty quickly.

Hoffman's idea isn't exactly new. When I think about some of my best students over the past two decades, they tend to be those that built something either while they were studying, or immediately after. For some, this was their own small business or entrepreneurial activity. For others, it was writing and publishing a research paper. Those were challenging tasks that set them apart from other students - a clear signal of quality. What Hoffman is essentially saying is that, with the signal from higher education itself being removed, the only remaining signal of quality is the signal from being a builder.

Where does that leave higher education staff? I think we can combine Robbins's and Hoffman's ideas, and chart a path forward. We don't need to start an OnlyStudents, and issue our own degrees, but we do need to cultivate closer relationships with our best students. Teach them to be builders. Encourage them to create things. Work with them and chart a path forward for their success. In other words, be a mentor.

Universities are absolutely going to hate this. Mentoring is not an activity that can be offered at scale. For example, there are simply not enough hours in the day for me to individually mentor all of the 350-plus students in my first-year economics class. Nor is mentoring easy to timetable, measure, standardise, or reward under current academic workload models. Universities are built around papers, credit points, learning outcomes, assessment rubrics, and student evaluations, all of which can be offered at scale. One-on-one mentoring doesn't work so well in that system. For example, postgraduate supervision already sits awkwardly within the system, with it being unclear whether it counts as teaching, or research. Nevertheless, mentoring may be one of the few ways that higher education can continue to offer something that is both valuable and difficult for generative AI to replicate.

If generative AI significantly reduces the signalling value of university education, and if students increasingly use it to avoid genuine learning, then the current mass higher education model looks increasingly fragile. Moreover, what remains is going to cost students a lot more. If having a high-quality mentor who can encourage a student to build is the path to their future employment, then it may be worth it. After all, high-quality signals are costly. That aspect of education, at least, won't have changed.

[HT: Marginal Revolution, for both the Hollis Robbins interview and the Auren Hoffman post]

Read more:

• Why all students need to understand adverse selection and signalling
• Signalling theory suggests that we may need two categories of university degree

Evidence that artificial intelligence is increasing the impact, but narrowing the scope, of research

There is growing evidence of positive impacts of generative artificial intelligence on productivity. This includes productivity in research (see this post, for example), including my own. However, some have questioned whether increasing research productivity comes at a cost of narrowing the scope of research.

So, I was interested to read this article by Qianyue Hao (Tsinghua University) and co-authors, published in the prestigious journal Nature (ungated earlier version here) late last year. They look at the impact of AI tools (not limited to generative AI) on the productivity of researchers and the quality of research. Specifically, they look at authors publishing in six representative fields: biology, medicine, chemistry, physics, materials science, and geology, across three 'eras': (1) the 'machine learning era ' (from 1980 to 2014), the 'deep learning era' (from 2015 to 2022), and the 'generative AI era' (from 2023 onwards). Hao et al. compare authors who publish 'AI augmented papers' with those who do not. An 'AI augmented paper' is one that uses methods such as:

...support vector machines and principal component analysis from the machine learning era, and convolutional neural networks and generative adversarial networks from the deep learning era. Large language models, which have emerged in recent years, also rank among the most frequently used methods...

Using a dataset that includes over 27 million papers with complete records that were published between 1980 and 2025, of which about 310,000 were 'AI augmented', Hao et al. find that:

...annual citations to AI papers are 98.70% higher than those to non-AI papers on average...

So, AI augmented research gathers more citations, which suggests that authors using AI in their research achieve greater impact. This is reinforced by evidence that AI augmented papers are published in higher quality journals (with Q1 journals being the highest ranked). Hao et al. report that:

...the proportion of AI papers in Q1 journals is 18.60% higher than that of non-AI papers in all journals; in Q2 journals, the AI proportion is 1.59% higher; whereas Q3 and Q4 journals hold a relatively lower proportion of papers with AI... These results indicate a heterogeneous distribution of AI-augmented papers across journals, with a higher prevalence in high-impact journals.

And AI appears to make authors more productive, as:

On average, researchers adopting AI annually publish 3.02 times more papers... and garner 4.84 times more citations... than those not adopting AI, with consistency.

All of these results seem to hold across all of the disciplines that Hao et al. consider. However, it is not all good news. Hao et al. use machine learning to create a measure of the 'breadth of scholarly attention'. Using that measure, they find that:

Compared with conventional research, AI research is associated with a 4.63% contracted median collective knowledge extent across science, which is consistent across all six disciplines... Moreover, when dividing these disciplines into more than two hundred sub-fields, the contraction of knowledge extent can be observed in more than 70% of them...

Of course, some of the differences here may be due to selection, as the types of researchers, and the types of research, involving AI use may be meaningfully different from those that don't. However, putting the selection issues aside, Hao et al. note that there is a tension between the individual researcher's incentive to produce a greater quantity of research that has higher impact, which would suggest greater use of AI, and the social incentive to produce a greater breadth of research.

So, the takeaway from this paper is that we need to consider researcher incentives, not just productivity. Specifically, this research suggests that the use of AI in research is leading to a 'prisoners' dilemma' outcome: each individual researcher acting in their own best interests (and using AI in their research) leads to an outcome that is worse for society overall (less breadth of research and more incremental gains).

Hao et al. conclude that:

The substantial academic benefits of AI use may be a driving force behind its accelerated rate of adoption; however, we also find unintended consequences from the increased prevalence of AI-augmented research. In all fields, AI-augmented research focuses on a narrower scope of scientific topics and reduces the scientific engagement of follow-on research, leading to more overlapping research work that slows the expansion of knowledge. Further, with a greater concentration of collective attention to the same AI papers, the adoption of AI seems to induce authors to converge on the same solutions to known problems rather than create new ones.

So, what is the solution here? Society probably wants research to be higher quality and have a broad scope. But individual researchers' incentives to use AI in their research appears inconsistent with that outcome. The traditional prisoners' dilemma is a repeated game (see here or here, for example), and the players of that game can avoid the worst outcome by cooperating. In this case, the researchers could cooperate by agreeing not to use AI in their research. The problem is that every researcher has an incentive to cheat on that agreement, since if they use AI, then that will be good for their career. This prisoners' dilemma is more difficult to ensure cooperation in than the traditional game, because there are not just two players who need to cooperate, but thousands (or millions). Ensuring cooperation in a prisoners' dilemma game with many players, each of whom is far better off cheating than cooperating, is almost impossible (which is why solving the problem of climate change is so difficult).

My own view is that the answer is not to keep AI out of research. That is not realistic, in the same way that it's not realistic to expect students not to use generative AI. The incentives need to be redesigned, but this will be no easy task. As long as universities, research funders, and publishers reward researchers for quantity, citations, and publication in top-ranked outlets, then we should expect more AI-augmented work, with a narrower scope than society might prefer. If we want AI to expand knowledge rather than simply accelerate competition within narrow foci, then we need institutions that also reward novelty, breadth, and the discovery of new questions. That is the economic challenge we must face up to.

[HT: Marginal Revolution]

This is not how generative AI should be used in research

I've been using ChatGPT Pro to help with drafting research papers this year, as I noted that I would do in this post from January. It has amped up my productivity a lot, allowing me to finish writing up two papers already, with a third on the way. These were papers where the analysis was already done, but it was the writing that was holding up the process. Having ChatGPT to help with the drafting seems to kickstart my writing, even though I have ended up extensively re-writing everything that ChatGPT produces. I find it a good disciplining tool as much as anything. Several colleagues have asked whether I am disclosing my generative AI use to journal editors when I submit. And I do. I have a standard 'generative AI use statement' that I include in my papers, that notes how it was used, and that I remain responsible for all of the content. You can see an example in this recent working paper.

However, not everyone is as careful with their generative AI use, or as transparent. Consider this example:

That is both infuriating and a sad indictment of the reviewing, editing, and publishing process, not least because, as on Reddit commenter noted, many authors see high-quality work rejected by journals, whereas a paper like this, with obvious flaws, has successfully been published. And it's not an isolated incident. This 2025 article by Artur Strzelecki (University of Economics in Katowice), published in the journal Learned Publishing (open access), catalogues over 1300 instances of likely unacknowledged and frankly stupid use of ChatGPT, up to September 2024.

Strzelecki's approach is to search for text strings that are almost certainly ChatGPT responses to a prompt asking it to generate text. The main example Strzelecki uses, which is in the title of the article, is "as of my last knowledge update". No human author is going to say that in a research paper. Similarly, "as an AI language model", "I don't have access to", and "certainly, here is" are highly indicative of ChatGPT use. There are circumstances where a human might use those phrases in a research paper, but it seems unlikely. Strzelecki screens out papers that mention ChatGPT, and manually checks each paper to ensure the text was not in some way legitimate, and that leaves 1362 articles.

How do these articles get published with this content intact? There are lots of stopping points where this could be caught and corrected (or prevented), but these articles have gotten through all of them. Strzelecki outlines the process. First, perhaps it is only one of the authors (and not all of them) that used ChatGPT. In which case, why didn't the other co-authors pick it up? Next, the paper is submitted to a journal, and often goes through a text review by the publisher. And then the editor or editors (including associate editors) looks at it, and decides whether it should be sent out for peer review. And then the peer reviewers (usually more than one, sometimes four or more) look at the paper in detail and provide comments. Then the editor receives the review reports and makes a decision. The paper may go through more than one round of review and editorial decision. And then, once accepted for publication, the article may be copy-edited. And at any of those stages, this text could be picked up. And yet, for over 1300 articles as of September 2024, the ChatGPT-generated text has not been picked up.

Strzelecki particularly focuses on 89 articles that have been published in journals indexed by Scopus or Web of Science, which should be the most credible journals. Of these:

...as many as 28 of them are in journals with Scopus percentile values of 90 and above. Two journals have a 99th percentile, indicating that they are the top journals in their field...

In total, 64 articles were found in journals considered to be in Q1, top quartile, recognized as the group of the best journals in their respective fields. Twenty-five articles are in the percentile range between 50 and 75, indicating that the journals in which these articles are found belong to Q2.

So, this phenomenon is not limited to low-ranked 'predatory' journals. In fact, looking at the list, there are several journals published by MDPI and Frontiers (for more on those publishers, see here). However, there are a whole lot published by Elsevier and Springer, publishers that we should expect much better of. Although, those are also publishers that publish a lot of journals, and a lot of articles, so perhaps that accounts for their higher numbers within the 89 articles that Strzelecki focuses on. Fortunately, I don't see any reputable journals in economics in the list, but I could be wrong.

Anyway, the takeaway is not so much that generative AI use is widespread in the write-up of research. It is that authors are using generative AI, not being transparent in their use of it, and that the quality control system by journals, even high-ranking journals, is terrible. Strzelecki makes a good point in the conclusion of his article that 89 out of over 2.5 million articles indexed in Scopus is only 0.000035% of the total indexed articles. However, this analysis is only picking up the really, really obvious cases. There will be far more use of generative AI that has not been adequately checked or acknowledged by authors, and not picked up in quality control.

I'm not against using generative AI in the write-up of research. Obviously, because I am doing the same thing. What needs to happen is that researchers need to be transparent and honest when they use generative AI, so that editors, reviewers, and the readers of research can see how it was used. That way, the users of research can evaluate for themselves whether they should believe, discount, or discard research depending on the ways and the extent of generative AI use. Without transparency, that important evaluation step is lost.

[HT: Artur Strzelecki]

Read more:

• Lessons from Joshua Gans on AI for economics research

Did employers value an AI-related qualification in 2021?

Many universities are rapidly adapting to education in the age of generative AI by trying to develop AI skills in their students. There is an assumption that employers want graduates with AI skills across all disciplines, but is there evidence to support that? This recent discussion paper by Teo Firpo (Humboldt-Universität zu Berlin), Lukas Niemann (Tanso Technologies), and Anastasia Danilov (Humboldt-Universität zu Berlin) provides an early answer. I say it's an early answer because their data come from 2021, before the wave of generative AI innovation that became ubiquitous following the release of ChatGPT at the end of 2022. The research also focuses on AI-related qualifications, rather than the more general AI skills, but it's a start.

Firpo et al. conduct a correspondence experiment, where they:

...sent 1,185 applications to open vacancies identified on major UK online job platforms... including Indeed.co.uk, Monster.co.uk, and Reed.co.uk. We restrict applications to entry-level positions requiring at most one year of professional experience, and exclude postings that demand rare or highly specialized skills...

Each identified job posting is randomly assigned to one of two experimental conditions: a "treatment group", which receives a résumé that includes additional AI-related qualifications and a "control group", which receives an otherwise identical résumé without mentioning such qualifications.

Correspondence experiments are relatively common in the labour economics literature (see here, for example), and involve the researcher making job applications with CVs (and sometimes cover letters) that differ in known characteristics. In this case, the applications differed by whether the CV included an AI-related qualification or not. Firpo et al. then focus on differences in callback rates, and they differentiate between 'strict callbacks' (invitations to interview), and 'broad callbacks' (any positive employer response, including requests for further information). Comparing callback rates between CVs with and without AI-related qualifications, they find:

...no statistically significant difference between treatment and control groups for either outcome measure...

However, when they disaggregate their results by job function, they find that:

In both Marketing and Engineering, résumés listing AI-related qualifications receive higher callback rates compared to those in the control group. In Marketing, strict callback rates are 16.00% for AI résumés compared to 7.00% for the control group (p-value = 0.075...), while broad callback rates are 24.00% versus 12.00% (p-value = 0.043...). In Engineering, strict callback rates are 10.00% for AI résumés compared to 4.00% for the control group (p-value = 0.163...), while broad callback rates are 20.00% versus 8.00% (p-value = 0.024...).

For the other job functions (Finance, HR, IT, and Logistics) there was no statistically significant effect of AI qualifications on either measure of callback rates. Firpo et al. then estimate a regression model and show that:

...including AI-related qualifications increases the probability of receiving an interview invitation for marketing roles by approximately 9 percentage points and a broader callback by 12 percentage points. Similarly, the interaction between the treatment dummy and the Engineering job function dummy in the LPM models is positive and statistically significant, but only for broad callbacks. AI-related qualifications increase the probability of a broad callback by at least 11 percentage points...

The results from the econometric model are only weakly statistically significant, but they are fairly large in size. However, I wouldn't over-interpret them because of the multiple-comparison problem (around five percent of results would show up as statistically significant just by chance). At best, the evidence that employers valued AI-related qualifications in 2021 is pretty limited, based on this research.

Firpo et al. were worried that employers might not have noticed the AI qualifications in the CVs, so they conducted an online survey of over 700 professionals with hiring experience and domain knowledge, but that survey instead shows that the AI-related qualification was salient and a signal of greater technical skills, but lower social skills. These conflicting signals are interesting, and suggestive that employers are looking for both technical skills and social skills in entry-level applicants. Does this, alongside the earlier results for different job functions, imply that technical skills are weighted more heavily than social skills for Engineering and Marketing jobs? I could believe that for Engineering, but for Marketing I have my doubts, because interpersonal skills are likely to be important in Marketing. Again though, it's probably best not to over-interpret the results.

Firpo et al. conclude that:

...our findings challenge the assumption that AI-related qualifications unambiguously enhance employability in early-career recruitment. While such skills might be valued in abstract or strategic terms, they do not automatically translate into interview opportunities, at least not in the entry-level labor market in job functions such as HR, Finance, Marketing, Engineering, IT and Logistics.

Of course, these results need to be considered in the context of their time. In 2021, AI-related skills might not have been much in demand by employers. That is unlikely to hold true now, given that generative AI use has become so widespread. It would be interesting to see what a more up-to-date correspondence experiment would find.

[HT: Marginal Revolution]

Read more:

ChatGPT and the labour market

More on ChatGPT and the labour market

The impact of generative AI on contact centre work

Some good news for human accountants in the face of generative AI

Good news, bad news, and students' views about the impact of ChatGPT on their labour market outcomes

Swiss workers are worried about the risk of automation

How people use ChatGPT, for work and not

Generative AI and entry-level employment

Survey evidence on the labour market impacts of generative AI

Who on earth has been using generative AI?

Who are the world's generative AI users? That is the question addressed in this recent article by Yan Liu and He Wang (both World Bank), published in the journal World Development (ungated earlier version here). They use website traffic data from Semrush, alongside Google Trends data, to document worldwide generative AI use up to March 2024 (so, it's a bit dated now, as this is a fast-moving area, but it does provide an interesting snapshot up to that point). In particular, Liu and Wang focus on geographical heterogeneity in generative AI use (measured as visits to generative AI websites, predominantly, or in some of their analyses, entirely ChatGPT), and they explore how that relates to country-level differences in institutions, infrastructure, and other variables.

Some of the results are fairly banal, such as the rapid increase in website traffic to AI chatbot websites, a corresponding decline in traffic to sites such as Google, and Stack Overflow, and that the users skew younger, more educated, and male. Those demographic differences will likely become less dramatic over time as user numbers increase. However, the geographic differences are important and could be more persistent. Liu and Wang show that:

As of March 2024, the top five economies for ChatGPT traffic are the US, India, Brazil, the Philippines, and Indonesia. The US share of ChatGPT traffic dropped from 70 % to 25 % within one month of ChatGPT’s debut. Middle-income economies now contribute over 50 % of traffic, showing disproportionately high adoption of generative AI relative to their GDP, electricity consumption, and search engine traffic. Low-income economies, however, represent less than 1 % of global ChatGPT traffic.

So, as of 2024, most generative AI use was in middle-income countries, but remember that those are also high-population countries (like India). Generative AI users are disproportionately from high-income countries once income and internet use (proxied by search engine traffic) are accounted for. Figure 12 in the paper illustrates this nicely, showing generative AI use, measured as visits per internet user:

Notice that the darker-coloured countries, where a higher proportion of internet users used ChatGPT, are predominantly in North America, western Europe, and Australia and New Zealand. On that measure, Liu and Wang rank New Zealand 20th (compared with Singapore first, and Australia eighth). There are a few interesting outliers like Suriname (sixth) and Panama (17th), but the vast majority of the top twenty countries are high-income countries.

What accounts for generative AI use at the country level? Using a cross-country panel regression model, Liu and Wang find that:

Higher income levels, a higher share of youth population, bet-ter digital infrastructure, and stronger human capital are key predictors of higher generative AI uptake. Services’ share of GDP and English fluency are strongly associated with higher chatbot usage.

Now, those results simply demonstrate correlation, and are not causal. And website traffic could be biased due to use of VPNs, etc., not to mention that it doesn't account very well for traffic from China or Russia (and Liu and Wang are very upfront about that limitation). Nevertheless, it does provide a bit more information about how countries with high generative AI use differ from those with low generative AI use. Generative AI has the potential to level the playing field somewhat for lower-productivity workers, and lower-income countries. However, that can only happen if lower-income countries access generative AI. And it appears as if, up to March 2024 at least, they are instead falling behind. As Liu and Wang conclude, any catch-up potential from generative AI:

...depends on further development as well as targeted policy interventions to improve digital infrastructure, language accessibility, and foundational skills.

To be fair, that sounds like a general prescription for development policy in any case.

Read more:

• How people use ChatGPT, for work and not