Is it better to publish more papers, or longer papers?

Academics get rewarded for their research output. More papers, published in higher-ranked journals, is a large part of the metric against which academics are judged. Notice that there are two aspects to that metric though - quantity (more papers) and quality (higher-ranked journals). Since it usually requires more effort to successfully publish a paper in a higher-ranked journal, is there a trade-off between quantity and quality of publications?

This 2014 article by my colleague John Gibson, published in the journal Economics Letters (sorry, I don't see an ungated version online), asks a related question about how we measure the quantity of research output - is it better to publish one long paper, or several papers that in total have the same number of pages? To answer that question, Gibson links the 2010 salaries of 223 economists from the University of California system to their publication histories (in terms of the number of publications, the quality of journals they were published in, and the number of publication pages), while controlling for seniority, experience, gender, campus, and whether the economist held a named chair or was a Nobel Prize winner.

Gibson notes that there is a wide range of journal ranking and weighting schemes, so he applies nine different ones. He finds that:

Six of the nine equations show a significant positive effect on salary from publishing more articles, conditional on the total number of quality-adjusted pages. The size of the effect is not trivial; on average, a standard deviation increase in the count of articles holding pages constant increases salary by 15 log points.

Ok, so publishing more articles is associated with a higher salary. And the difference is not small - 15 log points is equivalent to a salary increase of about 16 percent. What about more pages? On this, Gibson finds that:

...if an econometric ‘horse race’ is run between total pages and counts of articles, Vuong non-nested tests favor the model using total pages as being closer to the truth, even in the regressions where counts and pages are both statistically significant.

In other words, the total number of pages published seems to be a better predictor of academic salary than the total number of articles published, even though the number of articles also matters when holding the number of pages constant. How much does this difference matter? Gibson does some back-of-the-envelope calculations, finding that:

The predicted salary with a [Journal of the European Economic Association] article added to lifetime output is $195,980. Adding just as many pages to the literature with three Economics Letters articles increases predicted salary to $196,970—an annual gain over the single-article strategy of $990. This average economist is 16 years into their career, so with a 40-year career and five percent discount rate, the net present value of publishing three Economics Letters rather than one JEEA is $13,700.

Of course, these results are correlations rather than causal estimates, so we can't conclude that splitting a long paper into several shorter ones would necessarily increase an economist's salary. Nevertheless, the results imply that not only does quantity of research seem to matter, but how that quantity is broken up matters too. So, having lots of small articles published may lead to a higher salary for academic economists. That conclusion aside, one of the best aspects of this paper, you may have noticed, is that Gibson published it in... Economics Letters.

PhD students' growing use of generative AI in STEM subjects

I see a large number of initial research proposals, sent to me as part of a cold-emailing strategy by prospective PhD students. The vast majority of those proposals are now obviously written by generative AI. For the most part, that makes the proposal largely worthless in terms of judging the quality of potential PhD students. If anything, it is a negative signal about their ability to conduct primary research. If they can't write a short initial proposal without significant help from generative AI, how will they cope with writing research papers and a thesis?

No doubt I am being too harsh in my approach. However, when you receive three or four cold emails per day from prospective PhD students, as I do about this time of year, you have to be a bit ruthless. I know some colleagues that delete those emails unread. I at least want to give these prospective students a chance to capture my attention.

My own experience is based on a sample of one. What does the evidence tell us about the growing use of generative AI by PhD students more generally? We receive an answer to that question in the context of STEM PhD students in this new NBER Working Paper (sorry, I don't see an ungated version online) by Daniel Gross (Duke University), Dror Shvadron (University of Toronto), and Hansen Zhang (Duke University). They collected data on US-trained PhD students from 2019 to May 2026, ran their dissertation abstracts through the AI-detection tool Pangram, and then collected data from each student's LinkedIn profile to determine their post-graduation outcomes. The growth in generative AI writing in dissertation abstracts is striking, as shown in Figure 1 from the paper:

Prior to 2023, almost no dissertation abstracts are flagged as having AI content. By the first five months of 2026, 29.4 percent of abstracts contain AI writing (at least, according to Pangram). Looking across fields, AI adoption has been most rapid in civil engineering and computer science (both around 50 percent in 2026), and lowest in mathematics and statistics (17 percent in 2026).

Gross et al. then dig into the characteristics of the PhD students using generative AI in their dissertation abstracts, finding that:

Non-native English-speaking students use AI far more intensively than native speakers... Among 2026 graduates, 41% of dissertations by students educated in non-English-speaking countries have AI-generated abstracts, against 23% for students from English-speaking countries...

AI use also increases at institutions with lower research intensity and programs of lower rank. In the 2026 cohort, 28% of dissertations at R1 universities have AI writing, compared with 31% at R2 universities and 37% at other doctoral institutions, and this gradient is particularly strong among US-origin students (foreign-origin students write with AI at high rates at all institutions...).

Gross et al. then look at where students go after receiving their PhD, finding that:

Among US-origin graduates, those with AI-written dissertations are 7.4 p.p. less likely to be employed in academia in 2026 (17% over the 44% base rate) and 8.0 p.p. more likely to be employed in industry (16% over a 50% base), with no difference in government employment. These differences attenuate substantially for likely foreign-origin graduates. We also find differences within academia: conditional on academic employment, graduates with AI-written dissertations are 5.4 p.p. more likely to hold non-tenure-track positions (on a base of 12%), with no significant difference in the likelihood of tenure-track or postdoctoral positions.

Gross et al. explicitly caution against taking a causal interpretation of the effect of generative AI use on student outcomes. I didn't interpret the results that way, instead seeing them as perhaps indicating that students with dissertation abstracts written by generative AI may be lower-quality students, or those with less of an academic focus for their future employment. Alternatively, some students may simply be using AI to overcome language barriers, or to polish their writing, rather than to substitute for their own research skills. However, the results don't definitively establish either of those interpretations either. 

What the results do show is more banal - PhD students are increasingly using generative AI. Of course, it is important to keep in mind that these results tell us about the prevalence of AI-generated writing in dissertation abstracts, rather than the dissertation more generally. Since generative AI is quite good at summarising, and that is essentially the purpose of an abstract, I would expect AI-generated writing to be more prevalent in abstracts than in the main text of the dissertations.

Finally, while these results apply to STEM subjects, it seems likely that this is indicative of a more general trend across all fields. If this trend continues, it will not be long before AI-assisted dissertation writing becomes the norm rather than the exception. What we should be wondering now is whether that trend will hamper PhD students in their development as researchers, or not.

[HT: Marginal Revolution]

Read more:

• Lessons from Joshua Gans on AI for economics research
• This is not how generative AI should be used in research

Modelling the Nobel Prize in Economics

The Nobel Prize in Economics (technically, the Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel) will be announced next Monday. Who will win? It is a closely guarded secret, but of course there are prediction markets - at the time of writing, Kalshi has Ariel Pakes as the favourite, with Susan Athey a close second, and Richard Blundell a distant third.

In a discipline where modelling is ubiquitous, it is worth asking whether it is possible to model who will win the Nobel Prize. This recent working paper by Peter Dolton and Richard Tol (both University of Sussex) makes a good attempt. They build a dataset of past winners and candidates (based on research performance as well as winners of other awards), and then estimate a model that shows the factors correlated with winning the Nobel Prize.

Dolton and Tol start with a simple assumption, which is that the Nobel Prize Committee first chooses a field to award the prize to, and then they choose the best candidate within that field. They justify this assumption by showing that there is some regularity in the way that the award goes from field to field over time (based on their categorisation of 14 fields of economics). They use this to construct a 'transition matrix', that records how often the Nobel Prize goes from one field in one year to another the next year. 

Which fields get Nobel Prizes? Dolton and Tol show that larger fields (those with more candidates), those that have waited longer since their last prize win, and the transition matrix, are all statistically significantly correlated with which field wins the prize.

Who wins the Nobel Prize within the winning field? Dolton and Tol find that winning candidates are older (although the relationship with age is non-linear, and peaks at age 70-71 before declining), and those who have had a student already win a Nobel Prize are more likely to win.

Some important things come out of this paper. First, who will win the Nobel Prize in 2026? The 2025 winners were Joel Mokyr, Philippe Aghion, and Peter Howitt. Their research field, according to Dolton and Tol's categorisation, is 'Growth'. According to the transition matrix in Table E.8 of the paper, the most likely field to follow 'Growth', is 'Equilibrium and Welfare' (although 'Development and Economic History' and 'Macro' are also possible). 

I got ChatGPT to comb through Dolton and Tol's candidate list and identify the top candidates in 'Equilibrium and Welfare'. ChatGPT suggested Andreu Mas-Colell or Partha Dasgupta. I have heard Dasgupta's name in people's shortlists before, but not Mas-Colell. Of course, if the Committee considered they had already ticked the 'Economic History' box with Joel Mokyr last year, then next according to the transition matrix would be 'Games and Market Structure', 'Information', or 'Macroeconomics'. Across those fields, ChatGPT suggested many names, but noted that David Kreps is the strongest overall, to which I would add Ariel Pakes (note the consistency with the Kalshi market prediction). ChatGPT rated Oliver Blanchard the top candidate in Macroeconomics. Anyway, we will see next week!

Second, I love the list in Table 4 of the paper, which shows the economists who, according to Dolton and Tol's model, have had the greatest chance of winning the Nobel Prize but have not done so. Top of this list is Michal Kalecki (died in 1970), followed by Lionel Robbins (1984), Jacob Marschak (1977), Arthur Burns (1987), and Frank Hahn (2013). Other notable names in that list (at least, according to me) are Bill Phillips (died in 1975), Harold Hotelling (1983), Bill Baumol (2017), and Henri Theil (2000). There are a few on the list who are still alive, including Tim Besley, Robert Barro, Guido Tabellini, George Loewenstein, and Torsten Persson.

One reason why so many outstanding economists died without receiving the prize is that it was only first awarded in 1969, by which time there was a long backlog of worthy candidates. The third important thing to come out of this paper is a counterfactual exercise, predicting who would have won the Nobel Prize each year if the prize had been first awarded in 1901 (along with the other Nobel Prizes). Table F.15 in the paper has the results (up to 1976, after which it is assumed that the awards would continue as they have been). A number of important names appear in this list, including the first three winners being Leon Walras, Carl Menger, and Francis Edgeworth. My students would no doubt recognise Alfred Marshall (1910), Corrado Gini (1924), Joseph Schumpeter (1930), Arthur Pigou (1935), John Maynard Keynes (1938), and Joan Robinson (1958), among others. Interestingly, in this counterfactual exercise, the first woman to win the prize would have been Beatrice Webb in 1924 (jointly with Sidney Webb and Corrado Gini), some 85 years before Elinor Ostrom.

The Nobel Prize announcement is one of the highlights of my year. Now that I've considered the results from the model, with some assistance from my special adviser ChatGPT, I'm prepared to make my prediction: Kreps and Pakes. We'll find out next week!

[HT: Marginal Revolution]

The impact of a subtle wording change on measurement of worries about climate change

Consider the following statement: "The effects of climate change are too far in the future to really worry me". Would you say that you: strongly agree with that statement; tend to agree; neither agree nor disagree; tend to disagree; or strongly disagree? Now consider this alternative wording of the same statement: "The effects of climate change are too far in the future to really worry about". Would your level of agreement be the same to this second statement as to the first?

There is good reason to believe that the wording of that question matters, as shown in this new article by Lieke Voorintholt (Trier University), Adriaan Soetevent, and Gerard van den Berg (both University of Groningen), published in the Journal of Economic Psychology (ungated earlier version here). They argue that the single-word difference, from "worry me" to "worry about" may shift respondents' focus from themselves individually towards broader society. Therefore, they hypothesise that more survey respondents would report being worried when asked if they worry about the effects, compared to whether the effects worry them individually. They also hypothesise that the treatment effect would be greater for older respondents, since the "about" version prompts thinking beyond the respondent's own life expectancy.

Voorintholt et al. use data from the Innovation Panel of Understanding Society, which its website describes as "a test-bed for innovative ways of collecting data and for developing new areas of research". The survey is run in Great Britain, and their analysis is based on a sample of 2791 survey respondents from Wave 16 of the survey, conducted in 2023. Half of the respondents were randomised to be asked the "me" version of the question, and the other half were asked the "about" question. Voorintholt et al. term the "me" version the control group, because that is the wording that is used in the main Understanding Society survey. So, the treatment is simply the replacement of "me" with "about" in the wording of the statement.

Their main results are summarised in Figure 1 from the article:

When asked the "about" version, more respondents say they strongly disagree with the statement, while fewer respondents say they tend to agree with the statement. These differences can be interpreted as greater worries about climate change expressed by the treatment group than the control group. This is also supported by their quantitative analysis, which shows the average response among treated respondents is 0.18 points higher (on a 1-5 scale from strongly agree to strongly disagree) than control respondents. Voorintholt et al. find some suggestive evidence in support of larger effects for older respondents. The coefficient on an interaction term between treatment and an age dummy variable for those aged 66 and over is positive and relatively large, but is statistically insignificant.

Overall though, this research is a good reminder that even a one-word change can materially affect how survey respondents answer a question, and therefore how we measure important variables. The effect of subtle changes should not be underestimated. If you want to measure the level of concern that people express about climate change, it really matters whether respondents are asked whether climate change “worries me” or whether it is something to “worry about”.

How important is the apprenticeship model to entering a research career?

Those of us working in research careers can invariably share stories about working as a research assistant, cleaning datasets, coding, running models, doing literature reviews, and many of the other less-glamorous tasks that make up the research process. That was a key aspect of our apprenticeship into the world of research, and a gateway into a research career. But how important really is research assistance as an entry point into a research career?

That is the question addressed in this 2025 NBER Working Paper (ungated version here) by Ina Ganguli (University of Massachusetts, Amherst) and Raviv Murciano-Goroff (Boston University). They look at the impact of working in a university lab (an important subset of research assistance work) on subsequently pursuing a scientific career. Interestingly, Ganguli and Murciano-Goroff use changes in the local minimum wage as an exogenous source of variation in lab employment, so this paper also indirectly contributes to the literature on the employment effects of the minimum wage, in a context (research labs) that is not often the focus of that literature.

Their data comes from UMETRICS, which collates data on research grants across universities, and their dataset covers 32 universities over the period from 2000 to 2019 (although the dataset has coverage up to 2022, including those additional years would mean having to account for the COVID-19 pandemic).

First, using a dataset collated at the lab level, Ganguli and Murciano-Goroff use a staggered difference-in-differences approach to look at the impact of minimum wage changes on employment of undergraduates in the labs. This analysis essentially compares the change in undergraduate employment between the time before and the time after an increase in the minimum wage, between university labs that were affected by the minimum wage increase and those that were not. In that analysis, they find that:

...following minimum wage increases, labs decrease the employment of undergraduates by 7.4% on average...

So, not dissimilar to the literature on minimum wage effects on employment, when focused on young people in exposed occupations. Ganguli and Murciano-Goroff then use the minimum wage change as an instrument for students' exposure to laboratory research as an undergraduate. The key assumption is that minimum wages while an undergraduate affect later scientific careers only through their effect on lab employment opportunities.

Using a dataset of over 28,000 undergraduates and their subsequent career paths, Ganguli and Murciano-Goroff find that:

...decreased exposure to scientific work translates into significantly lower rates of undergraduate research assistants pursuing doctoral degrees or working in the life sciences sector after graduation. We find that working one fewer quarter in a lab during an undergrad student’s college years translates into between a 7.0 and 10.3 percentage point decrease in the rate of enrolling in a doctoral-level program. Given our sample of 28,283 students, this implies that if all students had experienced a minimum wage increase, roughly 500 fewer undergraduates in our sample would have pursued these advanced degrees.

So, the results imply that one fewer quarter of lab experience as an undergraduate reduces enrolment in a doctoral programme by 7.0 to 10.3 percentage points. That is a fairly large effect and should make graduate research programmes take notice of the importance of undergraduate research assistance opportunities for the pipeline into graduate research.

So, working in a lab as an undergraduate is a key pathway towards a career in the life sciences, and when those opportunities are restricted, students are less likely to embark on such a career. These results won't be too surprising to those of us who have been through an apprenticeship as a researcher. It is likely that I would be doing something very different right now if I hadn't been pulled into a lot of research projects towards the end of my undergraduate studies. I almost certainly wouldn't have pursued a PhD, or become an academic. Research assistant jobs are more than just jobs. They provide mentoring, information, skills, networks, references, and a chance to try out being a researcher in a safe setting.

These results also highlight two other things for me. First, they suggest that minimum wage increases may have a negative impact on the career pipeline into science. I'm unsure that this negative impact of the minimum wage has been identified before. However, we should be cautious because Ganguli and Murciano-Goroff are primarily interested in using minimum wage changes to identify the effect of undergraduate research experience on later careers, rather than estimating the overall long-run consequences of minimum wage increases for the scientific workforce. Nevertheless, their results suggest that this may be an important unintended consequence of higher minimum wages, and one that would be worth further research.

Second, given that research assistants do a lot of the 'drudge work' in research, if generative AI is also able to do a lot of that work, that further suggests a negative impact on the career pipeline into science through the rise of generative AI. I know others have written on this before (see here), so the challenge here is not a new idea.

On the plus side, that suggests another method that could be used to further test for the impacts of undergraduate (and graduate) research assistance on future academic careers, beyond focusing on life sciences (as Ganguli and Murciano-Goroff do). By comparing academic fields that are more (or less) exposed to generative AI (especially in the early days of generative AI), we might be able to tease out how important research assistance is to future academic careers across many fields.

Research apprenticeship is important, and these results give us a clear indication of how important it can be. Cleaning data, running models, searching the literature, and doing all the other seemingly mundane tasks of a research assistant are not just cheap ways for senior researchers to get research done. They are also how the next generation of researchers learns what research is, discovers whether they enjoy doing it, and gets started on a research career. If these opportunities are reduced, whether through higher minimum wages or through substitution by generative AI, we may save on some of the drudge work today, but at the cost of having fewer researchers tomorrow.

[HT: Marginal Revolution]

The tone and expression of academics on X (or Twitter)

In my previous post, I highlighted the apparent contribution of X (formerly Twitter) to toxicity on the Economics Job Market Rumors (EJMR) website. A natural follow-up question is whether and to what extent academics on X contribute to the toxicity on that platform and, by extension, to other forums such as EJMR. This recent article by Prashant Garg (Imperial College London) and Thiemo Fetzer (University of Warwick), published in the journal Nature Human Behaviour (open access), goes some way towards providing an answer.

Garg and Fetzer constructed a dataset of nearly 100,000 academics, including all of their Twitter [*] activity from 2016 to 2022. They then use large language models (ChatGPT-3.5 and GPT-4) to characterise each tweet in relation to content and tone. They assess each academic's stance on climate change, economic policy, and cultural issues. In terms of tone, they measure egocentrism (how often the academic refers to themselves in the first person), toxicity (based on the probability a tweet is classified as toxic by Google's Perspective API), and the balance between reason and emotion (measured as a ratio of 'affective terms' to 'cognitive terms' based on the Linguistic Inquiry and Word Count tool). The analysis is then largely descriptive, but nonetheless interesting.

Garg and Fetzer first find that:

...leading academics are not typically social media influencers... We found weak correlations between citation counts and Twitter metrics: citations and likes... citations and followers... and citations and content creation...

Garg and Fetzer observe that:

The weak correlation underscores that many prominent public intellectuals online gain visibility through public engagement rather than scholarly achievements, often holding lower academic credentials while commanding significant public attention, thus widening the gap between social media influencers and established academic experts.

I think that Garg and Fetzer overstate the case here. The weak correlations suggest that Twitter includes a cross-section of academics (in terms of academic quality), rather than that the top academics eschew Twitter (which would instead lead to negative correlations between measures of academic quality and Twitter engagement).

I'll put aside their results on political expression, which I round rather uninteresting. In contrast, the results in terms of tone demonstrate some interesting correlations. First, in terms of egocentrism (using self-referential terms such as 'I', 'me', 'my', and 'myself'):

Female academics... exhibit higher egocentrism than male academics...

Egocentrism increases with university ranking: academics at top-100 institutions... exhibit higher egocentrism than those from institutions ranked 101-500... US-based academics... show higher egocentrism than non-US academics

Then, in terms of toxicity:

Academics with high reach but low academic credibility... exhibit lower toxicity than those with the contrasting profile, that is, ones with low reach but high credibility...

Academics at top-100 universities... exhibit higher toxicity than those at institutions ranked 101-500... Moreover, US-based academics... exhibit higher toxicity than non-US academics...

And in terms of emotionality (or reason):

Emotionality is significantly higher among female academics... than male academics... In terms of reach and credibility, high-reach/low-credibility scholars... show significantly higher emotionality than low-reach/high-credibility scholars...

Finally, US-based academics... exhibit higher emotionality than non-US scholars...

Many of those differences will surprise no one, such as US-based academics being more egocentric and toxic in their expression on Twitter. Other differences seem to confirm familiar stereotypes, such as female academics using more emotional language than male academics. No doubt, some of the differences relate to differences in norms across different disciplines in terms of communication styles (both on Twitter and in general academic discourse). Garg and Fetzer don't control those other factors that might affect tone and expression. And before we get carried away about how toxic academics are on Twitter, Garg and Fetzer provide an important comparison with the general population. From Figure 6 in the paper:

Notice that academics (the blue line) exhibit far less toxicity (in the graph in the top middle) than the general population of Twitter users (the red line). Moreover, the trend in toxicity is downwards (for academics over the whole period from 2016 to 2023, and for the general population from 2021 to 2023). So, academics are not the main problem in terms of toxicity in the discourse on Twitter.

Nevertheless, there are important differences across academics, and one difference in particular stands out. Academics with high reach (those that are very active on Twitter) but low academic credibility (they are not highly credible academics, as measured by citations) exhibit less toxic expression on Twitter than other academics, particularly those who have low reach but high academic credibility. In their conclusion, Garg and Fetzer focus on this as a problem because:

...those with the greatest public reach may not represent top scholars, potentially distorting public perceptions

However, I see the opposite problem. In terms of tone, the top scholars with the lowest reach have the most toxic expression. Are those the sorts of academics that we want to promote even further on social media? I would suggest not.

What is a better option? First, more highly credible academics should be encouraged to engage in the social media discourse. However, it is important to recognise that credibility alone is not enough. What is needed are credible academics who also model constructive discourse without the toxicity, raising the standard of debate. However, as noted in yesterday's post, many high-quality (especially female) scholars are targets of hostility on social media. These are not separate issues.

Alternatively, we could raise the standard of academic discourse on Twitter more generally, without changing who is represented on the platform. That would reduce the toxic nature of the interactions. Stop laughing! It could happen. The tone and expression of academics on X (or Twitter) matters. Academics can set the standards for everyone else. We don't need to descend into the toxic culture wars that play out each day on social media. We are better than that, and if we show ourselves to be such, maybe more people will listen.

[HT: Marginal Revolution, last year]

*****

[*] I refer to the platform mostly as Twitter, because it didn't change names to X until July 2023, after Garg and Fetzer's dataset ends.

Evidence that artificial intelligence is increasing the impact, but narrowing the scope, of research

There is growing evidence of positive impacts of generative artificial intelligence on productivity. This includes productivity in research (see this post, for example), including my own. However, some have questioned whether increasing research productivity comes at a cost of narrowing the scope of research.

So, I was interested to read this article by Qianyue Hao (Tsinghua University) and co-authors, published in the prestigious journal Nature (ungated earlier version here) late last year. They look at the impact of AI tools (not limited to generative AI) on the productivity of researchers and the quality of research. Specifically, they look at authors publishing in six representative fields: biology, medicine, chemistry, physics, materials science, and geology, across three 'eras': (1) the 'machine learning era ' (from 1980 to 2014), the 'deep learning era' (from 2015 to 2022), and the 'generative AI era' (from 2023 onwards). Hao et al. compare authors who publish 'AI augmented papers' with those who do not. An 'AI augmented paper' is one that uses methods such as:

...support vector machines and principal component analysis from the machine learning era, and convolutional neural networks and generative adversarial networks from the deep learning era. Large language models, which have emerged in recent years, also rank among the most frequently used methods...

Using a dataset that includes over 27 million papers with complete records that were published between 1980 and 2025, of which about 310,000 were 'AI augmented', Hao et al. find that:

...annual citations to AI papers are 98.70% higher than those to non-AI papers on average...

So, AI augmented research gathers more citations, which suggests that authors using AI in their research achieve greater impact. This is reinforced by evidence that AI augmented papers are published in higher quality journals (with Q1 journals being the highest ranked). Hao et al. report that:

...the proportion of AI papers in Q1 journals is 18.60% higher than that of non-AI papers in all journals; in Q2 journals, the AI proportion is 1.59% higher; whereas Q3 and Q4 journals hold a relatively lower proportion of papers with AI... These results indicate a heterogeneous distribution of AI-augmented papers across journals, with a higher prevalence in high-impact journals.

And AI appears to make authors more productive, as:

On average, researchers adopting AI annually publish 3.02 times more papers... and garner 4.84 times more citations... than those not adopting AI, with consistency.

All of these results seem to hold across all of the disciplines that Hao et al. consider. However, it is not all good news. Hao et al. use machine learning to create a measure of the 'breadth of scholarly attention'. Using that measure, they find that:

Compared with conventional research, AI research is associated with a 4.63% contracted median collective knowledge extent across science, which is consistent across all six disciplines... Moreover, when dividing these disciplines into more than two hundred sub-fields, the contraction of knowledge extent can be observed in more than 70% of them...

Of course, some of the differences here may be due to selection, as the types of researchers, and the types of research, involving AI use may be meaningfully different from those that don't. However, putting the selection issues aside, Hao et al. note that there is a tension between the individual researcher's incentive to produce a greater quantity of research that has higher impact, which would suggest greater use of AI, and the social incentive to produce a greater breadth of research.

So, the takeaway from this paper is that we need to consider researcher incentives, not just productivity. Specifically, this research suggests that the use of AI in research is leading to a 'prisoners' dilemma' outcome: each individual researcher acting in their own best interests (and using AI in their research) leads to an outcome that is worse for society overall (less breadth of research and more incremental gains).

Hao et al. conclude that:

The substantial academic benefits of AI use may be a driving force behind its accelerated rate of adoption; however, we also find unintended consequences from the increased prevalence of AI-augmented research. In all fields, AI-augmented research focuses on a narrower scope of scientific topics and reduces the scientific engagement of follow-on research, leading to more overlapping research work that slows the expansion of knowledge. Further, with a greater concentration of collective attention to the same AI papers, the adoption of AI seems to induce authors to converge on the same solutions to known problems rather than create new ones.

So, what is the solution here? Society probably wants research to be higher quality and have a broad scope. But individual researchers' incentives to use AI in their research appears inconsistent with that outcome. The traditional prisoners' dilemma is a repeated game (see here or here, for example), and the players of that game can avoid the worst outcome by cooperating. In this case, the researchers could cooperate by agreeing not to use AI in their research. The problem is that every researcher has an incentive to cheat on that agreement, since if they use AI, then that will be good for their career. This prisoners' dilemma is more difficult to ensure cooperation in than the traditional game, because there are not just two players who need to cooperate, but thousands (or millions). Ensuring cooperation in a prisoners' dilemma game with many players, each of whom is far better off cheating than cooperating, is almost impossible (which is why solving the problem of climate change is so difficult).

My own view is that the answer is not to keep AI out of research. That is not realistic, in the same way that it's not realistic to expect students not to use generative AI. The incentives need to be redesigned, but this will be no easy task. As long as universities, research funders, and publishers reward researchers for quantity, citations, and publication in top-ranked outlets, then we should expect more AI-augmented work, with a narrower scope than society might prefer. If we want AI to expand knowledge rather than simply accelerate competition within narrow foci, then we need institutions that also reward novelty, breadth, and the discovery of new questions. That is the economic challenge we must face up to.

[HT: Marginal Revolution]

Seven decades of change in the demographics and research styles of top economics research

Back in 2013, Daniel Hamermesh (University of Texas at Austin) published this article in the Journal of Economic Literature (ungated earlier version here), which summarised changes in the demographics and research styles of top economics research, based on articles published between 1963 and 2011 in three top journals: the American Economic Review (AER), the Journal of Political Economy (JPE), and the Quarterly Journal of Economics (QJE). A new update last year (open access) from Hamermesh extends the analysis to include articles up to 2024.

In terms of demographics, the trends show a continuation and in terms of gender, Hamermesh notes that:

The progression that occurred from the 1960s and 1970s, when only a minute fraction of authors were women, to the early twenty-first century has, if anything, accelerated.

This will be welcome news, given the persistent gender gap in economics (see this post and the links at the end of it). It likely reflects the changing demographics of young economists, with a growing proportion of the young 'stars' in economics being women (and noting that it is young stars who often get published in the top journals that Hamermesh is considering).

In terms of the age structure of authors, Hamermesh reports that:

The changes from 2011 to 2024 continued those that started in the 1980s, but the rate of change has not accelerated. Indeed, most noticeable from 2011 to 2024 was a continuing sharp and statistically significant drop in the representation of the youngest group (and a nearly equal sharp rise among those 36–50)...

...the average age of authorship has increased steadily since 1973. 

Can I change my comment above about the young stars in economics? The increasing median age of authors in top journals seems to be a general trend across academia. Hamermesh then turns to research 'style', documenting a continued dramatic rise in the proportion of articles in those journals that are co-authored:

There were no four-authored papers as recently as 1983; today they account for 17 percent of articles. There were no papers with more than four authors in 2003; today nearly 12 percent of articles have five or more authors (with five articles written by six authors each and one by seven authors). Obversely, sole-authored papers are now quite scarce; and even two-authored papers today only account for slightly more than one-fourth of all articles (compared to a majority as recently as 2003).

Unsurprisingly, the increase in the number of co-authored articles means that the age diversity of author collaborations has increased over time as well. In terms of the types of research, he reports that:

The big changes are the continuing rise in empirical work based on original non-laboratory data and the rapid and even accelerating increase in experimental work. Today these two methods, which both involve collecting original data, account for over half of all published papers, compared to less than 4 percent four decades ago...

These trends are not all unrelated, of course. Experimental research, and the increasing use of large datasets, typically both require larger research teams. They also often require more detailed methods, which may involve both larger teams, and more experienced researchers. Larger teams might be more likely to include female team members. And larger teams often need someone to lead and coordinate all of the team members, and those leaders tend to be more experienced (and older) academics. So, it would not surprise me, if more detailed analysis was conducted, to see that the trends are interconnected.

Now, the interesting thing will be what happens going forward, given the increasing use of generative AI in research (see here, for example). Since generative AI can now do a lot of the work that research assistants and early career researchers previously did, will the trend towards larger research teams be reversed? How will that interact with the gender gap in research (given that the age of female economists skews younger at the moment). And how will it affect the age distribution of researchers (given that men, and younger people, are somewhat more likely to use generative AI). I'll be looking forward to Hamermesh's next update. Hopefully, we don't have to wait another 12 years.

[HT: Marginal Revolution, last year]

Changing their minds could be a good thing for economists

People don't like to change their minds. This may partly be an expression of loss aversion - we really want to avoid losses, including the loss of an idea that we previously thought was true. This leads to status quo bias - we prefer not to change things, and keep them the same, because changing things entails a loss. But what if changing our minds could make us better off? Would we be so reluctant to do so?

This 2025 paper by Matt Knepper (University of Georgia) and Brian Wheaton (UCLA) suggests that economists, at least, should not be afraid to change their minds, because doing so increases the number of citations to their research. Knepper and Wheaton investigate authors who undergo an 'ideological reversal' - previously publishing research that could be considered right-wing, before switching and publishing a paper that draws a left-wing-consistent conclusion, or the reverse (switching from left-wing to right-wing). Their main data source is every economics paper ever published in the top 100 economics journals indexed in Web of Science - some 200,000 articles. They also have a narrower dataset of papers referenced in meta-analyses on policy topics, including:

...the minimum wage, the economics of unions, the taxable income elasticity, the fiscal multiplier, intergenerational transfers, trade and productivity, trade and domestic employment, crowd-out, the gender wage gap, unemployment insurance, disability benefits, universal preschool, childcare and employment, immigration and wages, and more.

Knepper and Wheaton use this narrower dataset to train a machine learning model to categorise the rest of the papers in the dataset, as to how left-wing (or right-wing) the conclusions are. For instance, a paper that concludes that the minimum wage reduces employment is more right-wing, whereas one that concludes that there is no disemployment effect of the minimum wage is more left-wing. Knepper and Wheaton define an author as left-wing if they published more left-wing papers than right-wing ones over the previous five years, and the reverse for right-wing authors. They then use the larger dataset to investigate what happens to each economist who undergoes an 'ideological reversal'. They first outline some descriptive facts based on their dataset, including:

• Fact #1: The typical author mostly publishes results on one side of the political spectrum.

• Fact #2: Ideological reversals are not rare; they occur at least once for 40% of authors.

• Fact #3: Ideological reversals become much more common later in an author’s career, with authors essentially never undergoing a reversal in the first decade of their career.

• Fact #4: Most ideological reversals do not represent a permanent defection to the other side of the political spectrum, but rather the beginning of repeatedly publishing results on both sides of the spectrum.

• Fact #5: Ideological reversals occur much more frequently amongst authors who are (initially) classified as right-wing.

That does seem like a surprisingly high proportion of economists who undergo at least one ideological reversal. However, perhaps we should take comfort in that - if the results point in a particular direction, our conclusions should say that, even if that conclusion is inconsistent with our previous conclusions on the same topic.

Do these ideological reversals matter though? Knepper and Wheaton employ a difference-in-differences analysis, comparing the difference in citations (and other metrics) between authors who did, and did not, undergo an ideological reversal, between the time before, and after, the reversal occurred. In other words, they look at whether citation counts rise more for economists who have an ideological reversal than for otherwise similar economists who do not. The results are striking, with:

...a sharp clear increase in citation count following an ideological reversal with essentially no evidence of pre-trends... The citation boost accumulates to approximately 9 over a one-decade period and 30 over a two-decade period.

The results remain consistent when Knepper and Wheaton limit the analysis to papers published before the ideological reversal, and when they limit the analysis to papers in the meta-analysis only (showing that the machine learning approach doesn't drive the results). Knepper and Wheaton also find evidence consistent with no change in the quality of papers before and after the ideological reversal, and that:

Both left-to-right and right-to-left reversals are rewarded by increased citations of roughly the same magnitude. The boost in citations received subsequent to a left-to-right reversal is mostly driven by citations from right-wing authors, and the boost in citations received subsequent to a right-to-left reversal is mostly driven by citations from left-wing authors. Encouragingly, however, the new right-wing (left-wing) audience garnered by a left-to-right (right-to-left) reversal... also engages with and cites the author's previous left-wing (right-wing) papers. This dynamic suggests that ideological reversals help prevent the formation of echo chambers in economics academia and expose authors to opposite ideological findings.

This last result is particularly important, and I believe it allows us to conclude that economists need not fear ideological reversals. In doing so, they can attract a new audience from the other side of the ideological spectrum, bringing the two sides closer together. Hopefully through that, we end up with higher-quality research overall.

[HT: Marginal Revolution, last year]

This is not how generative AI should be used in research

I've been using ChatGPT Pro to help with drafting research papers this year, as I noted that I would do in this post from January. It has amped up my productivity a lot, allowing me to finish writing up two papers already, with a third on the way. These were papers where the analysis was already done, but it was the writing that was holding up the process. Having ChatGPT to help with the drafting seems to kickstart my writing, even though I have ended up extensively re-writing everything that ChatGPT produces. I find it a good disciplining tool as much as anything. Several colleagues have asked whether I am disclosing my generative AI use to journal editors when I submit. And I do. I have a standard 'generative AI use statement' that I include in my papers, that notes how it was used, and that I remain responsible for all of the content. You can see an example in this recent working paper.

However, not everyone is as careful with their generative AI use, or as transparent. Consider this example:

That is both infuriating and a sad indictment of the reviewing, editing, and publishing process, not least because, as on Reddit commenter noted, many authors see high-quality work rejected by journals, whereas a paper like this, with obvious flaws, has successfully been published. And it's not an isolated incident. This 2025 article by Artur Strzelecki (University of Economics in Katowice), published in the journal Learned Publishing (open access), catalogues over 1300 instances of likely unacknowledged and frankly stupid use of ChatGPT, up to September 2024.

Strzelecki's approach is to search for text strings that are almost certainly ChatGPT responses to a prompt asking it to generate text. The main example Strzelecki uses, which is in the title of the article, is "as of my last knowledge update". No human author is going to say that in a research paper. Similarly, "as an AI language model", "I don't have access to", and "certainly, here is" are highly indicative of ChatGPT use. There are circumstances where a human might use those phrases in a research paper, but it seems unlikely. Strzelecki screens out papers that mention ChatGPT, and manually checks each paper to ensure the text was not in some way legitimate, and that leaves 1362 articles.

How do these articles get published with this content intact? There are lots of stopping points where this could be caught and corrected (or prevented), but these articles have gotten through all of them. Strzelecki outlines the process. First, perhaps it is only one of the authors (and not all of them) that used ChatGPT. In which case, why didn't the other co-authors pick it up? Next, the paper is submitted to a journal, and often goes through a text review by the publisher. And then the editor or editors (including associate editors) looks at it, and decides whether it should be sent out for peer review. And then the peer reviewers (usually more than one, sometimes four or more) look at the paper in detail and provide comments. Then the editor receives the review reports and makes a decision. The paper may go through more than one round of review and editorial decision. And then, once accepted for publication, the article may be copy-edited. And at any of those stages, this text could be picked up. And yet, for over 1300 articles as of September 2024, the ChatGPT-generated text has not been picked up.

Strzelecki particularly focuses on 89 articles that have been published in journals indexed by Scopus or Web of Science, which should be the most credible journals. Of these:

...as many as 28 of them are in journals with Scopus percentile values of 90 and above. Two journals have a 99th percentile, indicating that they are the top journals in their field...

In total, 64 articles were found in journals considered to be in Q1, top quartile, recognized as the group of the best journals in their respective fields. Twenty-five articles are in the percentile range between 50 and 75, indicating that the journals in which these articles are found belong to Q2.

So, this phenomenon is not limited to low-ranked 'predatory' journals. In fact, looking at the list, there are several journals published by MDPI and Frontiers (for more on those publishers, see here). However, there are a whole lot published by Elsevier and Springer, publishers that we should expect much better of. Although, those are also publishers that publish a lot of journals, and a lot of articles, so perhaps that accounts for their higher numbers within the 89 articles that Strzelecki focuses on. Fortunately, I don't see any reputable journals in economics in the list, but I could be wrong.

Anyway, the takeaway is not so much that generative AI use is widespread in the write-up of research. It is that authors are using generative AI, not being transparent in their use of it, and that the quality control system by journals, even high-ranking journals, is terrible. Strzelecki makes a good point in the conclusion of his article that 89 out of over 2.5 million articles indexed in Scopus is only 0.000035% of the total indexed articles. However, this analysis is only picking up the really, really obvious cases. There will be far more use of generative AI that has not been adequately checked or acknowledged by authors, and not picked up in quality control.

I'm not against using generative AI in the write-up of research. Obviously, because I am doing the same thing. What needs to happen is that researchers need to be transparent and honest when they use generative AI, so that editors, reviewers, and the readers of research can see how it was used. That way, the users of research can evaluate for themselves whether they should believe, discount, or discard research depending on the ways and the extent of generative AI use. Without transparency, that important evaluation step is lost.

[HT: Artur Strzelecki]

Read more:

• Lessons from Joshua Gans on AI for economics research

The long persistence of retracted 'zombie' papers

When a paper is retracted by a journal, that understandably tends to negatively impact perceptions of the researcher and the quality of their research (see here). However, these 'zombie' papers can maintain an undead existence for some time, continuing to be cited and used, sometimes uncritically, because retractions take time and because publishers are not good at highlighting when an article has been retracted. They may even continue to accrue further citations even after being retracted. In terms of understanding the effect of retractions on the research system, a key question is: how long does it take for a paper to be retracted?

That is essentially the question that this new article by Marc Joëts (University of Lille) and Valérie Mignon (University of Paris Nanterre), published in the journal Research Policy (open access), addresses. Joëts and Mignon draw on a sample of 25,480 retracted research articles over the period from 1923-2023 (taken from the Retraction Watch database), and look at the factors associated with the time to retraction (that is, the time between first publication and when the article is retracted). First, they find that:

...the average time to retraction is approximately 1045 days (nearly 3 years), but there is significant variability, with a standard deviation of 1225 days... However, some extreme cases take much longer, with the longest retraction occurring 81 years after publication.

Joëts and Mignon use several different forms of survival model to evaluate the relationship between the characteristics of an article and the time to retraction. In this analysis, they find that:

Papers in biomedical and life sciences are generally retracted faster than those in social sciences and humanities, and articles published by predatory publishers are withdrawn more promptly than those from reputable journals. Collaboration intensity and type of misconduct also emerge as significant predictors of retraction delays.

The result for predatory journals seems somewhat surprising. However, Joëts and Mignon suggest that:

...predatory journals often publish papers with evident deficiencies that are more easily detectable by external parties, such as watchdog organizations or institutions, leading to quicker retractions when misconduct is identified. Additionally, the lack of formal editorial procedures in predatory journals may result in a less structured and faster retraction process...

Of course, a faster time to retraction doesn't make predatory journals good. It simply makes them less bad, since they almost certainly are a large source of low-quality research that deserves retraction (Joëts and Mignon don't report the proportion of retractions that come from predatory journals).

In terms of collaboration intensity, articles with more co-authors take longer to retract, presumably because more people are involved in the retraction process, or because disputes over who is to blame may take some time to resolve. For types of misconduct, retractions due to 'data issues' take the longest to occur, while those for 'peer review errors' and 'referencing problems' take the least. That likely reflects that it takes some time for data analyses to be replicated and for problems to surface, whereas problems with referencing are more likely to be readily apparent from a simple reading of the article.

Joëts and Mignon also do a lot of modelling of different editorial policy changes and their effects on the distribution of times to retraction, but I don't think we can read too much into that part of the article, as the results are mostly driven by the assumptions on how the policies affect retractions. Nevertheless, this paper provides some insight into why zombie papers can keep shambling through the literature: retractions are slow and the time to retraction depends on discipline, publisher type, collaboration, and the kind of misconduct involved.

Read more:

• Retractions hurt academic careers, and may be worst for senior researchers

Lessons from Joshua Gans on AI for economics research

I've been increasingly using generative AI (specifically, ChatGPT) to assist with research. I've been quite cautious though, worrying a lot about the quality of AI output, although my worries reduced substantially once ChatGPT started linking to its sources. However, I know many other researchers are using generative AI far more extensively and directly in their research than I am. My approach continues to be to use ChatGPT as an enthusiastic, but not fully polished, research assistant. Given my experience so far, I was interested to read the reflections of Joshua Gans on his year of using generative AI for economics research. His approach was:

I had lots of ideas for papers that I hadn’t developed, so I decided to spend the year working my way down the list. I would also add new ideas as they came to me. My proposed workflow was all about speed. Get papers done and out the door as quickly as possible, where a paper would only be released if I decided I was “satisfied” with the output. So it cut any peer reviews or discussions out during the process of generating research quickly, but I would send those papers to journals for validation. If I produced a paper that I didn’t think could be published (or shouldn’t be), then I would discard it. There were many such papers.

Like Gans, I have a lot of research ideas, and not enough time to pursue all of them. Many of my ideas would go nowhere, even if I did have time to pursue them. But for some others, I have later read research papers that have done something I had thought of earlier but not had time to do myself. There are therefore a lot of missed opportunities, because it isn't possible to perfectly identify the good ideas in advance - you need to try them out before you realise that they are uninteresting or dead ends. Being able to try out more ideas seems like a good thing.

The opportunity cost of spending time pursuing one research idea is not pursuing other ideas. If generative AI allows us to pursue research ideas in less time, then it lowers the opportunity cost of pursuing those ideas. However, as Gans notes:

When you lower the cost of doing something, you do more of it. Normally, the decision whether to continue or abandon a project gives rise to some introspection (or rationalisation) of whether continuing is worthwhile relative to the costs. When the going gets tough, you drop ideas that don’t look as great.

The issue with an AI-first approach is that its benefit, reducing the toughness of going, is also its weak point; you don’t face those decision points of continuing/abandoning as often. That means that you are more likely to end up completing a project. But this lack of decision points means that you end up pursuing more lower-quality ideas to fruition than you would otherwise.

In Gans's experience, AI increases research output, but it also weakens the stopping rule that would otherwise kill bad ideas early, by decreasing the marginal cost of continuing the research. When the marginal cost of continuing is lower, we spend more time on each bad idea before discarding it. And if we spend too long on bad ideas, the opportunity cost of pursuing an idea increases - time spent working on bad ideas is time not spent pursuing ideas that turn out to be better. As a result, the average quality of our research may decrease. That risk needs more careful consideration.

Although he doesn't note the potential increasing opportunity cost, Gans's conclusion seems to point in that direction:

My point is that the experiment — can we do research at high speed without much human input — was a failure. And it wasn’t just a failure because LLMs aren’t yet good enough. I think that even if LLMs improve greatly, the human taste or judgment in research is still incredibly important, and I saw nothing over the course of the year to suggest that LLMs were able to encroach on that advantage. They could be of great help and certainly make research a ton more fun, but there is something in the judgment that comes from research experience, the judgment of my peers and the importance of letting research gestate that seems more immutable to me than ever.

Generative AI has the potential to increase the quality, and the quantity, of research. Gans seems to have seen it in his work, and I've seen it already in my own work too. In fact, my experience so far has been that careful use of generative AI (for example, checking for literature gaps, or exploring econometric methods and robustness checks) has reduced the time wasted on fruitless research that would have gone nowhere. However, it is possible to use too much generative AI in research, just as it is possible to use too little. There is a middle ground, and Gans seems to be finding it from one direction (starting from over-using generative AI), and maybe I am finding it from the other (starting from under-using generative AI). The important thing seems to be ensuring that a human is kept in the loop (as Ethan Mollick noted in his book Co-Intelligence, which I reviewed here). Specifically, we can use generative AI for its strengths (in testing our initial ideas, mapping the literature, exploring alternative methods, or stress-testing assumptions). And we can keep the human in the loop by pausing the research at more key points to consider where it has gotten to and check our intuition, as well as continuing to seek peer review of the draft end-product.

So, Gans might be holding back on the generative AI this year, but I'll be further expanding my use. Starting with something related to this post: writing up some research on using AI tutors in teaching first-year economics, which is research I presented in a brown-bag seminar at Waikato last month (and I will have more on that in a future post).

[HT: Marginal Revolution]

Specialisation trends in economics, and who’s citing economics

Do research findings in one discipline affect research in other disciplines? This seems like an important question, and one to which we hope the answer is yes. And yet, for a long time disciplines seemed to be becoming more insular, only referencing research within their own narrow fields and sub-fields. More recently, it seems to me that the breadth of citation across disciplines has increased greatly. Call it the 'Google Scholar effect', if you like. Since research has become much more searchable, relevant past research has become easier to find and to cite. If there is a 'Google Scholar effect', we'd expect to see a big change in citation patterns across disciplines starting from 2004, when Google Scholar was introduced.

So, I was interested to read this 2024 article by Sebastian Galiani (University of Maryland), Ramiro Gálvez (Universidad Torcuato Di Tella) and Ian Nachman (Brown University), published in the journal Economic Inquiry (ungated earlier version here). They conduct a citation analysis to look at specialisation trends in economics. Specifically:

...we look for patterns that indicate specialization within a field of economics research, such as a narrowing of the topics it covers in a way that other fields do not, and patterns suggesting a decrease in citations from outside the field...

Their dataset includes:

...all articles published from 1970 up to and including 2016 in the so‐called economics Top 5 journals (The American Economic Review, Journal of Political Economy, The Quarterly Journal of Economics, Econometrica, and The Review of Economic Studies) and for all articles published from 1970 up to and including 2016 in a set of non‐Top 5 general research journals (The Economic Journal, International Economic Review, Economic Inquiry, and The Review of Economics and Statistics)...

Galiani et al. categorise each article into one of four 'fields' of economic research:

...(1) applied, (2) applied theory, (3) econometric methods, and (4) theory... The criteria used to assign a paper to a field were as follows: (1) Applied articles have an empirical or applied motivation. They rely on the use of econometric or statistical methods as a basis for analyzing empirical data, although they may deal with simple models that provide a theoretical framework for the analysis... (2) Applied theory articles construct a theoretical model to explain a fact, with the empirical analysis serving as a supplementary aspect rather than the primary focus of the paper. These articles typically have limited utilization of econometric or statistical analyses, though they may employ simulations (even with empirical data) or refine other techniques to test the implications of the models... (3) Econometric methods articles develop econometric or statistical methodologies... (4) Theory articles lack an empirical section; typically, they approach a topic through modeling and extensive use of formal mathematics and logic. These articles may incorporate a numerical example or a simple model calibration using theoretical data to illustrate the proposed model or to analyze its comparative statics.

The process of assigning articles to a field involves machine learning, which I'm not going to get into here. They also use natural language processing to detect 'latent topics' based on the content of each article, where each article is assigned to exactly ten topics. They use that approach, rather than using keywords or JEL codes because the topics that they identify use similar methods, making them more similar than articles that may have the same keywords but approach the research question in very different ways.

Galiani et al. then look at the long-term trends in the topics, content, and citations (both within and outside of economics) of their large sample of 24,273 articles. The results make for interesting reading, not just in terms of the trends, but in terms of the differences between the different fields of economics:

...theory and econometric methods have shown a narrowing focus on specific research topics since the 1990s, indicating a tendency toward specialization. Theory articles have experienced a significant increase in topics related to formal mathematical proofs and game theory, while econometric methods articles have shown a pronounced rise in topics related to computational statistics, estimators' asymptotic properties, and estimators' bounds. In contrast to applied papers, these fields do not exhibit rising trends in extramural citations (i.e., citations from other disciplines) and in citations from other fields within economics research (in the case of econometric methods, it shows a declining trend). These patterns also indicate a higher degree of specialization.

Applied papers have expanded their coverage to include diverse topics. Applied articles have seen a pronounced rise in topics related to impact analysis, causal analysis, and experimental economics. Over time, applied articles began to receive a higher proportion of citations from external fields, especially from disciplines such as medicine, psychology, law, and to a certain extent, education... Overall, these patterns indicate that applied papers are becoming more multidisciplinary. The case of applied theory articles is less conclusive. While they cover a broader range of topics (similar to applied papers), there has been no significant increase in extramural citations or citations from other fields of economics research (as observed with theory articles).

The observation that applied papers have received an increasing proportion of citations from outside economics over time is particularly interesting. Applied papers are more easily cited outside of economics because they often address research questions that other disciplines are also interested in. Galiani et al. identify that applied articles in economics are often cited in the education, medicine, and psychology fields, more so than articles from other fields of economics. This pattern of extramural citations does seem to become more apparent from the 1990s onwards, which would seem to suggest that Google Scholar (which was introduced in 2004) is not implicated, despite my expectation of a 'Google Scholar effect'.

That economic theory articles are not well cited outside of economics, and even outside of economic theory articles, will come as little surprise. Economic theory articles tend not to cover topics that are of interest outside of economics, and when they do, they are far too technical for those outside of economics to appreciate.

On the other hand, econometric methods should have a greater impact outside of the field, and it is disappointing that those papers do not seem to have an external impact. There are advances in econometrics, particularly in causal inference, that other disciplines should take advantage of. In contrast, economics increasingly involves methods drawn from applied statistics and computer science (including the machine learning and natural language processing that Galiani et al. used in this research).

Overall, the takeaway from this research is that economics is simultaneously becoming more specialised (within economic theory and econometric methods) and having a greater impact outside of the field (for applied papers). It would be interesting to see whether similar results would come from a similar analysis of articles in political science or psychology. If political science or psychology show the same pattern, that would tell us something important about how methods and results diffuse, or how they don't.

[HT: Marginal Revolution, last year]

Do men and women pitch science proposals differently, and does it matter for funding outcomes?

If male academics and female academics write academic papers and grant proposals differently, does that lead to different outcomes by gender? Past studies have worried about whether grant funding decisions are affected by gender bias (see here, for example), and differences in writing style may contribute to that. However, the article I discussed in this post from earlier this year concluded that there was little evidence of bias in grant funding, at least since 2000 in the US.

Nevertheless, I thought it would be interesting to read this 2020 article by Julian Kolev (Southern Methodist University), Yuly Fuentes-Medel, and Fiona Murray (both MIT), published in the journal AEA Papers and Proceedings (ungated here), because it not only looks at the grant funding decisions, but also at writing style. Kolev et al. focus on grant applications submitted to the Bill and Melinda Gates Foundation and the National Institutes of Health (NIH) over the period from 2008 to 2016. The sample includes 6931 Gates Foundation applications and 12,589 NIH applications.

Kolev et al. first subject the applications to textual analysis to the abstract of each application, evaluating the positivity of the text (and the extent to which the word "novel" is used), the readability (using the Flesch reading ease score), concreteness of the language (as opposed to abstractness), and three measures of how narrow or broad the abstract is. In this textual analysis, they find that:

...female applicants are less likely to present their research using positive vocabulary, they are more likely to write with high readability, and they prefer concrete language. Moving to our final three measures, we find an interesting dichotomy: even as female applicants use fewer broad words and more narrow words in their abstracts, we find that their research is characterized by lower MeSH concentrations, meaning that they cover a wider range of medical subjects in their work, at least within the NIH sample. Effect sizes are relatively small: the impact of gender ranges from approximately 0.04 to 0.08 standard deviations for our significant effects.

So, there are small but statistically significant differences in writing style between male academics and female academics in these funding proposal abstracts. Does that translate into differences in outcome? Kolev et al. test for whether the measures of writing style correlate with funding outcome, while controlling for:

...calendar time and application topic fixed effects, controls for total word count and the count of relevant words for dictionary-based metrics, and applicant publication history and gender.

In this analysis, Kolev et al. find that:

For Gates applicants, high levels of concreteness tend to improve the odds of funding; by contrast, at the NIH, we find a strong positive impact for MeSH concentration and marginal effects for both broad and narrow words.

So, the evidence is weak that writing style matters, or that writing style differences between the genders affect the success of funding applications. However, not so fast. There is a key problem with this analysis. If you read the quote above about the control variables in this second analysis, you may note that they control for gender. That might sound sensible, but if you're wanting to evaluate whether writing style differences between the genders affect funding outcomes, you don't want to control for both writing style and gender. What Kolev et al. have actually tested is whether writing style differences within each gender affect funding application outcomes, finding that they don't. As an example, their analysis doesn't answer the question of whether readability differences between male and female academics affect funding outcomes, it answers the question of whether readability differences matter overall (which it appears they don't), controlling for the average difference in funding outcomes between men and women. Those are quite different questions.

In other words, we probably want to know whether style mediates the effect of gender on funding outcomes, but this analysis doesn't do that. Instead, they should either run the analysis with gender as the main explanatory variable, then add the style variables and see if the coefficient on the gender variable shrinks, or run the analysis with interactions between gender and the style variables.

The results are, on the one hand, surprising. Quality of writing should matter. Stylistic differences should matter less. However, the quality of the proposed research should matter even more than the quality or style of writing. And this study wasn't even evaluating the quality or style of all of the writing (or the quality of the proposed research), only the style of writing in the abstract for the proposal. That, along with the issue with the second analysis above, make this paper of limited use for understanding whether there is a gender difference in funding outcomes (and, if there is, whether writing style differences contribute to the difference). The difference in writing style is an interesting result in itself, but we need to know more.

Read more:

• The gender gap in economics
• More on the gender gap in economics
• CORE on 'missing women in economics'
• Female representation in economics
• Justin Wolfers on why we should care about the gender gap in economics
• Why we should care about the gender gap in economics
• Could closing the gender gap in economics be as simple as providing students with information?
• A sobering report on the culture in the economics profession
• The gender gap in reviewing and editing for top economics journals
• The gender gap in U.S. economics education
• UWE and the gender gap in economics (results pending?)
• Three papers on the gender gap in economics seminars
• Disappointing results on simple information interventions to close the gender gap in economics
• Gender differences in fields of specialisation within economics
• Gendered citations of papers in the 'top five' economics journals
• The positive effect of mentoring for female economists
• What machine learning is telling us about the gender dynamics in economics seminars
• There's something about Wellesley
• The gender gaps in academia may not arise entirely from gender biases

Andrew Leigh on big data vs. randomised controlled trials

'Big data' has become the catchcry of many data scientists and researchers in recent years. It's also become increasingly used in economics. However, by itself the analysis of big data doesn't provide anything but big data correlations. Even when big datasets are available, there is still a place for randomised controlled trials (RCTs). That is the essence of this new article by Andrew Leigh (Parliament of Australia), published in the journal Australian Economic Review (sorry, I don't see an ungated version online).

It should come as no surprise that Leigh is pro-RCT. After all, he is the author of the book Randomistas (which I reviewed here), which was essentially a tribute to RCTs. Leigh clearly sees the rise of big data, and its increasing use as a substitute for RCTs, as a threat to good research. In the article, he takes great pains to point out instances where big data draws the wrong conclusions, compared with RCTs on the same topic. For example:

Randomised trials have demonstrated a strongly beneficial effect of statins on reducing cardiovascular mortality. Yet when they analysed a database covering the entire Danish population, researchers found that the chance of death from cardiovascular causes was one‐quarter higher among those who took statins than among those who did not. The explanation is straightforward: people who were prescribed statins were at elevated risk of having a heart attack. Yet even when researchers made statistical adjustments, using all the variables available in the database, they were unable to reproduce the well‐known finding that statins have a beneficial effect on cardiovascular mortality.

Analysis of the Danish database also suggested that the relative risk of cancer was 15% lower among patients who took statins, an effect that remained statistically significant even after controlling for other observed factors about the patients. Yet this result is at odds with the evidence from randomised trials. A meta‐analysis of randomised trials, covering more than 10,000 cases of cancer, found no effects of statins on the incidence of cancer, nor on deaths from cancer...

The observational data was doubly wrong. Observational data failed to replicate the well‐known finding that statins improve heart health. And observational data wrongly suggested that statins reduce the risk of cancer. Randomised trials, which were not biased by selection effects, provided the correct answer. 

That is only one example of many in the article. However, while Leigh is pro-RCT, he is not anti-big-data. He notes that:

Large data sets are a valuable complement to randomised trials. But big data is not a substitute for randomisation.

If we take anything away from Leigh's article, it should be that point. Big data is incredibly useful. However, it must be analysed using the tools of causal inference (of which randomised controlled trials are just one example) if we want to move beyond finding correlations. The problem with big data is compounded by a focus on statistical significance (as Ziliak and McCloskey noted in their book The Cult of Statistical Significance, which I reviewed here). Big datasets will find statistically significant correlations even when the size of the relationship is very small. That is an asset when causal methods are applied, but is very much a liability when big data are analysed without consideration of causality. RCTs are one way of disciplining our research approach in order to ensure that the effects we estimate are causal, and as Leigh notes:

While correlations in large data sets do not necessarily indicate causation, administrative data can be enormously helpful in ensuring the precision of estimates from randomised trials.

The article finishes with high-level strategies that policy makers and practitioners can use to ensure that RCTs are embedded within the analysis of public policy:

I advocate five approaches. Encourage curiosity in yourself and those you lead. Seek simple trials, especially at the outset. Ensure experiments are ethically grounded. Foster institutions that push people towards more rigorous evaluation. Collaborate internationally to share best practice and identify evidence gaps.

Those all sound like good approaches. I would add a sixth: Employ analysts with a thorough grounding in causal inference methods generally, if not RCTs specifically. We need more policy analysis that establishes causal evidence of impact.

Generative AI is mastering 'metrics

The capabilities of generative AI continue to grow. In the latest example, some enterprising economists have developed an agentic AI that can complete tasks using econometrics (the economist's statistical toolset) - 'mastering 'metrics', as Angrist and Pischke would say (see my review of their excellent econometrics text). The agentic AI approach is outlined in this new working paper by Qiang Chen (Shandong University) and co-authors. As they explain:

We propose and implement a zero-shot learning framework, called Econometrics AI Agent, that enables AI agents to acquire domain knowledge without costly LLM fine-tuning. The framework’s core component is an econometrics “tool library” implementing popular econometric methods, including IV-2SLS, DID, and RDD.

...we augment each econometric tool with detailed “prompts”—comprehensive method descriptions that specify inputs, hyperparameters, and outputs. These prompts are provided alongside corresponding Python implementations, creating a standardized interface between the econometric methods and the AI agent. This design allows the LLM to leverage both its general econometric knowledge and the specifically crafted prompts and tools, enabling it to conduct complex econometric analyses through multi-round interactions with users. The resulting framework empowers Econometrics AI Agent to independently handle applied econometric tasks, delivering comprehensive results that include parameter estimation, inference, and analytical discussions.

The Econometrics AI Agent that Chen et al. created is available here. That site also includes detailed installation instructions, and a helpful demonstration video. Coming back to the paper, Chen et al. show the capabilities of the model by testing it on several real-world problems:

We evaluate the Econometrics AI Agent through two sets of inquiries. The first comprises 18 exercises from the coursework assignments of a doctoral-level course titled “Applied Econometrics” at the University of Hong Kong, with Python-generated standard solutions. These exercises cover OLS & PanelOLS regression, propensity score matching, IV-2SLS regression, Difference-in-Differences (DID) analysis, and Regression Discontinuity Design. The second set consists of test datasets from randomly selected seminal articles in reputable journals, primarily accompanied by Stata-based replication packages.

They compare the performance of their agent, in terms of creating code that works correctly, and in terms of the resulting estimated coefficient of interest, in comparison to three alternatives:

...(i) direct LLM generation in Python code, (ii) direct LLM generation in Stata code, and (iii) baseline general-purpose AI agents without specialized econometric tools and domain knowledge.

So, this is an approach to replication, which is important (for example, see here), and is a point that I will return to later. Overall, in comparison to LLMs and general-purpose AI agents, the Econometrics AI Agent performs much better. In terms of the econometrics coursework assignments, Chen et al. find that:

The Econometrics AI Agent demonstrates superior performance with a 95% directional replication rate and average coefficient value errors below 3%. In contrast, both GPT-generated Python and Stata control groups show incorrect directions in over half of test cases. While the general AI Agent achieves a 78% directional replication rate, its coefficient values frequently deviate significantly from true values.

The rate of 'perfect replication' (which Chen et al. defined as the errors in the coefficient, standard error, and p-value all within 1% of the 'true' value) was 51.85 percent for the Econometrics AI Agent, but less than 30 percent for the other models. Turning to the published paper replications, Chen et al. find that the rate of 'perfect replication' was just 27.41 percent for the Econometrics AI Agent, but that was still far higher than the other models, which all had rates under 18 percent. In relation to those results, Chen et al. note that:

...the Econometrics AI Agent does show room for improvement. For example, its performance declines for complex econometric methods like DID and RDD compared to simpler approaches such as OLS and IV-2SLS. Similarly, results slightly deteriorate when moving from straightforward coursework problems to more sophisticated paper replication tasks. However, these limitations can be addressed through the AI agent’s domain knowledge architecture—specifically by developing customized tools and enhancing prompt instructions to better support complex algorithms and detailed requirements.

Indeed, it is the modular nature of the agent's architecture that may be its key advantage, allowing modules relevant to each econometric task to be added or updated over time. On this point, Chen et al. note that:

Unlike the costly and often infeasible process of fine-tuning an LLM to keep pace with rapid academic advances in developing new techniques, our agent can be updated simply by adding new tool functions and descriptions to the prompt library. This modularity allows the agent’s knowledge base to expand alongside the field’s developments, making the integration of recently published procedures as straightforward as adding new modules.

So, we can expect that the agent's capabilities, and its accuracy, will only improve over time. We may not be very far away from a time when empirical economists spend far less of their time on coding esoteric econometric code in order to extract meaningful results. What will we do with our free time? Maybe we'll be able to turn our attention to a greater variety of research questions.

There is a further positive aspect to these results. The replication crisis is real in many disciplines, including economics. Having AI agents that can automate the steps required to generate econometric results will decrease the time cost of completing replications. That means that we may expect more paper replications in the future (at least, more of the type of replication that Miguel and Christensen call a 'verification'). This move will certainly be a positive, leading to improvements in the quality of research in the future.

[HT: Marginal Revolution]

Read more:

• Using ChatGPT for economics research
• An update on using ChatGPT for economics research