A picture of the labour market impacts of generative AI is slowly emerging. At this stage, there is little consensus on what the impacts will be. I just stumbled across this working paper, by Jonathan Hartley (Stanford University) and co-authors, which I had put aside to read earlier this year. Unlike some of the research I have discussed in recent posts (linked at the end of this post), Hartley et al. make use of a nationally representative survey of US workers.
The survey has had three waves in the US (plus one Canadian wave), and the first US wave had over 4200 respondents (Hartley et al. don't report how many respondents there were for the other waves). The results make for interesting reading. First, in terms of who is using generative AI, they report that:
...LLM adoption at work among U.S. survey respondents above 18 has increased rapidly from 30.1% as of December 2024, to 43.2% as of March/April 2025, and to 45.9% as of June/July 2025...
Conditional on using Generative AI at work, about 33% of workers use Generative AI five days per week at work (every weekday). Roughly 12% of Generative AI users use such tools at work only 1 day at work. About 17% and 18% of Generative AI users use Generative AI tools at work two and three days per week respectively...
That is a lot of people using generative AI for work, and using it often when they do. It is interesting to sit these results alongside those of Chatterji et al. (whose paper I discussed in this post). They found growth in both work-related and non-work-related ChatGPT messages over time.
Who is using generative AI at work, though? Hartley et al. find that:
...Generative AI tools like large language models (LLMs) are most commonly used in the labor force by younger individuals, more highly educated individuals, higher income individuals, and those in particular industries such as customer service, marketing and information technology.
These results are similar to those of Chatterji et al., except that Hartley et al. also report gender differences (with greater use of generative AI by men), whereas Chatterji et al. report that the gender gap that was apparent among early adopters of ChatGPT has closed completely.
Hartley et al. then move on to estimating the productivity gains from generative AI. Given that this is survey-based, and not observational or experimental, we should take these results with a very large grain of salt. Hartley et al. ask their respondents how long it takes then to complete various tasks with and without generative AI. The results are summarised in Figure 12 in the paper:
Notice that every task is reported to take less time with generative AI (the green dots) than without (the blue dots). The productivity gains are different for different tasks. However, I find this figure and the data to be very fishy. How could generative AI create a huge decrease in time on 'Persuasion' tasks? Or 'Repairing' (which has one of the biggest productivity gains). Also, notice how almost every task takes between 25 and 39 minutes with generative AI. I strongly suspect that the research participants are anchoring their responses to this question on 30 minutes with GenAI for some reason. Without seeing the particular questions that are being asked though, it is hard to tell why. [*]
Hartley et al. then try to estimate the impact of generative AI on job postings, employment, and wages, using a difference-in-differences research design. They find no impact on job postings or employment, but significant impacts on wages. However, here things get strange. The coefficients that they report in Tables 6 and 7 of the paper are clearly negative, and yet Hartley et al. write that:
Our estimated coefficients... imply economically meaningful wage effects: a one-standard deviation increase in occupational Generative AI exposure corresponds to a significant increase in median annual wages...
Going back to their regression equations, their 'exposure to generative AI variable' is more positive when exposure is high, so a negative coefficient should imply that more exposure to generative AI is associated with lower wages. I must be missing something?
Given the deficiencies in the data and the regression modelling, I don't think that this paper really adds much to our understanding of the labour market effects of generative AI. Which is disappointing, because survey-based evidence would provide us with a complementary data source that would help us to triangulate with the results from other data sources and methods.
[HT: Marginal Revolution]
*****
[*] On a slightly more technical note, we might expect there to be as much variation (in relative terms) in the 'with GenAI' data as in the 'without GenAI' data. However, the coefficient of variation (the standard deviation expressed as a percentage of the mean) is 0.109 for the 'with GenAI' data, but 0.226 for the 'without GenAI' data. So, there is less than half the variation in the reported task times with GenAI than without. Again, that suggests that this data is fishy.
Read more:
No comments:
Post a Comment