I've written a couple of times about the impact of ChatGPT on the labour market (see here and here). Both times, I noted that we needed to investigate real-world cases of the impact of large language models on particular jobs. To understand why that is important, it is worth first considering why we can't be sure about the impact of a particular technology on employment.
New production technologies (by which, I mean any technology that is used by workers) make workers more productive. That could have one of two effects on employment among the workers affected by the technology. On the one hand, more productive workers generate more production for their employer. It increases the marginal product of labour (the amount of production that the marginal worker generates, or the amount of additional production that the employer receives by employing one more worker [*]). Assuming that doesn't affect the price of the output that the workers produce, making workers more productive increases the value of the marginal product of labour (which is the marginal product of labour multiplied by the value of the output produced [**]). Since workers are generating more value for the employer, the employer would want to employ more of them, and keep adding additional workers until the value of the marginal product of labour is equal to the wage. So, in this case, new technology increases employment - technology is labour-augmenting.
On the other hand, if the employer has a fixed amount of production to generate, then have more productive workers means that fewer workers are required in order to complete all of the production. In that case, new technology decreases employment - technology is labour-replacing. However, displaced workers may find employment elsewhere, and the technology may create new job opportunities. So, even if technology is labour-replacing, there is no certainty that it reduces employment overall.
Is ChatGPT labour-augmenting, or labour-replacing? It is too early for us to tell definitively. We need to look at the data. One thing we can be pretty sure about is that generative artificial intelligence (AI), of which ChatGPT and large language models are just one example, is going to increase worker productivity in some jobs. The work I cited in those earlier posts gave us a good idea of which jobs were most exposed, and this new NBER Working Paper (ungated version here) by Erik Brynjolfsson (Stanford University), Danielle Li, and Lindsey Raymond (both MIT), tells us about the potential productivity gains.
Specifically, Brynjolfsson et al. used the staggered introduction of a generative AI tool in a contact centre setting to look at its effects on worker productivity. As they explain:
We examine the staggered deployment of a chat assistant using data from 5,000 agents working for a Fortune 500 software firm that provides business process software. The tool we study is built on a recent version of the Generative Pre-trained Transformer (GPT) family of large language models developed by OpenAI... It monitors customer chats and provides agents with real-time suggestions for how to respond. It is designed to augment agents, who remain responsible for the conversation and are free to ignore its suggestions.
Brynjolfsson et al. look first at the effects of the chat on the number of chats that an agent is able to successfully resolve per hour. They find that:
...deployment of AI increases RPH [Resolutions Per Hour] by 0.30 calls or 13.8 percent.
They also find:
...a 3.8 minute decrease in the average duration of customer chats, a 9 percent decline from the baseline mean (shorter handle times are generally considered better)... a 0.37 unit increase in the number of chats that an agent can handle per hour. Relative to a baseline mean of 2.6, this represents a roughly 14 percent increase. Unlike average handle time, chats per hour accounts for the possibility that agents may handle multiple chats simultaneously. The fact that we find a stronger effect on this outcome suggests that AI enables agents to both speed up chats and to multitask more effectively... a small 1.3 percentage point increase in chat resolution rates, significant at the 10 percent level. This effect is economically modest, given a high baseline resolution rate of 82 percent; we interpret this as evidence that improvements in chat handling do not come at the expense of problem solving on average. Finally... no economically significant change in customer satisfaction, as measured by net promoter scores: the coefficient is -0.13 percentage points and the mean is 79.6 percent.
All of that suggests that the contact centre workers are more productive. However, the effects are heterogeneous by skill level and tenure. When they stratify their sample by skill level (as measured in the quarter prior to adoption of the AI system), Brynjolfsson et al. find that:
...the productivity impact of AI assistance is most pronounced for workers in the lowest skill quintile... who see a 35 percent increase in resolutions per hour. In contrast, AI assistance does not lead to any productivity increase for the most skilled workers...
And similarly, when they stratify their sample by tenure:
We see a clear, monotonic pattern in which the least experienced agents see the greatest gains in resolutions per hour.
Taken all together, this means that the AI system narrows the gap between high-quality and low-quality workers, and the gap between workers with more experience and workers with less experience. In fact, Brynjolfsson et al. note that:
AI helps new agents move more quickly down the experience curve... agents with two months of tenure and access to AI assistance perform as well as or better than agents with more than six months of tenure who do not have access.
When Brynjolfsson et al. look at the actual text of interactions between agents and customers, they find:
...suggestive evidence that AI assistance leads lower-skill agents to communicate more like high-skill agents.
They also find suggestive evidence that customers are happier (as measured by the sentiment in the conversations), and that the improvements in sentiment are greater for workers with lower skill, and workers with less tenure. Finally, we learn that:
...on average, the likelihood that a worker leaves in the current month goes down by 8.6 percentage points... We find the strongest reductions in attrition among newer agents, those with less than 6 months of experience. The magnitude of this coefficient, around 10 percentage points, is large given baseline attrition rates for newer workers of about 25 percent... we find a significant decrease in attrition for all skill groups, but no systematic gradient.
So, the employer has fewer workers leaving (lower attrition), but the effect on employment also depends on how many new workers they hire as well, which we don't know. The last point, about no differences in attrition by skill, is important. That's because, earlier in the paper, Brynjolfsson et al. note that:
Agents are paid an hourly wage and bonuses based on their performance relative to other agents.
If the workers are paid for their performance relative to other agents, and the AI makes lower-quality and shorter-tenured agents perform better, that will tend to increase the wages of lower-quality and shorter-tenured agents, and consequently decrease the wages of higher-quality and longer-tenured agents. So, I was a little surprised that there wasn't more attrition among the higher-skilled workers at least. And that would be a problem looking forward, because it is the interactions between high-quality workers and customers that you want to use to train future AI models. If the higher-skilled workers leave, then there will be lower-quality training data available.
So, what we learn from this paper is that there are relatively large productivity increases among these contact centre workers. And those workers are clearly satisfied with their job changes, as they don't leave their job as often. Those two effects mean more profits for the employer (better quality customer service, and lower costs of replacing workers).
Coming back to the question we started this post with, what does that mean for employment overall? Unfortunately, we don't get a straight answer to that question. Fewer workers are leaving the employer, but we don't know if the employer is offsetting that by hiring fewer new workers. It looks like we're going to have to wait for additional future research in order to better understand the impacts of generative AI on employment.
[HT: Marginal Revolution]
Read more:
*****
[*] Or, the amount of additional production that the employer receives from one additional labour hour. It doesn't really matter which way we define the marginal revenue product. The rest of the explanation works the same. I just find it a bit easier to talk about marginal productivity in terms of whole workers, rather than labour hours.
[**] This is also known as the marginal revenue product of labour.
No comments:
Post a Comment