Using ChatGPT for economics research

As I posted last month, ChatGPT convincingly passed the Test of Understanding of College Economics. Maybe ChatGPT is coming for the jobs of economists. In the meantime though, we can use it as a tool to improve our productivity in teaching and research. Two new papers show the way.

The first paper is this one by Kevin Bryan (University of Toronto), which is essentially a bunch of notes from a talk that Bryan gave at Princeton last month. Bryan offers a 'user's guide' to ChatGPT for economics research. Bryan summarises the main takeaways of his user's guide as:

1. Controlling the output of LLMs is difficult

2. The “Raw” ChatGPT online is far from state of the art

3. Hallucinations are mostly fixable (this guide includes some tips to avoid them)

4. The technology’s rate of improvements is fast

5. You should get access to the API. Most use cases for economists require using the API and accompanying with code. This will give you much more control on the output, and it is cheap to do so

Bryan then offers some examples of four uses of ChatGPT for economists:

1. Cleaning data

2. Programming and making graphs

3. Spelling and grammar checks

4. Summarizing literature

The examples are interesting and a useful starting point, but a little bit obvious to anyone who has used ChatGPT. Also, Bryan doesn't provide the associated ChatGPT prompts, which would have made them even more useful. An even better, and more comprehensive coverage, of uses of ChatGPT for economics research is provided in this new NBER Working Paper (alternative ungated version here), by Anton Korinek (University of Virginia). Korinek lays out:

...six different areas in which LLMs can be useful. In the process of ideation, LLMs can help to brainstorm, evaluate ideas, and provide counterarguments. In writing, they can synthesize text, provide examples, edit and evaluate text, and generate catchy tweets or titles for a paper. In background research, they can be useful for searching and summarizing the literature, translating text, explaining concepts, and formatting references. LLMs are also very capable in coding, writing code based on instructions in natural language, explaining code, translating code between programming languages, and even debugging code. For data analysis, LLMs can extract data from text, reformat data, classify text, extract sentiment, and even simulate humans to generate data. Finally, LLMs are starting to display emergent capabilities in mathematical derivations, starting from setting up models and working through derivations to explaining models.

Korinek provides examples that include both the prompts and ChatGPT's response, which helps to better illustrate its capabilities. Like Bryan, some of Korinek's examples are straightforward and obvious. However, some of them will be huge timesavers, especially:

Once references are found and it is verified that they are not hallucinated, LLMs are very capable of formatting them in the desired manner...

No more adjusting from one referencing standard to another for me (although it has to be said that EndNote and similar tools automate this already, but entering the references into EndNote can be time consuming in itself). Generating catchy titles and headlines is also going to prove really helpful (and no, I didn't use it for the title of this post). However, ChatGPT is not equally useful for every task. Korinek provides a helpful rating of ChatGPT for each task, in terms of its usefulness (1 = experimental; 2 = useful; 3 = highly useful). Here is Table 1 from the paper:

Those usefulness ratings are as of February 2023. ChatGPT and other large language tools are improving rapidly in their usefulness. By the time you read this post, it is likely that ChatGPT will have improved in its ability to perform one or more of those tasks beyond the point it was at when Korinek was testing it (based on text-davinci-003, which is slightly better than ChatGPT, but not as advanced as the more-recently-released GPT-4).

Korinek's paper also provides some optimism for economists who may worry about their future job security:

Ultimately, I believe that the most useful attitude towards the current generation of LLMs is to heed the lessons of comparative advantage that Ricardo taught us two centuries ago: LLMs increasingly have comparative advantage in generating content; humans currently have comparative advantage in evaluating and discriminating content. LLMs also have super-human capabilities in processing large amounts of text. All this creates ample space for productive collaboration...

If anyone should not forget the lessons of comparative advantage, it is economists.

[HT: Marginal Revolution for the Korinek paper, and David McKenzie at the Development Impact blog for the Bryan paper]