ChatGPT as an academic author

ChatGPT and subsequent large language models are going to change the world of writing. Having a tool that reduces some of the drudgery of drafting writing will be a big help for people who do a lot of writing. Academics do a lot of writing, so perhaps there are opportunities for ChatGPT to contribute to academic writing. I've played around a little, and I can see that these tools would be useful for writing quick outlines that can be re-written later. The extent of hallucination and made-up facts makes ChatGPT infeasible to fully replace the traditional task of writing for now. The good and the bad of ChatGPT as an academic author were highlighted earlier this week, when Gail Pacheco gave us an example of a spoof economics paper written by ChatGPT, on the Asymmetric Information blog.

I suggest reading Gail's ChatGPT paper. The prompt she used was "Can you write a fake economics paper on ‘The efficiency of using AI to write an economic paper’, including a table and references". ChatGPT fulfilled the requirements to the letter. However, the statistics were entirely made up, as were the references (and Gail even checked them!). Putting aside the hallucinated facts, the writing itself was fairly good, and possibly indistinguishable in terms of quality from some papers that I have reviewed, written by PhD students and/or authors whose first language is not English. This suggests that, as I suspected, ChatGPT might be a good tool for creating a first draft, or a broad outline, of a paper.

How much can ChatGPT help? For now, not much. It can't draw on the real academic literature, so literature review and introductory sections will require quite a bit of work. However, in future that might be addressed if large language models have access to Google Scholar or web searches.

Also, the made-up facts and statistics are problematic. However, if future large language models can draw from actual statistics generated by the human author using statistical software, then suddenly the task of writing papers changes dramatically. Imagine exporting regression tables from Stata or R, handing them over to ChatGPT, and having ChatGPT write a serviceable first draft of the results section of a paper, or a discussion of the results linking back to the previous literature cited in the earlier sections of the paper. That would be a game changer.

Journal editors (of which I am one) had better be ready for a flood of AI-generated papers (with real analysis included, or not!). It's already happening in fiction publishing. Academic publishing will not be too far behind. For now, the quality of those papers will be pretty bad, and hopefully they won't pass the desk-reject stage. If they do, competent reviewers should be able to recognise any made-up statistics or references, and reject the paper. That will not be quite so simple in the future.

Artificial intelligence is going to not only change teaching and assessment, but it is going to change the way that we communicate our research. The robots are coming for our (academic) jobs!