Do research findings in one discipline affect research in other disciplines? This seems like an important question, and one to which we hope the answer is yes. And yet, for a long time disciplines seemed to be becoming more insular, only referencing research within their own narrow fields and sub-fields. More recently, it seems to me that the breadth of citation across disciplines has increased greatly. Call it the 'Google Scholar effect', if you like. Since research has become much more searchable, relevant past research has become easier to find and to cite. If there is a 'Google Scholar effect', we'd expect to see a big change in citation patterns across disciplines starting from 2004, when Google Scholar was introduced.
So, I was interested to read this 2024 article by Sebastian Galiani (University of Maryland), Ramiro Gálvez (Universidad Torcuato Di Tella) and Ian Nachman (Brown University), published in the journal Economic Inquiry (ungated earlier version here). They conduct a citation analysis to look at specialisation trends in economics. Specifically:
...we look for patterns that indicate specialization within a field of economics research, such as a narrowing of the topics it covers in a way that other fields do not, and patterns suggesting a decrease in citations from outside the field...
Their dataset includes:
...all articles published from 1970 up to and including 2016 in the so‐called economics Top 5 journals (The American Economic Review, Journal of Political Economy, The Quarterly Journal of Economics, Econometrica, and The Review of Economic Studies) and for all articles published from 1970 up to and including 2016 in a set of non‐Top 5 general research journals (The Economic Journal, International Economic Review, Economic Inquiry, and The Review of Economics and Statistics)...
Galiani et al. categorise each article into one of four 'fields' of economic research:
...(1) applied, (2) applied theory, (3) econometric methods, and (4) theory... The criteria used to assign a paper to a field were as follows: (1) Applied articles have an empirical or applied motivation. They rely on the use of econometric or statistical methods as a basis for analyzing empirical data, although they may deal with simple models that provide a theoretical framework for the analysis... (2) Applied theory articles construct a theoretical model to explain a fact, with the empirical analysis serving as a supplementary aspect rather than the primary focus of the paper. These articles typically have limited utilization of econometric or statistical analyses, though they may employ simulations (even with empirical data) or refine other techniques to test the implications of the models... (3) Econometric methods articles develop econometric or statistical methodologies... (4) Theory articles lack an empirical section; typically, they approach a topic through modeling and extensive use of formal mathematics and logic. These articles may incorporate a numerical example or a simple model calibration using theoretical data to illustrate the proposed model or to analyze its comparative statics.
The process of assigning articles to a field involves machine learning, which I'm not going to get into here. They also use natural language processing to detect 'latent topics' based on the content of each article, where each article is assigned to exactly ten topics. They use that approach, rather than using keywords or JEL codes because the topics that they identify use similar methods, making them more similar than articles that may have the same keywords but approach the research question in very different ways.
Galiani et al. then look at the long-term trends in the topics, content, and citations (both within and outside of economics) of their large sample of 24,273 articles. The results make for interesting reading, not just in terms of the trends, but in terms of the differences between the different fields of economics:
...theory and econometric methods have shown a narrowing focus on specific research topics since the 1990s, indicating a tendency toward specialization. Theory articles have experienced a significant increase in topics related to formal mathematical proofs and game theory, while econometric methods articles have shown a pronounced rise in topics related to computational statistics, estimators' asymptotic properties, and estimators' bounds. In contrast to applied papers, these fields do not exhibit rising trends in extramural citations (i.e., citations from other disciplines) and in citations from other fields within economics research (in the case of econometric methods, it shows a declining trend). These patterns also indicate a higher degree of specialization.
Applied papers have expanded their coverage to include diverse topics. Applied articles have seen a pronounced rise in topics related to impact analysis, causal analysis, and experimental economics. Over time, applied articles began to receive a higher proportion of citations from external fields, especially from disciplines such as medicine, psychology, law, and to a certain extent, education... Overall, these patterns indicate that applied papers are becoming more multidisciplinary. The case of applied theory articles is less conclusive. While they cover a broader range of topics (similar to applied papers), there has been no significant increase in extramural citations or citations from other fields of economics research (as observed with theory articles).
The observation that applied papers have received an increasing proportion of citations from outside economics over time is particularly interesting. Applied papers are more easily cited outside of economics because they often address research questions that other disciplines are also interested in. Galiani et al. identify that applied articles in economics are often cited in the education, medicine, and psychology fields, more so than articles from other fields of economics. This pattern of extramural citations does seem to become more apparent from the 1990s onwards, which would seem to suggest that Google Scholar (which was introduced in 2004) is not implicated, despite my expectation of a 'Google Scholar effect'.
That economic theory articles are not well cited outside of economics, and even outside of economic theory articles, will come as little surprise. Economic theory articles tend not to cover topics that are of interest outside of economics, and when they do, they are far too technical for those outside of economics to appreciate.
On the other hand, econometric methods should have a greater impact outside of the field, and it is disappointing that those papers do not seem to have an external impact. There are advances in econometrics, particularly in causal inference, that other disciplines should take advantage of. In contrast, economics increasingly involves methods drawn from applied statistics and computer science (including the machine learning and natural language processing that Galiani et al. used in this research).
Overall, the takeaway from this research is that economics is simultaneously becoming more specialised (within economic theory and econometric methods) and having a greater impact outside of the field (for applied papers). It would be interesting to see whether similar results would come from a similar analysis of articles in political science or psychology. If political science or psychology show the same pattern, that would tell us something important about how methods and results diffuse, or how they don't.
[HT: Marginal Revolution, last year]
No comments:
Post a Comment