It seems it's better to publish economics papers with U.S. data

This post follows on from yesterday's post about the length of article titles, which showed a strong negative correlation between title length and research quality (so papers with shorter titles were more likely to be published in better journals, and attracted more citations). There are, of course, a lot of other factors that affect where journal articles get published. One gripe for many researchers outside the U.S. or the U.K. is how hard it is to get published in top journals using data from outside the U.S. or the U.K. Until relatively recently, that gripe was based on purely anecdotal evidence. However, a 2013 article by Jishnu Das, Quy-Toan Do, Karen Shaines (all from the World Bank), and Sowmya Srikant (FI Consulting), published in the Journal of Development Economics (ungated version here) provides some empirical evidence on this.

Das et al. use data on over 76,000 papers published in 202 economics journals over the period from 1985 to 2005, and the disparity in data sources for published economics papers is clear:

Over the 20 year span of the data, there were 4 empirical economics papers on Burundi, 9 on Cambodia and 27 on Mali. This compares to the 37,000 or so empirical economics papers published on the U.S. over the same time-period. This variation is also reflected among the highly selective top-tier general interest journals (henceforth top-tier journals) of the economics profession (American Economic Review, Econometrica, The Journal of Political Economy, The Quarterly Journal of Economics and The Review of Economic Studies). American Economic Review has published one paper on India (on average) every 2 years and one paper on Thailand every 20 years. The first-tier journals together published 39 papers on India, 65 papers on China, and 34 papers on all of Sub-Saharan Africa. This compares to 2383 papers on the U.S. over the same time period.

They then go on to show about 75 percent of the cross-country variation in the geographical focus of research is explained by GDP per capita and by population. Countries that have higher GDP are more likely to be the focus of research. This is a disappointing result if you are interested in developing countries (as the authors of the paper clearly are). Surprisingly though:

...the U.S. is not an outlier in the volume of research that is produced on it... the volume of research for the U.S. lies well within the predicted confidence interval and excluding the U.S. leads to the same coefficient estimates as its inclusion. In other words, a lot more is produced on the U.S. because it is rich and it is big; the natural comparator for the U.S. would be all of Europe and here, the volume of research is very similar.

However, when it comes to the elite journals, the U.S. is a clear outlier:

The difference between the U.S. and the rest of the world is substantial — 6.5% of all papers published on the U.S. are in the top tier journals relative to 1.8% of papers from other countries.

For comparative purposes, over their 20-year period there were 2,383 publications on the U.S. published in the top five economics journals, and one on New Zealand (yes, you read that right, just one - I don't know which article it was, sorry).

So, is this discrimination against non-U.S. research? Perhaps. Or, it could be as simple as the U.S. having a greater density of top-quality researchers, who are more likely to publish in top-quality journals, and who, because they are located in the U.S., have readier access to U.S. data. Or, perhaps the quality of U.S. data is higher. Das et al. point out that data quality has a superstar effect to it (similar to the superstar effects in the labour market that I have written about before):

Researchers converge on the “best” dataset even if it is 1% better than other data available, and the initial work creates a focal point for further research with the same data.

Again, like the paper I discussed yesterday, there isn't necessarily a causal interpretation to these results (doing research on the U.S. doesn't necessarily cause papers to be accepted into top journals). But it is disappointing, particularly given the quality of linked administrative data that we have in New Zealand through the Integrated Data Infrastructure, which (I think) should be attractive for publication in top journals.

Title length and research quality

There are a number of quirks that seem to be related to research quality, perceived research quality, or the academic rewards from research. In a new paper published in the Journal of Economic Behavior and Organization (sorry I don't see an ungated version online), Yann Bramoullé (Aix-Marseille University) and Lorenzo Ductor (Middlesex University London) explore the relationship between the length of a journal article's title and measures of its quality. Using a dataset of half a million articles in economics journals over the period from 1970 to 2011, they demonstrate a number of interesting things:

Articles with shorter titles tend to be published in better journals. They tend to be more cited and to get higher novelty scores. Moreover, these tendencies are more pronounced in better journals... Moreover, including novelty in the regressions on journal quality and citations has essentially no impact on the estimates. This means that the observed relation between title length and citations is not explained by the fact that novel articles tend to have shorter titles and to be more cited. Together, these results show that title length correlates well with the overall scientific quality of a paper.

So, there is a strong negative relationship between title length and the quality of the article, even after controlling for the quality of the authors, and the quality of the publication. Bramoullé and Ductor have obviously taken this on board, as their article title ("Title length") is as short as possible. But why would title length be related to research quality? Bramoullé and Ductor suggest a couple of possible explanations:

On one hand, title length could have a causal impact on journal quality or citations. A short title could make an article easier to memorize, affecting citations and, possibly, editorial decisions... On the other hand, title length could proxy for the true, unobserved qualities of the article. Articles with a strong potential to influence subsequent research could thus both generate more citations and have shorter titles.

The second explanation seems more intuitive. Many (maybe all?) highly cited papers are those that steer research in new directions. For example, the Kahneman and Tversky paper that I mentioned yesterday, entitled "Prospect Theory: An Analysis of Decision under Risk" (note: 51 characters including spaces and the colon), is the most cited paper in economics (with over 8000 citations in these author's dataset, and over 50,000 in Google Scholar). Subsequent papers in the new research area opened up by the highly-cited papers are likely to have slightly longer titles, as they build on the original theory, refute it, apply it to new areas or new sub-fields, and so on. However, that is of course a causal interpretation, and the authors' results don't establish causality here. Nevertheless, an interesting quirk - I wonder if it extends to blog posts?

[Update]: I forgot to mention that the authors do attempt to control for novelty in their analysis, which would seem to mitigate against the intuitive explanation in my last paragraph. However, I don't find their measure of novelty (the number of 'atypical' keywords the article uses) to be particularly convincing, since novel articles may well use keywords that themselves are common, even though the content is not.

[HT: Marginal Revolution]

Book Review: The Undoing Project

I've never read a Michael Lewis book before, not even Flash Boys or Moneyball (though I have seen the movie of the latter). To be clear, I haven't been actively avoiding his writing but like Malcolm Gladwell, his books get so much press that I feel like I've read them without actually reading them. The Undoing Project is different. I have read a few short reviews, but not enough to give away the bulk of the content. In it, Lewis tells the story of two psychologists who have had a substantial impact on economics - 2002 Nobel Prize winner Daniel Kahneman, and Amos Tversky, who surely would have shared Kahneman's Nobel if not for his untimely passing in 1996.

I really enjoyed Lewis's writing style, which I would describe as a flowing biographical narrative. It is easy to see why so many of his books have been made into movies, and this one would also lend itself to the screen. There are a lot of quirky stories embedded within it. For instance, take this bit:

Apart from that short note, Amos seldom mentioned his army experiences, in print or conversation, unless it was to tell a funny or curious story - how, for instance, during the Sinai campaign, his battalion captured a train of Egyptian fighting camels. Amos had never ridden a camel, but when the military operation ended, he won the competition to ride the lead camel home. He got seasick after fifteen minutes and spent the next six days walking the caravan across the Sinai.

Similarly, Lewis paints a hilarious picture of Kahneman and Tversky tearing around the Sinai desert in a jeep, surveying Israeli troops during the war. The anecdotes add a lot of value to the story.

However, the book didn't add much, if anything, to my understanding of behavioural economics. If you want a good treatment of that, you would be much better off with Richard Thaler's excellent Misbehaving (which I reviewed earlier this year). However, Lewis does offer a really clear and thorough description of the development of Kahneman and Tversky's collaboration, from how it began, through its peak with the development of Prospect Theory, and onto the decline in their relationship after they both moved from Hebrew University in Israel to North American institutions. I especially liked this bit, on their choice to move on from regret minimisation as a theory:

Amazingly, Danny and Amos did not so much as pause to mourn the loss of a theory they'd spent more than a year working on. The speed with which they simply walked away from their ideas about regret - many of them obviously true and valuable - was incredible. One day they are creating the rules of regret as if those rules might explain much of how people made risky decisions; the next, they have moved on to explore a more promising theory, and don't give regret a second thought.

That new theory was Prospect Theory, which Lewis notes was named as such "purely for marketing purposes", so that it would be distinct (they originally labelled it "value theory"). Interestingly, Lewis avoided the temptation to talk about their lack of regrets when they moved on from regret as a theory (but see, I couldn't resist - that's lazy blog writing, that is).

One thing that comes clearly through in the book is how surprising (as well as how intense) the partnership between Kahneman and Tversky was. Take this bit on their differences:

Danny was a holocaust kid; Amos was a swaggering Sabra - the slang term for a native Israeli. Danny was always sure he was wrong. Amos was always sure he was right. Amos was the life of every party; Danny didn't go to parties. Amos was loose and informal; even when he made a stab at informality, Danny felt as if he had descended from some formal place. With Amos you always just picked up where you left off, no matter how long it had been since you last saw him. With Danny there was always a sense you were starting over, even if you had been with him just yesterday. Amos was tone-deaf but would nevertheless sing Hebrew folk songs with great gusto. Danny was the sort of person who might be in possession of a lovely singing void that he would never discover. Amos was a one-man wrecking ball for illogical arguments; when Danny hear an illogical argument, he asked, What might that be true of? Danny was a pessimist. Amos was not merely an optimist; Amos willed himself to be optimistic, because he had decided pessimism was stupid. When you are a pessimist and the bad thing happens, you live it twice, Amos liked to say. Once when you worry about it, and the second time when it happens.

That last bit made me laugh out loud, because I've had that exact conversation with my wife on more than one occasion, and my view accords with Amos's. The book does jump around a little bit in time, which I found a little disconcerting. However, that betrays my preference for linearity in the storyline. Overall, this was an excellent read, and I'm looking forward to cracking open some other Michael Lewis books in the future (in spite of thinking I already know what they say).

Could closing the gender gap in economics be as simple as providing students with information?

In a new paper published in the journal Economics of Education Review (sorry I don't see an ungated version anywhere), Hsueh-Hsiang Li (Colorado State University), reports on a randomised controlled trial that she ran in the introductory economics classes at Colorado State. Specifically:

During the semester, treatments such as the provision of information on career prospects, average earnings, and grade distributions were provided to women in the treatment group. A nudging message was also sent to female students in the treatment group with a midterm grade above the median. Additionally, half of the treated female students were invited to attend mentoring activities throughout the semester.

Few eligible female students (around 5%) took up the mentoring, so that doesn't explain the effects, which were that:

The treatment effect of interventions on female students with grades above the median is substantial. The treatments increase the probability of these female students majoring in economics by 5.41 – 6.27 percentage points. The effects are even larger for freshmen and sophomores among these high-performing female students, who are 11.2 – 12.6 percentage points more likely to declare economics as their major.

To summarise, there was no treatment effect for below-median-grade female students, in terms of whether they went on to major in economics. The effect was entirely concentrated among above-median-grade female students. Interestingly, the effect of the intervention was actually negative for male students, which Li explains:

 Conversely, the information treatment appears to reduce male students’ likelihood of declaring economics as their major by 2.67 percentage points. The effect is larger (−5 percentage points) among male students in the lower classes (i.e., freshmen and sophomores). Because male students in the treated group received information about careers in the economics profession as well as the grade distribution information with no nudges, the negative effect is likely attributable to their reaction to the grade information... students were overly optimistic about their grade performance upon entering the class. The overconfidence is particularly pronounced among male students.

Overconfident male students obviously get the message that they aren't nearly as good at economics as they thought, when given more information about how the rest of the class is performing, and respond by being less likely to take an economics major. Unsurprisingly, this effect was concentrated among male students below the median grade (and the effect disappeared once Li controlled for students' GPA). On the other hand, female students (at least, those above the median grade) respond to the information by being more likely to take an economics major.

The results are interesting. However, I was more interested that prior to the intervention the information on grade distribution was not available to students. At Waikato, until we shifted to Moodle this year, students in economics papers could previously see detailed information on the grade distributions for each assessment (Moodle doesn't have this functionality, but for the tests and exam I routinely provide the mean, median, top mark, and pass rate to students, so at least there is some information). Encouraging top students in the introductory economics class to enrol in an economics major also seems like a no-brainer, and something that most schools would already routinely do.

So, in spite of the large and statistically significant effects in this study, it seems to me that the results are not generalisable to other settings because the intervention is so routine. However, if your university isn't already doing these things, then now is a good time to start!

Read more:

• Could role models close the gender gap in economics?