This week in research #108

Here's what caught my eye in research over the past week:

• Biehl, Neto, and Gomez (with ungated earlier version here) find that, in Florida, the opening of a Dollar General store drives some nearby firms out of business but provides positive spillovers in terms of revenues and employment for the firms that survive
• Butler, Butler, and Singleton (open access) find that referees add substantially more time in the second half than the first half of games at the FIFA World Cup and UEFA European Championship, and that referees allow more stoppage time when the score is close in the second half, but only at the World Cup
• Moyo and Gwatidzo (open access) find, using the synthetic control method, that although hosting the FIFA World Cup in 2010 had no positive effects on GDP in South Africa, it significantly increased tourism inflows
• Botha and de New (open access) find that the apparent decrease in measured financial literacy in Australia between 2016 and 2020 was entirely a consequence of moving from in-person to telephone interviews
• Sokolov and Libman (open access) conduct an online “beauty contest” experiment with a sample of academic economists in Russia, and find that economists who rely on theories assuming common knowledge of rationality do not expect more rational behaviour from their colleagues (so, even economists who believe in rationality don't believe that other economists are rational)

'First in family' as a measure of disadvantage in higher education

In higher education policy circles, it is an article of faith that students who are the first in their family (usually in the sense that neither their parents, nor any older siblings, has already studied at university) appear to be at higher risk of being unsuccessful in university education. The rationale links to Bourdieu's concept of social capital - students being able to tap into who they know (their family) and importantly what their family knows (about university study) matters. Family members with past university experience can help with university-specific knowledge - things like how to choose majors, manage workload, seek extensions, interpret feedback, and navigate various university systems and processes. This all makes the challenges of studying at university a little easier.

So, I was surprised to learn from this 2020 article by Anna Adamecz-Völgyi, Morag Henderson, and Nikki Shure (all University College London), published in the journal Economics of Education Review (ungated earlier version here), that there is actually limited empirical evidence supporting first-in-family as an indicator of disadvantage. It is that empirical gap that Adamecz-Völgyi et al. attempt to fill, but the interesting thing about this paper is not so much that they find support for first-in-family as a measure of disadvantage, but the mechanism through which it works.

Adamecz-Völgyi et al.  use data from 7707 students from the Next Steps (formerly the Longitudinal Study of Young People in England, LSYPE), which followed a cohort of young people born in 1989/1990. The 'age 25' wave of that study captures most of the cohort after they have completed university education. Adamecz-Völgyi et al. look at various measures of disadvantage, and how well they predict students participating in, or graduating from, higher education. Aside from first-in-family, their battery of disadvantage measures (which they refer to as Widening Participation (WP) measures) includes whether the student had special education needs at high school, whether they were eligible for free school meals, whether their parents were of low social class (based on occupation), whether their family was part of the 20 percent most deprived families (based on a measure of deprivation), whether they had care responsibilities while at high school, whether they were non-white ethnicity, whether they have a disability, whether they lived in a single-parent household, whether they had ever been in care, and whether they lived in an area of high socioeconomic deprivation.

Adamecz-Völgyi et al. use a few different methods to establish whether first-in-family (which they refer to as 'potential FiF', because they only have data on parental education, and not the education of older siblings) is a good predictor of disadvantage (in terms of participating in, or completing) in higher education, including: (1) comparing the predictive power of each variable in separate models (compared using the 'Area under the Receiver Operating Characteristic' curve (AUC)); (2) looking at whether adding first-in-family to a model that already includes a parsimonious set of other measures of disadvantage improves predictions; and (3) using a 'random forest' model to rank the predictors in terms of importance. The AUC is a measure of how often the model correctly predicts a binary variable (in this case, whether a student enrols/does not enrol in university, or whether they do/do not complete university). The random forest model identifies which variables are the most important by running many regressions with different selections of variables. In their analyses, Adamecz-Völgyi et al. find that:

When we compare potential FiF to other WP indicators, it emerges as the most important measure until we condition on prior attainment and all measures end up similarly predictive. We provide evidence that the effects of family background manifest in educational attainment at an early age and pre-university educational attainment is the most important channel of the effect of parental education on HE participation and graduation.

So, this research supports the common belief that first-in-family is a good measure of disadvantage. Moreover, it shows that first-in-family picks up some dimension of disadvantage that other common measures do not. However, the mechanism through which first-in-family affects higher education participation and success is almost entirely through the students' success in pre-university education. Students who are first-in-family at university tend to have worse performance in high school, and that largely accounts for their lower performance in university. Adamecz-Völgyi et al. conclude that:

...being potential FiF (and having social and economic disadvantages in general) matters all along the production function of a child's human capital from early childhood to university. Thus, the educational achievement measures that a university can use are contaminated by this pre-existing disadvantage carried along since early childhood (or probably, since birth). They do not reflect the child's true capacity, but rather the interaction of their innate abilities and family circumstances. Thus, WP measures that simply favour the disadvantaged student out of two students having the same level of pre-university attainment are not enough to widen participation: on average, those from disadvantaged backgrounds are not going have the same pre-university educational attainment levels than those from advantaged backgrounds. The attainment gap must be addressed explicitly by CA [Contextual Admissions] measures.

Instead, I see two ways that universities may respond to these results. Conditional on pre-university educational attainment, first-in-family students do not have worse higher education outcomes than other similarly-prepared students. The reason first-in-family students do less well on average is that they tend to enter university with lower prior educational attainment, and it’s that pre-university gap that accounts for most of the observed difference. On one hand, as Adamecz-Völgyi et al. argue, first-in-family is an indicator of disadvantage, and from a social justice perspective universities should try to mitigate sources of disadvantage whenever they are apparent. On the other hand, these results could be read as suggesting that universities shouldn't worry about first-in-family students, because they perform as well as otherwise similarly-prepared students. The problem is the lack of pre-university educational attainment, and that needs to be addressed in pre-university education, not at university. University-level support may help at the margin, but it risks being an ambulance at the bottom of the cliff. Moreover, two otherwise similar students in terms of pre-university educational attainment could be treated very differently under targeted support policies (such as Contextual Admissions) when one is first-in-family and the other is not, raising issues of fairness.

I'm not going to take a stand on which of those two perspectives (social justice or fairness) is more important. They both have merit. If you accept first-in-family as a measure of disadvantage, the actionable question is whether universities can cost-effectively close preparedness gaps after entry, or whether they should rely on advocating for changes in pre-university education. At least, this research can provide us with confidence that first-in-family is indeed a suitable measure of disadvantage in higher education.

Lessons from Joshua Gans on AI for economics research

I've been increasingly using generative AI (specifically, ChatGPT) to assist with research. I've been quite cautious though, worrying a lot about the quality of AI output, although my worries reduced substantially once ChatGPT started linking to its sources. However, I know many other researchers are using generative AI far more extensively and directly in their research than I am. My approach continues to be to use ChatGPT as an enthusiastic, but not fully polished, research assistant. Given my experience so far, I was interested to read the reflections of Joshua Gans on his year of using generative AI for economics research. His approach was:

I had lots of ideas for papers that I hadn’t developed, so I decided to spend the year working my way down the list. I would also add new ideas as they came to me. My proposed workflow was all about speed. Get papers done and out the door as quickly as possible, where a paper would only be released if I decided I was “satisfied” with the output. So it cut any peer reviews or discussions out during the process of generating research quickly, but I would send those papers to journals for validation. If I produced a paper that I didn’t think could be published (or shouldn’t be), then I would discard it. There were many such papers.

Like Gans, I have a lot of research ideas, and not enough time to pursue all of them. Many of my ideas would go nowhere, even if I did have time to pursue them. But for some others, I have later read research papers that have done something I had thought of earlier but not had time to do myself. There are therefore a lot of missed opportunities, because it isn't possible to perfectly identify the good ideas in advance - you need to try them out before you realise that they are uninteresting or dead ends. Being able to try out more ideas seems like a good thing.

The opportunity cost of spending time pursuing one research idea is not pursuing other ideas. If generative AI allows us to pursue research ideas in less time, then it lowers the opportunity cost of pursuing those ideas. However, as Gans notes:

When you lower the cost of doing something, you do more of it. Normally, the decision whether to continue or abandon a project gives rise to some introspection (or rationalisation) of whether continuing is worthwhile relative to the costs. When the going gets tough, you drop ideas that don’t look as great.

The issue with an AI-first approach is that its benefit, reducing the toughness of going, is also its weak point; you don’t face those decision points of continuing/abandoning as often. That means that you are more likely to end up completing a project. But this lack of decision points means that you end up pursuing more lower-quality ideas to fruition than you would otherwise.

In Gans's experience, AI increases research output, but it also weakens the stopping rule that would otherwise kill bad ideas early, by decreasing the marginal cost of continuing the research. When the marginal cost of continuing is lower, we spend more time on each bad idea before discarding it. And if we spend too long on bad ideas, the opportunity cost of pursuing an idea increases - time spent working on bad ideas is time not spent pursuing ideas that turn out to be better. As a result, the average quality of our research may decrease. That risk needs more careful consideration.

Although he doesn't note the potential increasing opportunity cost, Gans's conclusion seems to point in that direction:

My point is that the experiment — can we do research at high speed without much human input — was a failure. And it wasn’t just a failure because LLMs aren’t yet good enough. I think that even if LLMs improve greatly, the human taste or judgment in research is still incredibly important, and I saw nothing over the course of the year to suggest that LLMs were able to encroach on that advantage. They could be of great help and certainly make research a ton more fun, but there is something in the judgment that comes from research experience, the judgment of my peers and the importance of letting research gestate that seems more immutable to me than ever.

Generative AI has the potential to increase the quality, and the quantity, of research. Gans seems to have seen it in his work, and I've seen it already in my own work too. In fact, my experience so far has been that careful use of generative AI (for example, checking for literature gaps, or exploring econometric methods and robustness checks) has reduced the time wasted on fruitless research that would have gone nowhere. However, it is possible to use too much generative AI in research, just as it is possible to use too little. There is a middle ground, and Gans seems to be finding it from one direction (starting from over-using generative AI), and maybe I am finding it from the other (starting from under-using generative AI). The important thing seems to be ensuring that a human is kept in the loop (as Ethan Mollick noted in his book Co-Intelligence, which I reviewed here). Specifically, we can use generative AI for its strengths (in testing our initial ideas, mapping the literature, exploring alternative methods, or stress-testing assumptions). And we can keep the human in the loop by pausing the research at more key points to consider where it has gotten to and check our intuition, as well as continuing to seek peer review of the draft end-product.

So, Gans might be holding back on the generative AI this year, but I'll be further expanding my use. Starting with something related to this post: writing up some research on using AI tutors in teaching first-year economics, which is research I presented in a brown-bag seminar at Waikato last month (and I will have more on that in a future post).

[HT: Marginal Revolution]

Try this: The Opportunity Atlas

It's hard to believe that, in over twelve years of blogging, I have never blogged about any of Raj Chetty's research. That's not because I haven't read it. If anything, it's because it is so detailed that it defies a short blog take. For example, we read two related papers published in the journal Nature (here and here, both open access) in the Waikato Economics Discussion Group back in 2022. Ordinarily, I would follow up with a blog post, but they are so in-depth that I couldn't find the time to summarise them effectively [*]. Three years later, they are sitting in a virtual pile of read-but-not-yet-blogged-about papers [**].

Anyway, Chetty and co-authors have suddenly made it much easier for me to summarise their extensive research on social mobility in the US. That's because you can see the data in action for yourself now, at The Opportunity Atlas. This very cool online tool allows you to see social mobility in action. Social mobility is effectively how much a child's socioeconomic position in adulthood depends on their socioeconomic position when they were growing up.

On The Opportunity Atlas, you can choose from a range of outcomes in adulthood, and see where the mean outcome is, if they grew up in a household at different rankings of parental income (1st, 25th, 50th, 75th, or 100th percentile). You can also look separately by gender and by race (Black, White, Hispanic, Asian, Native American). The interface is quite intuitive to use. For example, here's the basic map of expected (mean) income at age 35, for children who grew up in households at the 25th percentile of parental income:

The red areas, such as the South, have lower social mobility, because children who grew up there in households at the 25th percentile have lower incomes as adults. In contrast, the blue areas (in the north and west) have higher social mobility, because children who grew up there in households at the same 25th percentile have higher incomes as adults.

The tool is very flexible. It's very easy to switch to looking at other outcome variables, and for other percentiles of parental income, as well as zooming in on particular areas. For example, here's the teenage birth rate for Black women who grew up in households at the lowest (1st) percentile of parental income in Los Angeles:

The greyed-out Census tracts are those where there are too few Black women who grew up in the lowest income households for the data to be reported. However, the map shows a band of high teenage birth rates for mothers who grew up in the lowest income households, that stretches from South Central to Compton.

Importantly, the underlying data can be downloaded from the Opportunity Insights website. The cool thing about the data underlying the Atlas is that it is based on the census tract where the child grew up, not the census tract where they live as an adult. That means that the Atlas is showing you the adult outcomes for children who grew up in a particular area, not the adult outcomes of adults who live there today. That is explained in this new article by Chetty (Harvard University) and co-authors, published in the journal American Economic Review (ungated earlier version here).

That article outlines the methods underlying the dataset. In short:

...we use de-identified data from the 2000 and 2010 decennial censuses linked to data from federal income tax returns and the 2005–2015 American Community Surveys to obtain information on children’s outcomes in adulthood and their parents’ characteristics. We focus in our baseline analysis on children in the 1978–1983 birth cohorts who were born in the United States or are authorized immigrants who came to the United States in childhood...

We construct tract-level estimates of children’s incomes in adulthood and other outcomes, such as incarceration rates and teenage birth rates by race, gender, and parents’ household income level—the three dimensions on which we find children’s outcomes vary the most. We assign children to locations in proportion to the amount of their childhood they spent growing up in each census tract. In each tract-by-gender-by-race cell, we estimate the conditional expectation of children’s outcomes given their parents’ household income using a univariate regression whose functional form is chosen based on estimates at the national level to capture potential nonlinearities.

Chetty et al. then go on to show why it matters that we look at social mobility based on the place where children grew up, rather than contemporary poverty rates or adult outcomes, and finally give some short use cases for the dataset. I won't go into detail on those (you should read the paper), but one of the things that Chetty et al. do show is that because the effects change slowly over time, looking at outcomes today for children who grew up in a particular census tract in the 1980s still provides meaningful information that can be used for targeting social programmes today.

It's important to note that the Opportunity Atlas by itself doesn't show us causal estimates of adult outcomes. However, Chetty et al. establish how much of the effect is causal using a couple of different methods: (1) using data from the Moving to Opportunity experiment; and (2) a quasi-experiment that looks at how the effects differ depending on how many years a child was 'exposed' to a particular Census tract). Both methods both imply that roughly 62%) of the observed variation across census tracts reflects causal neighbourhood exposure effects, not just higher-opportunity families sorting into better places.

In the conclusion, Chetty et al. highlight a number of applications where the Opportunity Atlas data has already been used:

For researchers, the Opportunity Atlas data provide a new tool to study the determinants of economic opportunity. For example, recent studies have used the Opportunity Atlas data to analyze the effects of lead exposure, pollution, neighborhood redlining, and the Great Migration on children’s long-term outcomes (Manduca and Sampson 2019; Colmer, Voorheis, and Williams 2019; Park and Quercia 2020; Aaronson, Hartley, and Mazumder 2021; Derenoncourt 2022). Other studies use the Atlas statistics as inputs into models of residential sorting (Aliprantis, Carroll, and Young 2024; Davis, Gregory, and Hartley 2019) and to understand perceptions of inequality (Ludwig and Kraus 2019). The ongoing American Voices Project (https://americanvoicesproject.org/) is interviewing families in neighborhoods with particularly low or high levels of upward mobility to uncover new mechanisms from a qualitative lens.

I can see a number of use cases for this as well. For instance, there is probably a lot of value in using the Opportunity Atlas data alongside the data on racial diversity and segregation from the Mixed Metro project (which also offers data down to the Census tract level). Also related is this from a footnote in the Chetty et al. paper:

Understanding how neighborhood effects change with the composition of the neighborhood is an important question that warrants further work...

This also makes me think (again) that we need more detailed work on social mobility in New Zealand, building on the work of my colleagues Niyi Alimi and Dave Maré (see here). One of the amazing things about Chetty's research is that it is now looking at the neighbourhood (Census tract) level, and that sort of spatial disaggregation offers a lot of opportunity for detailed follow-up research and policy action. And with StatsNZ's Integrated Data Infrastructure, we have the basic framework necessary to do this sort of work in New Zealand as well. We could use that to build our own Opportunity Atlas for New Zealand.

*****

[*] So, in lieu of a separate blog post, here's the short summary of those two papers. In the first paper, Chetty et al. use billions of Facebook friendship links to measure local social capital, especially "economic connectedness" (cross-class friendships). They find that places with higher economic connectedness have much higher upward social mobility. In the second paper, the same group of authors show that cross-class friendship gaps come from both who people are exposed to (whether schools, neighbourhoods, or groups) and "friending bias" (less cross-class befriending even when exposed).

[**] In case you're wondering, there are currently 45 papers in that virtual pile, and it seems to be growing. I'm reading research faster than I'm blogging about it. I might have to start blogging about multiple papers in a single post to keep from falling further behind!