Learning communities and the academic performance of women and minorities in STEM

I'm a big fan of peer mentoring approaches in higher education. They can help students in directly improving their grades by connecting students with senior peers in the same subject. However, the biggest impact on grades is probably indirect - having access to senior peers helps first-year students to understand the resources that are available to them, how to deal with particular styles of assessment, how best to allocate their time, and where to get additional help if they need it. These indirect impacts are likely to be largest for students who are first-in-family to attend university, because students who have parents or older siblings who have graduated already have access to that assistance.

A community of learning is a slightly broader concept than peer mentoring. There are various flavours, but mostly they involve connecting students more closely with peers, academic mentors, and study advisors. If resourced well, communities of learning have great potential to improve student outcomes, again particularly for first-in-family and disadvantaged students.

So, I was interested to read this 2017 article by Lauren Russell (Dartmouth College) recently, which was published in the journal Economics of Education Review (sorry, I don't see an ungated version online). Lauren investigates the impact of the Experimental Study Group (ESG) at MIT, which:

...aims to make the transition to MIT easier, especially for freshman who come from non-elite high schools and/or traditionally underrepresented groups in STEM. ESG features small classes and teaching methods that differ from mainstream versions of introductory subjects. Students co-enroll in courses and take advantage of dedicated study-spaces to foster peer networks. Finally, students are intentionally mentored by both MIT upperclassmen and ESG faculty. In this way, ESG combines a policy-relevant bundle of treatments designed to address obstacles to academic success at the undergraduate level.

ESG is over-subscribed, so the 55 available places each year are allocated by lottery. Russell uses that random assignment to extract the causal impact of ESG on academic outcomes, using data for 2011-15 (which are the only years for which full lottery results are available). ESG applicants, as you may expect, are different from the average MIT freshman, and:

...applicants are more likely to be female, international, or a first-generation college student. They are more likely to be black or Hispanic and less likely to be white or Asian. Finally, they are more likely to have applied for financial aid and are more likely to be Pell Grant recipients.

So, it is worth noting at the outset that ESG applicants are those most likely to benefit from a programme like this. They are also the right students to target the programme to, if it is effective. However, for all ESG enrolees on average, the results are not good:

The standardized treatment effect point estimate suggests that ESG participation increases academic performance by 0.07 of a standard deviation, though this effect is not statistically significant.

However, the results become substantially more positive when looking at subgroups, specifically female students, minority students, and low-income students:

ESG raises the academic performance of female students by 0.4-0.5 of a standard deviation with this effect statistically significant at the 5% level in the specification without controls... and the 1% level in the specification with full controls... The effects for the other two subgroups are similarly positive and large (0.3-0.4 of a standard deviation) but the standard errors are too large to infer much based on the statistical significance or lack thereof.

Those effects are large, but the lack of statistical significance for minority and low-income students should dampen our enthusiasm somewhat. That also leads to the obvious question: why is it female students that benefit? Russell looks at the interaction between student gender and instructor gender, and finds that:

...among non-ESG applicants, female students earn course letter grades 0.23 GPA points (0.29 of a standard deviation) higher when a GIR course is taught by a female instructor rather than a male instructor. They are also 11 percentage points more likely to take another course in the same subject area... The magnitude of the female student-female instructor interaction on course grade is even larger for ESG applicants. For female ESG applicants, the female instructor effect is 0.38 GPA points (0.53 of a standard deviation). This analysis suggests that the academic performance of these students may be even more sensitive to instructor gender than students who do not apply to ESG...

However, despite the apparent size of the effect, instructor-student gender interactions explain only 15 percent of the effect of ESG on grades for female students. So, we are kind of left in the dark as to how it is working so well for female students. It certainly isn't through 'excitement and the extent of hands-on coursework', 'reported self-confidence', or 'social connections and mental health', all of which are statistically insignificantly affected by ESG.

On the plus side, and going back to the full sample, ESG does appear to have an effect on choice of major, as it:

...increases the probability that a student will (single) major in math, computer science, or electrical engineering by about 10 percentage points...

The effect is particularly large for minority students (but not for female or low-income students). I don't think we should read too much into those results, particularly given that we are unsure about the mechanisms through which they are working.

So, what do we take away from this research? It provides some suggestive evidence of a positive effect of the ESG intervention on students, particularly female students. It is targeted at students who would be likely to benefit, but perhaps it is not targeted enough? Or perhaps, there is little to gain from this kind of intervention in this population of students. Remember that this is a sample of MIT students. They are already high-achievers, so the potential gains from a successful community of learning are likely to be modest. To give us a better understanding of how well these programmes work, we need to test them on a student population that has more potential to gain from them.