Students vs. representative samples in experimental economics

Experimental economics is an excellent tool for testing economic theories, and the effects of economic institutions, in an environment where other factors can be carefully controlled by the experimenter. In lab experiments, where the environment and the choices being made by participants are mostly artificial, most of the conditions related to the decision can be controlled (in contrast with field experiments), which eliminates most sources of bias that can affect people's decision-making. However, most of the samples used in lab experiments are convenience samples made up of university students, or even university economics students. One might rightly question whether the results obtained from lab experiments are sensitive to this selective sample, and whether the results are really generalisable to the population.

That is why I found this 2015 paper by Alexander Cappelen (Norwegian School of Economics), Knut Nygaard (Oslo and Akershus University College of Applied Sciences, Erik Sorensen, and Bertil Tungodden (both Norwegian School of Economics), and published in the Scandinavian Journal of Economics (ungated earlier version here), of great interest. In the paper, the authors compare the results of two commonly used experiments (the dictator game; and a generalised trust game), for three samples:

1. 120 economics students;
2. 119 humanities, science, or social science (excluding economics) students; and
3. 136 participants recruited from a sampling frame representative of the whole population of Norway.

Because of the nature of the two experiments the authors run, it allows them to tease out some of the underlying moral motives for peoples decisions. That is, they could infer whether people made decisions based on efficiency (maximising the gains from the experiment for everyone), equity (everyone receiving the same gains from the experiment), or reciprocity (if the other person helps you in the experiment, you help them). They found that:

...students differ fundamentally from a representative sample, both in the relative importance assigned to different moral motives and in the level of pro-sociality. Moreover, we show that one needs to be careful when generalizing about gender effects on the basis of selected student samples. Both for the dictator game and the trust game, the role of gender in the student samples does not carry over to the representative sample. Finally, we show that economics students behave less pro-socially compared to non-economics students, but the two student groups are similar in the relative importance they assign to different moral motives.

We find that both equality and efficiency are important motivational forces among male representatives, whereas female representatives seem to move from a concern for equality in non-strategic environments to a focus on reciprocity in strategic environments.

So perhaps there is good reason to be a little skeptical of experimental results based on student-only samples. And in case you think that it isn't that big and issue, the authors point out in the introduction:

Among the papers published on social preferences in the top five economics journals from 2000 to 2010, only four out of 24 papers report from experiments on non-student samples, and only two of the papers report from experiments performed outside the lab...

Eight of the 24 papers published in the top five economics journals from 2000 to 2010 report from experiments on economics students, whereas nine papers rely on other student populations or do not report detailed background information on the students.

So it's clearly possible that this could be a big problem for generalising from these results. Which actually relates to a point I have made before about experimental results from student samples.