Reason to be increasingly skeptical of survey-based research

I've used a lot of different surveys in my research, dating back to my own PhD thesis research (which involved three household surveys in the Northeast of Thailand). However in developed countries, the willingness of people to complete surveys has been declining for many years. That in itself is not a problem unless there are systematic differences between the people who choose to complete surveys and those who don't (which there probably are). So, estimates of many variables of interest are likely to be biased in survey-based research. Re-weighting surveys might overcome some of this bias, but not completely.

A recent paper (ungated) in the Journal of Economic Perspectives by Bruce Meyer (University of Chicago), Wallace Mok (Chinese University of Hong Kong), and James Sullivan (University of Notre Dame), makes the case that things are even worse than that. They note that there are three sources of declining accuracy for survey-based research:

1. Unit non-response - where participants fail to answer the survey at all, maybe because they refuse or because they can't be contacted (often we deal with this by re-weighting the survey);
2. Item non-response - where participants fail to answer specific questions on the survey, maybe because the survey is long and they get fatigued or because they are worried about privacy (often we deal with this my imputing the missing data); and
3. Measurement error - where participants answer the question, but do not given accurate responses, maybe because they don't know or because they don't care (unfortunately there is little we can do about this).

Meyer et al. look specifically at error in reporting transfer receipts (e.g. social security payments, and similar government support). The scary thing is that they find that:

...although all three threats to survey quality are important, in the case of transfer program reporting and amounts, measurement error, rather than unit nonresponse or item nonresponse, appears to be the threat with the greatest tendency to produce bias.

In other words, the source of the greatest share of bias is likely to be measurement error, the one we can do the least to mitigate. So, that should give us reason to be increasingly skeptical of survey-based research, particularly for survey questions where there is high potential for measurement error (such as income). It also provides a good rationale for increasing use of administrative data sources where those data are appropriate, especially integrated datasets like Statistics New Zealand's Integrated Data Infrastructure (IDI), which I am using for a couple of new projects (more on those later).

Finally, I'll finish on a survey-related anecdote which I shared in a guest presentation for the Foundations of Global Health class at Harvard here today. Unit non-response might be an increasing issue for survey researchers, but there is at least one instance I've found where unit non-response is not an issue. When doing some research in downtown Hamilton at night with drinkers, we had the problem of too many people wanting to participate. Although in that case, maybe measurement error is an even larger problem? I'll blog in more detail on that research at a later date.

[HT: David McKenzie at the Development Impact blog]