What's in a (porn star) name, for identifying survey respondents?

In social science research, we usually want to maintain the confidentiality and anonymity of the respondents to our surveys and interviews. However, there are times when we will want to follow up with respondents at some later date, and if the first round of surveys was anonymous it is impossible to match up the first round respondents' responses with the later responses. So, I was interested to read this short 2011 article (open access) by Megan Lim, Anna Bowring, Judy Gold, and Margaret Hellard (all from the Burnet Institute in Melbourne), published in the journal Sexually Transmitted Diseases.

In the article, the authors discuss asking each survey respondent what their "porn star" name is. They explain:

"We trialed the uniqueness and reliability of a novel identifying characteristic: first pet's name and first street - colloquially known as a "porn star name".

The authors provide a table of examples of porn star names, of which 'Honey Scotsburn' and 'Precious Duckholes' were two (I'm not making this up - check the paper). They then go on to test whether they could match respondents from a baseline and follow-up survey based on the porn star name. Porn star names were unique to 99% of their 1281 respondents to the baseline survey, and adding month/year of birth was enough to provide 100% uniqueness. When re-contacted later, they were able to match 76% of respondents between the two surveys using only the porn star name, and using month/year of birth they could further match 96% of those who provided a partially-consistent porn star name.

It seems this is a pretty unique way of matching respondents between waves of a survey while maintaining plausible anonymity for those respondents. However, the authors note that "calling the identifier a PSN... might also have made the question seem trivial to some participants and resulted in false responses". Of course, this could all be a joke - Lim and Hellard were also two co-authors on research about the survival of teaspoons. Even if this paper was taken seriously (and it could be, since it addresses a real issue), it seems the research community isn't interested - this paper has only been cited twice since it was published in 2011.

Reason to be increasingly skeptical of survey-based research

I've used a lot of different surveys in my research, dating back to my own PhD thesis research (which involved three household surveys in the Northeast of Thailand). However in developed countries, the willingness of people to complete surveys has been declining for many years. That in itself is not a problem unless there are systematic differences between the people who choose to complete surveys and those who don't (which there probably are). So, estimates of many variables of interest are likely to be biased in survey-based research. Re-weighting surveys might overcome some of this bias, but not completely.

A recent paper (ungated) in the Journal of Economic Perspectives by Bruce Meyer (University of Chicago), Wallace Mok (Chinese University of Hong Kong), and James Sullivan (University of Notre Dame), makes the case that things are even worse than that. They note that there are three sources of declining accuracy for survey-based research:

1. Unit non-response - where participants fail to answer the survey at all, maybe because they refuse or because they can't be contacted (often we deal with this by re-weighting the survey);
2. Item non-response - where participants fail to answer specific questions on the survey, maybe because the survey is long and they get fatigued or because they are worried about privacy (often we deal with this my imputing the missing data); and
3. Measurement error - where participants answer the question, but do not given accurate responses, maybe because they don't know or because they don't care (unfortunately there is little we can do about this).

Meyer et al. look specifically at error in reporting transfer receipts (e.g. social security payments, and similar government support). The scary thing is that they find that:

...although all three threats to survey quality are important, in the case of transfer program reporting and amounts, measurement error, rather than unit nonresponse or item nonresponse, appears to be the threat with the greatest tendency to produce bias.

In other words, the source of the greatest share of bias is likely to be measurement error, the one we can do the least to mitigate. So, that should give us reason to be increasingly skeptical of survey-based research, particularly for survey questions where there is high potential for measurement error (such as income). It also provides a good rationale for increasing use of administrative data sources where those data are appropriate, especially integrated datasets like Statistics New Zealand's Integrated Data Infrastructure (IDI), which I am using for a couple of new projects (more on those later).

Finally, I'll finish on a survey-related anecdote which I shared in a guest presentation for the Foundations of Global Health class at Harvard here today. Unit non-response might be an increasing issue for survey researchers, but there is at least one instance I've found where unit non-response is not an issue. When doing some research in downtown Hamilton at night with drinkers, we had the problem of too many people wanting to participate. Although in that case, maybe measurement error is an even larger problem? I'll blog in more detail on that research at a later date.

[HT: David McKenzie at the Development Impact blog]