Back in 2020, I wrote a post that questioned the foundations of subjective wellbeing, based on this article by Bond and Lang. They essentially showed that the conclusions that are drawn from studies of subjective wellbeing (or happiness) depend on how the ordinal variable (subjective wellbeing, being measured on a 0-10 scale) is converted to a cardinal variable. They concluded that:
It is essentially impossible to rank two groups on the basis of their mean happiness using the types of survey questions prevalent in the literature.
It seems that Bond and Lang's result is not an aberration. I recently read this 2017 article by Carsten Schröder (Free University Berlin) and Shlomo Yitzhaki (Hebrew University), published in the journal European Economic Review (ungated earlier version here), which came to a very similar conclusion. Schröder and Yitzhaki first rightly note that:
Well-being (life satisfaction or happiness) is a latent variable that is impossible to observe directly and that has no natural quantitative measurement unit. Data are collected in scientific surveys that ask questions like, “All in all, how satisfied are you with your life at the moment?” Respondents answer these questions by ranking their satisfaction levels on a pre-defined scale, usually ranging from 4 to 11 points (and sometimes more), with the individual points assigned terms such as “very bad,” “bad,” or “good.”... In a scale like this, we know that “very bad” is lower than “bad” and that “bad” is lower than “good”, but it is not clear whether the distance between “very bad” and “bad” is greater or smaller than the difference between “bad” and “good”. The ordinal nature of this data means that any monotonic increasing transformation of the scale is allowed.
They then apply some simple monotonic transformations to subjective wellbeing data from the German Socio-Economic Panel (SOEP), and the results are not good (at least, for those who hope to use subjective wellbeing and want it to be meaningful). IN a simple comparison of the gender gap in life satisfaction, Schröder and Yitzhaki find that:
...in only one case is a simple comparison of average satisfaction robust to monotonic increasing transformations. In the other 47 cases, an admissible transformation that can reverse the sign of the gender gap in satisfaction exists...
In other words, they can reverse the gender gap by simply transforming the data in a way that shouldn't affect the results. So much for simple comparisons of mean differences. Looking at regression models, they also find that:
...using random effects does not prevent the reversal of coefficients. Using fixed effects avoids reversals but the significance of the coefficients hinges on the transformation. Note, however, that the robustness of the sign of the fixed-effects regression coefficients to monotonic increasing transformations, in general, is not guaranteed. Reversals of signs of coefficients can be shown for this type of model as well...
So, pretty standard linear regression models aren't robust. What about ordinal models? After all, they are designed specifically to deal with ordinal data. Schröder and Yitzhaki first note that:
...they result, by definition, in non-intersecting cumulative distributions. That is, these models assume a dominance relationship of the cumulative distributions that the raw data do not necessarily support...
Then they find that, comparing life satisfaction between German and non-German respondents:
In three of the five significant cases, the empirical distributions intersect. Of course, the importance and frequency of violations of dominance depend on the empirical context.
So, the distributions intersect, and so the ordinal models are finding statistically significant differences when they should not. Overall, the results are not flattering for users of subjective wellbeing data, and should especially be of concern to those who are trying to look beyond GDP as a measure of wellbeing (as noted in my recent book reviews here and here). No doubt there is more to come on this topic.
Read more:
No comments:
Post a Comment