Consider the following statement: "The effects of climate change are too far in the future to really worry me". Would you say that you: strongly agree with that statement; tend to agree; neither agree nor disagree; tend to disagree; or strongly disagree? Now consider this alternative wording of the same statement: "The effects of climate change are too far in the future to really worry about". Would your level of agreement be the same to this second statement as to the first?
There is good reason to believe that the wording of that question matters, as shown in this new article by Lieke Voorintholt (Trier University), Adriaan Soetevent, and Gerard van den Berg (both University of Groningen), published in the Journal of Economic Psychology (ungated earlier version here). They argue that the single-word difference, from "worry me" to "worry about" may shift respondents' focus from themselves individually towards broader society. Therefore, they hypothesise that more survey respondents would report being worried when asked if they worry about the effects, compared to whether the effects worry them individually. They also hypothesise that the treatment effect would be greater for older respondents, since the "about" version prompts thinking beyond the respondent's own life expectancy.
Voorintholt et al. use data from the Innovation Panel of Understanding Society, which its website describes as "a test-bed for innovative ways of collecting data and for developing new areas of research". The survey is run in Great Britain, and their analysis is based on a sample of 2791 survey respondents from Wave 16 of the survey, conducted in 2023. Half of the respondents were randomised to be asked the "me" version of the question, and the other half were asked the "about" question. Voorintholt et al. term the "me" version the control group, because that is the wording that is used in the main Understanding Society survey. So, the treatment is simply the replacement of "me" with "about" in the wording of the statement.
Their main results are summarised in Figure 1 from the article:
When asked the "about" version, more respondents say they strongly disagree with the statement, while fewer respondents say they tend to agree with the statement. These differences can be interpreted as greater worries about climate change expressed by the treatment group than the control group. This is also supported by their quantitative analysis, which shows the average response among treated respondents is 0.18 points higher (on a 1-5 scale from strongly agree to strongly disagree) than control respondents. Voorintholt et al. find some suggestive evidence in support of larger effects for older respondents. The coefficient on an interaction term between treatment and an age dummy variable for those aged 66 and over is positive and relatively large, but is statistically insignificant.
Overall though, this research is a good reminder that even a one-word change can materially affect how survey respondents answer a question, and therefore how we measure important variables. The effect of subtle changes should not be underestimated. If you want to measure the level of concern that people express about climate change, it really matters whether respondents are asked whether climate change “worries me” or whether it is something to “worry about”.
No comments:
Post a Comment