Getting people drunk and seeing what they do is an interesting strand of research (not nearly as interesting as getting crayfish drunk and seeing what they do, though!). Most experiments involving drunk people are conducted in controlled laboratory settings (for example, see here or here). The benefit of the lab environment is that the particular experimental treatment that participants are being exposed to (which could be their level of intoxication) can be cleanly controlled, and the experimental task they are completing can be stripped to its essentials. However, the external validity of lab experiments is always going to be in question - do those choices extend to the real world, where things are messier? When we're talking about the effects of alcohol, messy might be the best way to describe things.
The question of external validity of lab experiments is essentially the focus of this new working paper by Iain Long, Kent Matthews, and Vaseekaran Sivarajasingam (all Cardiff University). They run a series of experiments at a student bar in Cardiff, and then run the same experiments again with the same participants a week later, in the lab. Specifically, the experiments use Raven's Progressive Matrices (RPM) to test research participants' cognitive ability (technically, they are a test of fluid intelligence, but the difference need not concern us here).
Long et al. argue that the difference in outcomes of the RPM test between the two observations (in the bar, when participants are intoxicated; and in the lab, when participants are not intoxicated) represents a combination of alcohol impairment of the research participants and the impact of the bar environment. They further note that previous lab experiments have failed to demonstrate a statistically significant impact on RPM performance, so if there is any impact observed in their experiment, then the bar environment must be the source. Long et al.'s reasoning is attractive, but I'm not sure it is correct (more on that in a moment).
Based on their sample of 106 research participants, they find that when they restrict their analysis only to the sessions in the bar, that breath alcohol concentration (BAC) has no significant impact on performance in the RPM test. However, when they analysis across both sessions (bar and lab), they find that:
...our results now appear highly significant and robust. Relative to being sober in the control environment, the average participant (whose BAC is 0.36) gets one fewer question correct when they have been drinking in the bar.
That's one fewer question correct, out of ten (when the average score in the lab was 7.8, and the average score in the bar was 6.6). The effect is quite sizeable, and because it only appears when they analyse the results comparing [bar+intoxicated] with [lab+not intoxicated] and not when they compare [bar+intoxicated] with [bar+slightly less intoxicated], Long et al. conclude that it shows:
...early evidence in favour of the hypothesis put forward by lab experiments that suggest that intoxication alone cannot explain the changes in behaviour that are commonly observed when people consume alcohol.
That's fair enough. They've shown that more intoxicated people perform essentially the same as less intoxicated people in the bar setting. They've shown that people perform worse in the bar setting than in the lab setting. However, there is a key missing piece of the puzzle here. They haven't shown that more intoxicated people perform essentially the same as less intoxicated people in the lab setting. Instead, they rely on other studies that show no difference, but those other studies were conducted in different settings (and, remember, external validity may be an issue).
Long et al. do posit some features of the bar environment that might explain the results:
Over-crowding, sexual competition... high temperatures... inaccessible bar and toilet facilities... noise levels... and competitive games... are all thought to contribute.
Possibly. They can't test any of those mechanisms. However, let me posit one more problem for their analysis. Their interpretation assumes that the bar environment treatment has the same linear impact for all research participants. However, it is possible that the bar environment exerts a larger negative effect on performance for less intoxicated people than more intoxicated people. Less intoxicated people may be more negatively affected by the noise, or the distractions, of the bar environment. More intoxicated people are already less focused than less intoxicated people. If that were the case, then the difference that Long et al. estimate would overstate the impact of the bar environment, because the bar environment narrows the difference in performance between more intoxicated and less intoxicated research participants. Again, they could have teased this out if they had a lab treatment at different levels of intoxication.
The working paper then goes on to look at impacts on over-confidence, finding that participants were not more overconfident when they were more intoxicated within the bar setting, but comparing the bar setting with the lab, participants in the bar were more over-confident. The same caveats apply to this analysis at with that based on performance in the RPM test.
Lab experiments suffer from perceived problems of external validity. Running so-called 'lab-in-the-field' experiments like this one are a solution to that problem. However, what this particular experiment demonstrates is that when the environment cannot be controlled, the results can get a bit messy.
[HT: Steve Tucker]
Read more:
No comments:
Post a Comment