Bullshitting as a signal of intelligence

Yesterday's post was a book review of Carl Bergstrom and Jevin West's excellent book Calling Bullshit. One question that probably didn't receive enough coverage in the book is the question of why people bullshit. I guess Bergstrom and West felt that it was self-evident. However, this recent article by Martin Turpin (University of Waterloo) and co-authors, published in the journal Evolutionary Psychology (open access) may provide an answer. They argue that:

The ability to produce satisfying bullshit, with its emphasis on impressing others without regard for truth or meaning... may represent an energetically inexpensive strategy for both signaling one’s intelligence, and deceiving others to one’s advantage.

How does that work? First, it's worth considering what a signal is, and why we provide signals. Signalling is a way of overcoming problems of adverse selection, which is a problem of asymmetric information. Each person knows how intelligent they are, but we don't know. This is asymmetric information. Since we don't know who is intelligent and who is not, it makes sense for us to assume that everyone has low intelligence. This is what we call a pooling equilibrium, and that pooling equilibrium would create problems. If you believe that everyone has low intelligence, it's hard to believe anything you are told (which will make bullshitting as a solution to this problem pretty ironic).

How can someone reveal that they are intelligent? They could just tell you, "Hey, I'm smart". But, anyone can do that. Telling people you are intelligent is not an effective signal. To be effective, a signal needs to meet two conditions:

1. It must be costly; and
2. It must be costly in such a way that those with low quality attributes (in this case, those who are less intelligent) would not be willing to attempt the signal.

Bullshitting is costly - there is a chance that you are caught out in your bullshitting, and people then think less of you (the cost is mostly social in nature). If more intelligent people are better at bullshitting, then it is less likely that they get caught out. That would make bullshitting less attractive of a strategy for less intelligent people. In that case, bullshitting would be an effective signal of intelligence.

Are more intelligent people better at bullshitting? That is the question that Turpin et al. answer in their article. Their research was split into two studies. In the first study, 483 undergraduate students were recruited and given two tests of intelligence (the 'Wordsum' test of verbal intelligence; and Raven's Progressive Matrices (RPM), which is a test of abstract reasoning and general intelligence:

...we presented participants with ten concepts (e.g., cognitive dissonance) four of which were fake (i.e., subjunctive scaling, declarative fraction, genetic autonomy, neural acceptance). Participants’ first task (bullshit willingness task) was to rate their knowledge of each concept on a 5-point scale ranging from “never heard of it” to “know it well, understand the concept.” Responses given to fake concepts were summed to create an index of participants’ bullshit willingness, with higher scores indicating a greater tendency to bullshit (i.e., feign knowledge of fake concepts). Next, a subset of participants (Bullshit Producers) were presented with each of the ten concepts individually and... were instructed to “produce the most convincing and satisfying explanation” they could for each concept. For concepts they were unfamiliar with, participants were instructed to “be creative and make up an explanation that you think others will find convincing and satisfying.”

Turpin et al. then had another subset of participants (Bullshit Raters) rate how accurate and satisfying each of the bullshit explanations was. They found that there was:

...significant positive correlations between participants’ bullshit ability and Wordsum scores... as well as between bullshit ability and RPM scores... Therefore, we find initial evidence of bullshit ability sharing a modest positive association with measures of intelligence.

The second study reported in the article expanded on the first with a new sample of 534 undergraduates, by asking the Bullshit Raters about the perceived intelligence of the Bullshit Producers. In addition to similar correlations with measured intelligence as Turpin et al. found in the first study, they found that:

...the perceived intelligence of Bullshit Producers was positively correlated with assessments of their bullshit ability... This association is consistent with the hypothesis that producing satisfying and seemingly accurate explanations of completely fictional concepts is perceived by individuals as a signal of intelligence.

In other words, bullshit ability does appear to be a signal of intelligence. However, there is one inconsistency. From the first study:

...participants’ willingness to bullshit was negatively associated with scores on the Wordsum... and RPM... suggesting that those scoring higher on our measures of cognitive ability were less willing to bullshit.

If bullshitting really is a signal of intelligence, then we would expect more intelligent people to be more willing to bullshit. However, perhaps more intelligent people have a better theory of mind, and so they are more accurate in assessing other people's receptiveness to bullshit. That would mean that less intelligent people are more willing to bullshit because they incorrectly believe that they can mislead others. Alternatively, perhaps the measure of bullshit willingness (which is made up of the number of made up concepts that research participants claimed to know) is at fault. Perhaps less intelligent people, in an experimental or survey setting, are more likely to claim to know things that they don't? Either way, before we can say for certain that bullshitting is an effective signal of intelligence, we'd want to establish that, ceteris paribus, more intelligent people are more likely to bullshit. That is a task for future research.

[HT: Marginal Revolution]

Book review: Calling Bullshit

When I led the redevelopment of our Bachelor of Management Studies with Honours degree back in 2018 (see more here), one of the key innovations was to introduce a number of general management 'transferable skills' papers across the final two years. One of those papers was titled "Using Data for Decisions", and the intention was not to have students working directly with data, but rather to have them asking questions about data and about analyses that others have conducted. Those are core skills for a modern manager, who often does not do their own analysis (at least in medium to large firms). The lecturers of that paper have since taken it in a different direction (making students do more work with data), but if we had held close to the original ideal, then Carl Bergstrom and Jevin West's book Calling Bullshit would no doubt have been a key reading.

Interestingly, the genesis of the book was a very popular college course of the same title, which Bergstrom and West teach at the University of Washington. In the Preface, they note their intention for the book:

We hope it will show you that you do not need to be a professional statistician or econometrician or data scientist to think critically about quantitative arguments, nor do you need extensive data sets and weeks of effort to see through bullshit.

So, you can see that this aligns with my original vision of the BMS(Hons) Using Data for Decisions paper. Of course, the book isn't just about data, it is about bullshit. Why? Bergstrom and West explain that:

The world is awash with bullshit, and we're drowning in it.

Politicians are unconstrained by facts. Science is conducted by press release. Silicon Valley startups elevate bullshit to high art. Colleges and universities reward bullshit over analytic thought... Bullshit pollutes our world by misleading people about specific issues, and it undermines out ability to trust information in general.

Bergstrom and West define bullshit as:

Bullshit involves language, statistical figures, data graphics, and other forms of presentation intended to persuade or impress an audience by distracting, overwhelming, or intimidating them with a blatant disregard for trust, logical coherence, or what information is actually being conveyed.

Fully the first ten chapters of the book (out of eleven) are essentially devoted to alerting the reader to bullshit, and (hopefully) teaching them how to spot it. Along the way, Bergstrom and West teach you about the difference between correlation and causation, selection bias, data visualisation ducks, glass slippers, and the problems associated with big data. Many of these ideas (and the examples they use) will not be new to people who are widely read in this area, but the collection of these together in one place is highly useful. They also offer one of the best explanations of what a p-value is (and what it isn't) that I have read in a popular book.

It is not until the final chapter that Bergstrom and West turn their attention to how they propose that we should refute bullshit, because of course it is not enough simply to call bullshit unless you are prepared to back up your calls and stand by them. This was the only part of the book that lacked extensive examples that would have helped to illustrate the 'dos' and 'don'ts' of calling bullshit.

I really enjoyed this book, and it is clear why the Calling Bullshit course at the University of Washington is so popular. Bergstrom and West write in a way that is easy to read, and they don't bury us in jargon (even when describing big data). And that is part of the point they are trying to make - you don't need to understand the technical language in order to think critically about quantitative claims that are being made. If you want to add a critical edge to your thinking about the array of infographics, research claims, and statistics that we are bombarded with on social media (and traditional media), then I highly recommend this book.

Substance abuse treatment centres and local crime

When a substance abuse treatment centre opens, does that decrease local crime, or increase it? Theory doesn't help us much here, because you can argue theoretically in both directions. On the one hand, a treatment facility reduces drug use, so may reduce drug-induced aggressive behaviour, reduce crimes by drug users looking to earn money to fuel their habits, and reduce violence between drug dealers. On the other hand, a facility attracts drug users to collect in an area, and changes the types of people and places they interact with, all of which might increase crime (concentrated in areas around the facility) - at least, that's the NIMBY argument against substance abuse treatment centres. Due to the theoretical ambiguity, we need to answer the question of the relationship between substance abuse treatment centres and crimes empirically.

That's what this 2018 article by Samuel Bondurant, Jason Lindo (both Texas A&M University), and Isaac Swensen (Montana State University), published in the Journal of Urban Economics (ungated earlier version here), does. They use annual county-level data on the location of substance abuse treatment centres from the US Census Bureau's County Business Patterns, and crime data from the FBI's Uniform Crime Reports, and from the National Center for Health Statistics' Multiple Cause of Death database. They find that:

...county-level homicide rates are reduced by SAT facilities. Specifically, the estimates indicate a 0.25% decline in intentional homicide death rates associated with an additional SAT facility...

...the point estimate for the effect on sexual assault is also negative, suggesting that SAT facilities reduce sexual assault as well, it is not close to being statistically significant at conventional levels. The estimated effect on aggravated assaults also suggests a reduction in crime associated with SAT facilities, though this estimate is only marginally statistically significant...

The point estimates indicate that a SAT facility reduces robbery by 0.11%, motor vehicle theft by 0.12%, burglary by 0.05%, and larceny by 0.06%. The estimated effects larceny are not statistically significant at conventional levels.

In other words, substance abuse treatment centres are associated with lower crime rates. Interestingly though, in the Discussion section of their article, Bondurant et al. weigh up the costs and benefits of these facilities:

...these calculations suggest that the county-level benefits of an additional facility - in terms of drug-related mortality and criminal activity - are between 5.4 and 7.65 million dollars, depending on whether effects on homicides are included. Reductions in crime account for approximately 20-40% of these benefits...

A back-of-the-envelope calculation indicates that the annual costs of treatment for a SAT facility are approximately 1.1 million dollars... These calculations suggest that the benefits of expanding treatment facilities far outweigh the associated treatment costs.

Bondurant et al. argue that their results demonstrate causal effects, because the effects occur only after a change in the number of facilities, and not before (they establish this by looking at lead and lag variables of the number of facilities). I don't find those results convincing enough to believe that the results overall are causal. Also, one bit I found interesting in this research is that they use the number of facilities in the prior year as their explanatory variable. This is explained later in the paper as:

...a choice we made to avoid attenuation bias that would likely be caused by the fact that newly opened (or closed) facilities would only affect counties for some fraction of the year.

This attenuation bias is likely to be a problem in a lot of the literature that looks at the relationship between facilities of various types (like alcohol outlets, marijuana dispensaries, or treatment centres) and crime (or other measures of harm). That includes my own work on the relationship between alcohol outlets and measures of harm (e.g. see here). The number of facilities is measured at one point in time, but the events (crimes, etc.) occur over a period of time that might be before or after the number of facilities has changed. This leads to measurement error in the number of facilities, and measurement error leads to attenuation bias, where the statistical model underestimates the size of the relationship between the number of facilities and crime. This problem is likely to be greatest for annual data (and, in contrast, would likely not be much of a problem at all for daily data).

The choice of using the number of facilities from the year before is an interesting solution, but I think it still creates measurement error, only the measurement error is in the opposite direction to what would happen in the case of most research of this type. When a new facility opens during a year, the standard approach assumes that facility is open for the whole year, overstating the number of facilities. In Bondurant et al.'s analysis, it effectively assumes that facility is closed for the whole year, understating the number of facilities. So, there is still measurement error.

Despite the lack of explicit causal analysis, the cost-benefit analysis is reasonably convincing (to me, at least). The benefits of substance abuse treatment centres outweigh the costs (in the US), and there should be more of them.

Read more:

• Marijuana legalisation and local crime rates

Gorillas in the midst of data

In a famous experiment, Daniel Simons and Christopher Chabris tested people's selective attention, asking them to count the number of times a basketball is passed. If you haven't seen or heard of this test, you should try it out for yourself:

Did you see the gorilla? Some years ago, I was shown a different video with the same premise, and I totally missed it. The whole point is that, when we are very focused on a particular task, we can totally miss other important things that are going on.

That brings me to this 2020 article by Itai Yanai (NYU Langone Health) and Martin Lercher (Heinrich Heine University), published in the journal Genome Biology (open access). Yanai and Lercher gave students two datasets, which each contained data on body mass index (BMI) and the number of steps taken each day. One dataset was data points for women, and one dataset was data points for men. Yanai and Lercher then placed the students into two groups, and gave the two groups different instructions:

The students in the first group were asked to consider three specific hypotheses: (i) that there is a statistically significant difference in the average number of steps taken by men and women, (ii) that there is a negative correlation between the number of steps and the BMI for women, and (iii) that this correlation is positive for men. They were also asked if there was anything else they could conclude from the dataset. In the second, “hypothesis-free,” group, students were simply asked: What do you conclude from the dataset?

When you're given a dataset with no idea of what to look for, it is natural to start with some simple tabulations or data visualisations. And, if students merged the two datasets together and graphed BMI against steps per day, they saw this (from Figure 1a of the paper):

Yes, that is a gorilla waving at you from the data. Interestingly, Yanai and Lercher find that:

...overall, students without a specific hypothesis were almost five times more likely to discover the gorilla when analyzing this dataset...

Specifically, nine of 14 students who were given no hypotheses found the gorilla, but only five of 19 students who were given hypotheses did so. This was a small-scale study, and not entirely serious, but it does illustrate a serious point: when we are focused simply on hypotheses, we may miss important features of the underlying data, and that's why it's a good idea to start with some simple tabulations and visualisations (and I will admit, I'm as guilty as anyone of jumping this step, especially for datasets that I think I know well).

Yanai and Lercher draw a distinction between 'day science' and 'night science':

There is a hidden cost to having a hypothesis. It arises from the relationship between night science and day science, the two very distinct modes of activity in which scientific ideas are generated and tested, respectively... With a hypothesis in hand, the impressive strengths of day science are unleashed, guiding us in designing tests, estimating parameters, and throwing out the hypothesis if it fails the tests. But when we analyze the results of an experiment, our mental focus on a specific hypothesis can prevent us from exploring other aspects of the data, effectively blinding us to new ideas. A hypothesis then becomes a liability for any night science explorations... Night science has its own liability though, generating many spurious relationships and false hypotheses. Fortunately, these are exposed by the light of day science, emphasizing the complementarity of the two modes, where each overcomes the other’s shortcomings.

So, when we look at data, we need to do so both by day and by night.

[HT: David McKenzie at Development Impact, especially for the clever title of this post]