In our most recent Waikato Economics Discussion Group session, we discussed this recent article by Alberto Chong (Georgia State University) and Marco Chong (Bethesda-Chevy Chase High School), published in the journal Kyklos (ungated earlier version here). They used hand-collected data from ten seasons of the US version of MasterChef (from 2010 to 2020), to investigated whether fear of failure leads to better, or worse, performance. This is an important question, as they note that:
...fear of failure is rather common and widespread in societies. In the United States, for instance, it has been estimated that around 30 % of the population is terrified of failure, and it ranks among the worst fears that the population endure in this country...
Fear of failure has previously been studied in a number of contexts, including sports, business, and education. Most of the literature finds that fear of failure reduces performance. That may be because fear of failure leads to emotional paralysis, inhibiting people from their 'usual' level of performance. On the other hand, fear of failure could lead to increased performance, by providing additional focus that leads to greater creativity and the ability to take calculated risks. MasterChef is a particularly interesting context in which to study this, because:
As the home cooks are judged by world class chefs and restaurateurs and watched by television audiences that range in the millions the potential shame and embarrassment of failing under these circumstances is very significant... The fact that the judges tend to be rather harsh with the contestants further compounds to this, more so given that these home cooks come with high self-esteem and egos, as they are typically considered as cooking luminaries in their immediate circles of friends, families, coworkers, neighbors or clubs and associations.
In other words, failure in MasterChef is very public, and directed at someone who is probably not used to failure. Chong and Chong collected data on the final ranking of 197 contestants across the ten seasons of MasterChef in their sample. They measured fear of failure as the sum of two variables:
The first is the number of times that a contestant ends up among the bottom three entries in any particular cooking challenge. The second is the number of times that a contestant ends up surviving a Pressure Test...
Chong and Chong also created a measure of 'extreme fear of failure', which was simply the number of times that a contestant survived a Pressure Test. They then look at the relationship between their measures of fear of failure and the contestants' final ranking in the season, controlling for individual characteristics, the number of rounds the contestant participated in, and a measure of positive reinforcement (made up of the number of times the contestant won a Mystery Box or elimination or team challenge, plus the number of times that they placed in the top three in a challenge). Chong and Chong find that:
...on average, individuals that are on the verge of being eliminated, but are able to survive and stay in the competition, end up doing better in the final rankings, all else being equal. In particular, we find that the higher the number of times a contestant is put in this situation, the higher his or her final placement will be among all the contestants. Overall, we find that depending on the measure used, an increase in one unit in our fear of failure index is linked to an increase of between almost one position to four positions in the final competition placement, the latter in situations of extreme fear of failure.
In other words, contestants who experience a greater fear of failure over the course of the season, perform better. Or do they?
Think carefully about how Chong and Chong measured fear of failure - the number of times that a contestant won a Mystery Box or elimination or team challenge, plus the number of times that they placed in the top three in a challenge. Contestants that last longer in the season will obviously find themselves in those situations more often than contestants who are eliminated early. So, contestants who last longer in the season will rank higher overall, and have a higher measure of fear of failure, simply because they lasted longer in the season. In other words, there is a clear survivorship bias in their analysis.
But wait! Didn't Chong and Chong control for the number of rounds that the contestant participated in? They did, and in theory that should mean that their results compare participants in terms of fear of failure, holding the number of rounds they participated in constant. However, that's not quite the case, because of the way that the number of rounds variable is included. The number of rounds variable is a measure of the exposure of each contestant to the potential for fear of failure. But that assumes that the exposure variable is linear, when it really isn't. If being at risk of elimination is randomly allocated to contestants, then there is a 15 percent (3/20) chance of being in the bottom three when there are 20 contestants, but a 50 percent (3/6) chance of being in the bottom three when there are just six contestants left. The effect of the number of rounds is clearly non-linear, but they control for a linear relationship.
To illustrate the problem here, I constructed some simulated data and ran some analyses (in Excel). I assumed that there were initially 20 contestants, and that each round, one of them was eliminated. I randomly determined which contestant was eliminated, and which two of the other contestants were in the bottom three. I ran this through until a 'final' 17th round, where there were four contestants remaining. I then calculated a measure of fear of failure (the number of times the contestant was in the bottom three), and the ranking of each contestant. Then, I ran a multiple regression model, with ranking as the dependent variable, and fear of failure as the explanatory variable, controlling for the number of rounds that the participant survived for. The outcome was that fear of failure had a coefficient of -0.350 with a p-value of 0.009 (highly statistically significant). In other words, in the data that was simulated totally at random, the result was a statistically significant relationship. It's picking up survivorship bias.
Then, instead of controlling for the number of rounds, I calculated a measure of expected exposure to fear of failure. This was the probability that a contestant would find themselves in the bottom three in a round, then summed up for each round that they participated in. When I run the same multiple regression, but controlling for expected exposure instead of the number of rounds, the coefficient on fear of failure was -0.21 and statistically insignificant (p-value of 0.38).
And, just in case anyone thinks all of this was purely coincidental, I created a whole new random dataset, and repeated the exercise. The second time, controlling for rounds the coefficient on fear of failure was -0.22, and statistically significant with a p-value of 0.035. When controlling for expected exposure, the coefficient on fear of failure was -0.27 and statistically insignificant (p-value of 0.13).
I'm sure I could run the analysis many times more and get similar results. While my data setup is not identical to theirs, it does enough to illustrate that their measure of fear of failure is susceptible to survivorship bias, because a randomly simulated dataset leads to similar results (albeit with a smaller coefficient).
Chong and Chong say that their data are available on reasonable request. This study is crying out for a replication with a better measure of fear of failure. That could either be a measure of expected exposure to fear of failure (as in my simulated dataset), or dummy variables to each number of rounds. Approaching the analysis either way (or both) would make a great Honours project for a suitably motivated student.
No comments:
Post a Comment