Book review: How Big Things Get Done

There are certain books that shouldn't need to be written. Inevitably, those are the books that, in reality, most need to be written. That is certainly the case for How Big Things Get Done, by Bent Flyvbjerg and Dan Gardner. This is a book about big projects, and importantly, how those projects succeed or, as is often the case, how they fail. As the authors note in the preface, it is a book that aims to answer a number of important questions:

Why is the track record of big projects so bad? Even more important, what about the rare, tantalizing exceptions? Why do they succeed where so many others fail?

The book draws on decades of Flyvbjerg's academic research on big projects, as well as his experience both consulting on, and being directly involved in, big projects. Through this work, Flyvbjerg has developed a massive database of projects, their cost and benefit estimates at the time the project began, and the cost over-runs and benefit shortfalls that so often resulted. The numbers do not make for easy reading, and the examples that Flyvbjerg uses range from transport infrastructure to It projects to nuclear power stations to the Olympic Games. On the latter, the book is a useful complement to Andrew Zimbalist's book Circus Maximus (which I reviewed here).

Flyvbjerg and Gardner spend a lot of time discussing failed projects, but devote substantial space to discussing successes, such as Terminal 5 at Heathrow. Many of us will remember the opening of Heathrow for the terrible problems associated with baggage handling in the first few days of opening, but the project itself delivered on time and on budget. Once you read this book, you'll realise just how extraordinary that accomplishment is.

Flyvbjerg and Gardner use the comparison between successful projects and failures to draw a number of lessons. Most of the lessons seem obvious, but clearly those lessons have not been learned well enough in the 'big projects' space, because they are so often not heeded. The biggest lesson of all is to 'think slow, act fast'. Thinking slow means spending substantial time planning before the project begins, ensuring that the risks are well known and have been planned for, before the first spade turns the first sod. Acting fast means completing the project as quickly as possible, to avoid the 'unknown unknowns' from impacting the project - the more delays, the more time there is for something unforeseen to happen.

The 'think slow, act fast' approach seems inconsistent with Silicon Valley's approach to development (as ably described in Jonathan Taplin's 2017 book Move Fast and Break Things, which I reviewed here). Flyvbjerg and Gardner anticipate that counterexample, and note that the two are not inconsistent at all, because:

Planning is doing: Try something, see if it works, and try something else in light of what you've learned. Planning is iteration and learning before you deliver at full scale, with careful, demanding, extensive testing producing a plan that increases the odds of the delivery going smoothly and swiftly.

That is, more or less, what the big tech firms do. Flyvbjerg and Gardner note that iteration is key to those firms' development process, and is generally successful (or where it isn't, the firm can rapidly iterate to something new). In contrast, most big projects are delivered using a 'think fast, act slow' approach that is doomed to failure. 

I really enjoyed this book, even though it does seem quite depressing at times, just how badly big projects are at delivering on their promises (both in terms of costs, and in terms of benefits). The book is not only well researched, but draws on many interviews that Flyvbjerg has completed with people in the industry. The writing did make me wonder what Gardner's contribution was - the whole book is written as if by Flyvbjerg alone (with lots of "I" and "my"), which seems an odd stylistic choice for a co-authored book. Nevertheless it is an enjoyable read, and definitely recommended.

What does the Cantril Ladder really measure?

Imagine a ladder with steps numbered from 0 at the bottom to 10 at the top. The top of the ladder represents the best possible life for you and the bottom of the ladder represents the worst possible life for you. On which step of the ladder would you say you personally feel you stand at this time?

Now, consider the question you probably just answered. What factors played into your answer? What sorts of things contribute to the best possible life for you, compared with the worst possible life for you? If we used your answer to that question as a measure of life satisfaction, what is it really measuring?

That's not an unimportant question. The first paragraph of this post is a commonly used way of measuring life satisfaction, known as the Cantril ladder (see here). It is used in the Gallup World Poll, and is recommended by the OECD as a way of measuring subjective wellbeing. When researchers (or governments, or others) measure life satisfaction or happiness, it is often the Cantril ladder that is being used.

The question of what the Cantril ladder measures was explored in this recent article by August Nilsson (Lund University), Johannes Eichstaedt (Stanford University), Tim Lomas (Harvard University), Andrew Schwartz (Stony Brook University), and Oscar Kjell (Lund University), published in the journal Scientific Reports (open access, with non-technical summary on The Conversation). Nilsson et al. looked at the framing of the Cantril ladder, and investigated how nearly 1600 people responded to different framings of the question, and the words that they used to describe the top and the bottom of the scale in those different framings, and where they would 'prefer to be' on the scale. The first framing was the traditional Cantril ladder. The second framing essentially replaced the ladder metaphor with the word "scale" (but left the rest intact). The third framing removed references to the "bottom" and "top" (as well as the ladder metaphor). The fourth framing did all of that plus changed "best possible life" to "happiest possible life" (and "worst possible life" to "unhappiest possible life"). And the fifth and final framing instead replaced "best possible life" to "most harmonious life" (and "worst possible life" to "least harmonious life").

Nilsson et al. found that:

The ladder and bottom-to-top scale anchor descriptions influenced respondents to use significantly more words from the LIWC dictionaries Power and Money when interpreting the Cantril Ladder... compared to when these anchors were removed. Of all the words respondents used to describe the top of the Cantril Ladder, 17.3% fell into the Power and Money dictionaries. This language was reduced by more than a third when the ladder was removed in the no-ladder condition (absolute difference of 6.0%, d = 0.35, p < 0.001), and more than halved when the bottom-to-top scale descriptions were removed too (absolute difference of 10.3%, d = 0.64, p < 0.001). Further, for the Cantril Ladder, words in the Power and Money dictionaries occurred 3.3 times as frequently compared to the alternative Harmony anchor condition (absolute difference of 12%, d = 0.77, p < 0.001).

They interpret those results as meaning that:

...the original Cantril Ladder influenced respondents to focus more on money in terms of wealth (whereas when the ladder framing was excluded, they focused more on financial security) than the other conditions.

Were you thinking about the financial aspects of life when you answered the question above? The results seem to suggest that is more common than thinking about social relationships or the various other contributors to our subjective wellbeing. Nilsson et al. don't explore the use of words other than in the 'Power' and 'Money' domains, but it would have been interesting to see some others to compare with.

It's not surprising that financial security, income, or wealth are important contributors to subjective wellbeing or life satisfaction. We should expect people to be better able to satisfy their needs when they have greater financial resources available to them. However, the results on research participants' preferred level on the ladder are genuinely surprising, because:

...over 50% did not prefer the highest level (of 10) in any of the study conditions, and less than a third preferred the top of the Cantril Ladder, which had a significantly lower average preferred level than all the other study conditions.

In other words, even though the top of the Cantril ladder is framed as the 'best possible life', around two-thirds of research participants said that they would prefer not to be at the top of the ladder. This proportion was lower (but still not zero) for other framings, as shown in Figure 4 from the article (where the dark blue part of the bar shows the proportion of research participants who responded that 10 was their preference):

What was your preferred level on the ladder? Did you want to have the best possible life (that is, 10 on the scale)? Or would you prefer to be somewhere just below the best possible life? What do you think about in answering the question on your preferred level? Maybe research participants want 'room to grow' and become even happier or more satisfied with their lives? I have no idea. Nilsson et al. have given us something to really think about here, but unfortunately the article doesn't go far enough in exploring why people don't prefer the top of the ladder. There is definitely scope for further follow-up research on this point.

In addition to being surprising, that last result may call into question how the Cantril ladder is interpreted (on top of the arguments about the validity of happiness data generally - see here, and here, and here). If the top of the scale is not the top of the scale, or if it is different for different research participants, then how do we interpret an average across all people responding to the question? That should make researchers worry, and makes follow-up research even more important.

[HT: New Zealand Herald, back in April]

Read more:

• Happiness is dead?
• More evidence that life satisfaction may be dead
• The reports of the death of life satisfaction may have been greatly exaggerated

This week in research #47

Here's what caught my eye in research over the past week:

• Geerling, Mateer, and Wooten (open access working paper) identify a group of “rising stars” in the economics teaching field (where I'm ranked #27 in the world according to their ranking, and #5 outside of the US)
• Li and Xia find that students just above a letter-grade cutoff in an introductory course are 3.6% more likely to major in the same field as that course, using data from the National University of Singapore
• Divle, Ertac and Gumren find in an online experiment that although working in a team is more profitable and participants also expect this, a large proportion shy away from teamwork, and that research participants primed with COVID-19 are less likely to self-select into teamwork
• Dickinson and Waddell find, using data from GitHub, that the transition to Daylight Saving Time reduces worker activity, but that the effects are relatively short-lived, although when using more detailed hourly data losses appear in the early working hours of work days into a second week following the initiation of Daylight Saving Time
• Naidenova et al. look at twelve years of data from professional Counter-Strike: Global Offensive games and find that there is a substantial decrease in the performance of esports players during overtime, which they attribute to 'choking under pressure', although the impact is less in online competitions compared to live events
• Martínez-Alfaro, Silverio-Murillo, and Balmori-de-la-Miyar (open access) find in an audit study that job applications from transgender candidates received 36% fewer positive responses than those from cisgender candidates in Mexico

Book review: Wonderland (Steven Johnson)

When I think about the dramatic changes in society that have occurred since the end of the Industrial Revolution, one of the trends that stands out (to me) is the massive increase in leisure time. In the 19th Century, most people worked far more hours than they do today. The recent decades of that trend were well-described in Daniel Hamermesh's book Spending Time (which I reviewed here). What was left unexplored in that book was the way that leisure pursuits have affected the economy and society.

That is the purpose of Steven Johnson's book Wonderland, which is subtitled "How play made the modern world". Johnson describes the book as:

...a history of play, a history of the pastimes that human beings have concocted to amuse themselves as an escape from the daily grind of subsistence. This is a history of what we do for fun.

The book is comprised of chapters devoted to fashion and shopping, music, food, entertainment, games, and our use of public space. Each chapter is well written and well resourced, and a pleasure to read. Johnson is a great storyteller and the stories he presents are interesting and engaging.

However, from the first chapter, I struggled with the overall thesis of the book, which is that changes in leisure pursuits drove broader societal changes and economic changes. This is most glaringly demonstrated in the first chapter, where Johnson contends that it was the desire for fashion that drove the Industrial Revolution:

When historians have gone back to wrestle with the question of why the industrial revolution happened, when they have tried to define the forces that made it possible, their eyes have been drawn to more familiar culprits on the supply side: technological innovations that increased industrial productivity, the expansion of credit networks and financing structures; insurance markets that took significant risk out of global shipping channels. But the frivolities of shopping have long been considered a secondary effect of the industrial revolution itself, and effect, not a cause... But the Calico Madams suggest that the standard theory is, at the very least, more complicated than that: the "agreeable amusements" of shopping most likely came first, and set the thunderous chain of industrialization into motion with their seemingly trivial pursuits.

In spite of the excellent prose, I'm not persuaded by the demand-side argument for the Industrial Revolution, which flies in the face of lots of scholarship in economic history (as well as in history). Now, it may be that the first chapter just made me grumpy. But Johnson draws several conclusions which are, at best, a selective interpretation of the evidence. And at times, he makes comparisons that are somewhat odd, such as a comparison between the tools and technologies available to artists and scientists and those available to musicians in the 17th Century, concluding that there were fewer and less advanced tools available to artists and scientists than for musicians. There doesn't seem to be any firm basis to make such a comparison (how does one measure how advanced technologies in different disciplines are, in order to compare them?).

The final chapter, though, was a highlight to me. There was a really good discussion of the role of taverns in the American Revolution. And in that discussion, Johnson acknowledges that it is difficult to establish a causal relationship (which made me again wonder why he was unconcerned about the challenges of causality between shopping and the industrial revolution earlier in the book). I really appreciated the discussion of the work of Jürgen Habermas, Ray Oldenburg, and the "third places" (places of gathering that are neither work, nor home). It reminded me of my wife's excellent PhD thesis on cafés.

Overall, I did enjoy the book in spite of my griping about the overall thesis and the way that Johnson sometimes draws conclusions from slim evidence. If you are interested in the history of leisure pursuits, I recommend it to you.