Book Review: The Flaw of Averages

I just finished reading The Flaw of Averages, by Sam Savage. Reading the blurb, I would have thought this was a book that presented arguments that I would have a lot of sympathy for. The core argument that underlies the book is that so often decision-makers are looking for a single number that they can use for decision-making (often this is the average), but using that single number results in flawed and costly mistakes, because it ignores the fact that the single number is drawn from a distribution of possible numbers. Savage essentially argues for using simulation modelling, a particular form of which he has developed, called probability management.

In my own work, preparing population projections for local councils and other decision-makers, I often struggle with the decision-makers' needs for a single magic number that they can use for decision-making. Along with Jacques Poot, we pioneered the use of stochastic models for sub-national population projections in New Zealand (see this paper, for one example, or the longer ungated version here). Stochastic models explicitly display the uncertainty in future projections of the population, and there are a few regularities that Jacques and I noticed, such as projections being more uncertain for areas with smaller populations, and surprisingly more uncertainty for slower-growing or stable populations (compared with faster growing populations).

Towards the end of the book, there is a good quote that illustrates why decision-makers prefer not to have to deal with uncertainty, and prefer to focus on a single magic number:

Unfortunately, most organizations don't know how to deal with distributions. They generally ignore that part of the forecast, relying instead on the single number, and, presto, they're back to square one with the Flaw of Averages...

So, as has been my experience, you can provide decision-makers with the extra information on the uncertainty of a projection (or forecast), but you can't make them use it!

Savage's book can essentially be broken down into three parts. In the first part of the book, he essentially tries to make us forget all of the complicated terminology used in what he refers to as 'steam era' statistics, and instead replace the complicated 'red words' with 'green words' that have the Savage stamp of approval. However, in my opinion the green words are more ambiguous and sometimes plain wrong. For instance, Savage would have us replace "utility theory" (a red word) with "risk attitude" (a green word). Now, risk attitudes and utility theory are related, but not so much that you can replace both terms with one of them! Savage is also highly uneven in his disdain for complicated 'red words' - academic terms from finance such as the Capital Asset Pricing Model seem to get a free pass. Given that a lot of the book uses examples drawn from finance, this seems a little biased.

The second section of the book is the highlight. In these chapters, Savage uses personal stories of decision-makers and firms such as the oil company Shell and the pharmaceutical company Merck, to illustrate how simulation modelling can substantially improve the quality of decision-making. This is the really interesting stuff, and if the book had stuck to this, I feel it would have been much better.

The third section is essentially an extended infomercial for Savage's particular implementation of simulation modelling, probability management. While the examples extend those from earlier in the book, they're really just trying to sell the reader on the tools that Savage has developed.

Overall, I found that the personal stories of models in the real world are great. However, the book seems to have too many purposes and as a result, it doesn't execute as well on any of them as it might. In particular, it's a pity the first part of the book was essentially just a rant against terminology that Savage finds offensive. Moreover, Savage hasn't been as careful as he might with his examples. Fairly early in the book, he presents decision-making based on decision trees. However, despite his strong encouragement for us not to reduce decision-making to single numbers, in that chapter he uses expected value calculations - which reduces the decision to being based on a single number!

Overall, I wouldn't recommend this book for the general reader. If you want to understand why simulation modelling is important (or why it is important not to reduce analyses to a single number), it is useful for that, but I would skip through and start reading from about Chapter 16, and stop when your tolerance for the infomercial at the end is exhausted.