Book review: The Tyranny of Metrics

I'm pretty sure that you will have heard the saying "Not everything that can be counted counts, and not everything that counts can be counted" or some variation of it, often attributed to Einstein, but actually comes from a book by William Bruce Cameron (no relation to me). Now imagine an entire book devoted to that topic. The book you are imagining is The Tyranny of Metrics, by historian Jerry Muller. Aside from the Cameron quote above, the premise can also be summarised as:

There are things that can be measured. There are things that are worth measuring. But what can be measured is not always what is worth measuring; what gets measured may have no relationship to what we really want to know. The costs of measuring may be greater than the benefits. The things that get measured may draw effort away from the things we really care about. And measurement may provide us with distorted knowledge - knowledge that seems solid but is actually deceptive.

The book is well written and easy to read. Muller first lays out his critique of measurement and 'metric fixation' (as he terms it), then moves on to providing case studies demonstrating the evils of a fixation on metrics in many fields: colleges and universities, schools, medicine, policing, the military, business and finance, and philanthropy and foreign aid. The case studies are mostly good and illustrate the overall point well. For instance, take this bit on the unintended consequences of metric fixation in higher education:

A mushroom-like growth of administrative staff has occurred in other countries that have adopted similar systems of performance measurement, such as Australia. In most such systems, metrics has [sic] diverted time and resources away from doing and toward documenting, and from those who teach and research to those who gather and disseminate the data for the Research Assessment Exercise and its counterparts.

Anyone in a western university can relate to that, and that section of the book could be read alongside the late David Graeber's excellent book, Bullshit Jobs (which I reviewed here). However, not all of the case studies offered the same clarity of illustration of unintended consequences. In particular, I felt like the military and philanthropy sections were a little strained.

After the case studies, Muller moves onto a more general digression arguing that transparency is not always the best approach. I thought that section diverged a bit too much from the message of the book and wasn't really necessary. The conclusion brought things together nicely though:

As we've seen time and again, measurement is not an alternative to judgment: measurement demands judgment: judgment about whether to measure, what to measure, how to evaluate the significance of what's been measured, whether rewards and penalties will be attached to the results, and to whom to make the measurements available.

As you can see, the book is not simply a polemic against measurement and metrics in all their forms. Muller is arguing for a more sensible approach to measurement and the use of metrics, one that recognises their limitations and the potential pitfalls that their use entails. Anyone involved in business or policy formulation must recognise that the use of particular metrics will create incentives. And we should always keep Goodhart's Law in mind: "When a measure becomes a target, it ceases to be a good measure".

Notwithstanding the few gripes I have, I really enjoyed this book, and recommend it to anyone who is thinking about implementing metrics, or anyone who is looking to craft an argument against their implementation.