Tim Harford on the hidden costs of cost-benefit analysis

Last week I wrote a post about the cost-benefit principle. On the Waikato Economics Discussion Group on Facebook, we had a brief but interesting discussion about how cost-benefit analysis is most valuable in the absence of uncertainty. The algorithm discussed in my post above is overly conservative, since if there is insufficient details on benefits or costs, it suggests that you should not adopt the AI technology. That might be appropriate for a decision-maker who is very risk averse, but for many it would lead to passing up on alternatives where the chances of costs outweighing benefits is very low, simply because the costs or benefits are uncertain.

The cost-benefit principle generally works well as a theoretical framework for rational decision-making. However, cost-benefit analysis, as employed by policy-makers and consultants, can often go wrong, and Tim Harford had a recent post on this topic:

On the surface, cost-benefit analysis is all very rational, and admirably well-suited to decide whether to build a bridge or abolish a regulation. However a new article in the Journal of Benefit-Cost Analysis by Bent Flyvbjerg and Dirk Bester argues that cost-benefit analysis “is broken”. Flyvbjerg and Bester assemble and analyse a data set of more than 2,000 public projects — bridges, buildings, bus transit systems, dams, power plants, railways, roads and tunnels. They conclude that costs have been systematically underestimated. This is broadly true regardless of when or where the project was commissioned, or what type of project it is.

Even allowing for this tendency for projects to run over budget, there are more catastrophic cost overruns than one might expect. There is less data on benefits, but there is some evidence of benefits being overestimated for public transit projects, and no sign that the large cost overruns are in any way compensated for by surprisingly large benefits.

These findings are sobering, if not entirely surprising. As Flyvbjerg and Bester are at pains to point out, “cost overrun” isn’t really the right description of what keeps happening. “Cost underestimate” is better. The problem is not that every project engineer in the world is incapable of delivering to a reasonable budget; it is that the budgets are never reasonable. The infrastructure we see around us is born of rose-tinted spreadsheets and hype.

Yikes! But not surprising, to anyone who pays attention to the costs of large government projects. But it makes projects that fail a benefit-cost test even more important to avoid. Can we do better than cost-benefit analysis? Harford notes that:

So what is the point of cost-benefit analysis if all the costs and benefits being analysed are flights of fantasy? The only answer I can think of, with apologies to Winston Churchill, is that cost-benefit analysis is the worst form of evaluating decisions, except for all those other forms which have been tried from time to time.

Alternatives include decision-making by HIPPO (highest-paid person’s opinion), decision-making by sound bite or decision-making by whatever polls well with people who may not have spent a moment’s thought on the matter. These are not good ways to think through complex, long-term infrastructure projects or wide-reaching regulations.

So, we are stuck with cost-benefit analysis for large projects, even if it is imperfect (and getting more difficult as we understand more about measuring costs and benefits). We probably shouldn't apply the overly conservative say-no-if-you-can't-quantify-something approach, or we would never say yes to any large project (although some may argue that would be an improvement over the current approach). However, we do need to know where the key areas of uncertainty in costs and benefits lie, even if they can't be precisely quantified. And, we definitely need to stop underestimating the costs of these projects. Then, at least, we might avoid the worst public investment outcomes.