A meta-analysis of meta-analyses of the value of statistical life

Individual research studies provide a single estimate (or a small number of related estimates) of whatever the researchers are trying to measure. There are many reasons that a single study might provide a biased estimate of what is being measured, including the data that are used and the methods that the researchers choose. For someone else reading the literature on a particular topic, it can be difficult to identify what the 'best' measure is, when there are many different estimates, all based on different data and methods. In those situations, meta-analysis can help, by combining the published estimates in a particular literature into a single overall estimate. Meta-analysis can even take account of publication bias, where statistically significant estimates are more likely to be published than statistically insignificant results.

But what should you do when there are multiple meta-analyses to choose from, each using different collections of estimates from other studies, and employing different methods? Is it time for a meta-analysis of meta-analyses? A meta-meta-analysis?

That is essentially what this 2021 NBER Working Paper by Spencer Banzhaf (Georgia State University) does for the literature on the value of statistical life in the US. As Banzhaf explains:

The Value of Statistical Life (VSL) is arguably the single most important number used in benefit-cost analyses of environmental, health, and transportation policies...

When choosing a VSL or range of VSLs, analysts must sift through a vast literature of hundreds of empirical studies and numerous commentaries and reviews to find estimates that are (i) up to date, (ii) based on samples representative of the relevant policy contexts, and (iii) scientifically valid... The US EPA is the only one of the three agencies that uses a formal meta-analysis. It uses a value of $9.4m with a 90 percent confidence interval of $1.3m to $22.9m (US EPA 1997, 2020). However, even today, these estimates are based on very old studies published between 1974 and 1991...

Perhaps one reason for this surprising gap is that we now have an embarrassment of riches when it comes to summarizing VSL studies. With so many to choose from, the process of selecting which meta-analysis to use, and defending that choice, might feel to some analysts almost like picking a single "best study."... Comparing these meta-analyses, many analysts may conclude that, as with the individual studies underlying them, each of them has a bit of something to offer, that no single one is best. Thus, the old problem of selecting a single best study has just been pushed back to the problem of selecting a single best meta-analysis.

Banzhaf collates the results from five recent meta-analyses of the VSL in the US, and applies a 'mixture distribution' approach:

Essentially, I place subjective mixture weights on eight models from five recent meta-analyses and reviews of VSL estimates applicable to the United States. I then derive a mixture distribution by, first, randomly drawing one of the eight meta-analyses (the mixture component) based on the mixture weights and, second, randomly drawing one value from the distribution describing that component's VSL (e.g., a normal distribution with given mean and standard deviation), and, finally, repeating these draws until the simulated mixture distribution approximates its asymptotic distribution.

Banzhaf finds that:

...the overall distribution has a mean VSL of $7.0m in 2019 dollars... The 90% confidence interval ranges from $2.4m to $11.2m.

For comparison, the VSL in New Zealand used by Waka Kotahi is $4.42 million (as of June 2020, which equates to about US$2.8 million). However, what was interesting about this paper wasn't so much the estimate, but the method for deriving the estimate (by combining the results of several meta-analyses). That's not something I had seen applied before, but with the growth of meta-analysis across many fields in science and social science, methods for combining meta-analysis estimates need active consideration.

Also, I thought New Zealand was an outlier in basing the VSL on seriously outdated estimates. New Zealand's VSL was estimated at $2 million in 1991, and has been updated since then by indexing to average hourly earnings. There are two sources of bias that are quite problematic when we continue to use essentially the same measure for over thirty years. First, VSL measures are based on either revealed preferences (how people react to the trade-off between money and risk of death), or stated preferences (how people say they would react to the same trade-off). However, preferences change over time, and our VSL estimate is still based on the trade-off as established in 1991. If New Zealanders have become more risk averse (in relation to risk of death) over the last thirty years, then the VSL will be an underestimate. Second, it assumes that hourly earnings is the best way of indexing the measure over time. This is adequate for a measure that was based on observed hourly wages at the time it was first estimated (such as if the VSL was based on trade-offs in the labour market). However, it assumes that the risk profile of jobs across the labour market is constant over time, and that certainly isn't the case. It is likely that risks are lower now, so the indexed VSL may be too high.

Given that these two biases work in opposite directions, it seems to me that it is well past due for an update of the underlying estimate. Unfortunately, we can't easily rely on a meta-analysis (or a meta-meta-analysis) as we lack a large number of underlying estimates. There is a clear opportunity for more New Zealand-based research on this.

[HT: Marginal Revolution, last year]