Book review: The Tyranny of Metrics

I'm pretty sure that you will have heard the saying "Not everything that can be counted counts, and not everything that counts can be counted" or some variation of it, often attributed to Einstein, but actually comes from a book by William Bruce Cameron (no relation to me). Now imagine an entire book devoted to that topic. The book you are imagining is The Tyranny of Metrics, by historian Jerry Muller. Aside from the Cameron quote above, the premise can also be summarised as:

There are things that can be measured. There are things that are worth measuring. But what can be measured is not always what is worth measuring; what gets measured may have no relationship to what we really want to know. The costs of measuring may be greater than the benefits. The things that get measured may draw effort away from the things we really care about. And measurement may provide us with distorted knowledge - knowledge that seems solid but is actually deceptive.

The book is well written and easy to read. Muller first lays out his critique of measurement and 'metric fixation' (as he terms it), then moves on to providing case studies demonstrating the evils of a fixation on metrics in many fields: colleges and universities, schools, medicine, policing, the military, business and finance, and philanthropy and foreign aid. The case studies are mostly good and illustrate the overall point well. For instance, take this bit on the unintended consequences of metric fixation in higher education:

A mushroom-like growth of administrative staff has occurred in other countries that have adopted similar systems of performance measurement, such as Australia. In most such systems, metrics has [sic] diverted time and resources away from doing and toward documenting, and from those who teach and research to those who gather and disseminate the data for the Research Assessment Exercise and its counterparts.

Anyone in a western university can relate to that, and that section of the book could be read alongside the late David Graeber's excellent book, Bullshit Jobs (which I reviewed here). However, not all of the case studies offered the same clarity of illustration of unintended consequences. In particular, I felt like the military and philanthropy sections were a little strained.

After the case studies, Muller moves onto a more general digression arguing that transparency is not always the best approach. I thought that section diverged a bit too much from the message of the book and wasn't really necessary. The conclusion brought things together nicely though:

As we've seen time and again, measurement is not an alternative to judgment: measurement demands judgment: judgment about whether to measure, what to measure, how to evaluate the significance of what's been measured, whether rewards and penalties will be attached to the results, and to whom to make the measurements available.

As you can see, the book is not simply a polemic against measurement and metrics in all their forms. Muller is arguing for a more sensible approach to measurement and the use of metrics, one that recognises their limitations and the potential pitfalls that their use entails. Anyone involved in business or policy formulation must recognise that the use of particular metrics will create incentives. And we should always keep Goodhart's Law in mind: "When a measure becomes a target, it ceases to be a good measure".

Notwithstanding the few gripes I have, I really enjoyed this book, and recommend it to anyone who is thinking about implementing metrics, or anyone who is looking to craft an argument against their implementation.

Is there a signalling explanation for the high cost of article re-formatting?

My post yesterday highlighted the high cost of re-formatting papers for submission to academic journals, with a total cost estimated at US$1.1 billion per year. I was reflecting on this today, and perhaps there is a reason for this high cost, aside from each journal's publishers wanting to maintain a particular style that distinguishes the journal from other journals.

Perhaps the time cost of formatting (and re-formatting) is a way of dealing with asymmetric information, specifically adverse selection. At the time of submission (i.e. before peer review, and before the editor has even read the abstract), the quality of a paper submitted to a journal is known to the authors (presumably), but not to the publisher. The quality is therefore private information. Since the publisher doesn't know whether any particular article submission is high quality or not, their best option (aside from editorial and peer review, which I will come to in a moment) is to assume that every submission is low quality. This is a pooling equilibrium - all article submissions are pooled together as if they are the same quality. The publisher may as well pick randomly from this pool of submissions of unknown quality, leading to a journal that gains a reputation for low quality articles (this is basically the business model of some publishers that offer 'pay-to-publish'). Authors with high quality articles would avoid those journals, lowering the quality of the article submissions further. Eventually, only the lowest quality articles get submitted, and published. This is a problem of adverse selection, because authors want their high quality articles to be published, but in the end, only low quality articles get published.

The way to solve an adverse selection problem is to reveal the private information - in this case, to identify which article submissions are high quality, and which are low quality. That would lead to a separating equilibrium, where low quality articles are rejected and high quality articles are accepted and published. The publisher can reveal this information through editorial and peer review. The editor reads the abstract (and perhaps the paper), and decides whether it is worthwhile sending for review, and if not the submission is desk-rejected. If the paper is sent out for peer review, its quality is judged by the peer reviewers. These processes are a form of screening - where the uninformed party (the publisher) tries to reveal the private information (about the quality of the article submission).

However, screening is not the only way to deal with an adverse selection problem. And the problem with screening through editorial and peer review is that it takes up a lot of time. The editor has to spend time reading and making decisions, and the peer reviewers have to spend time reading and writing reports. The alternative to screening is signalling - where the informed party (the authors) reveal the private information themselves.

Now, of course, if you simply ask the authors whether their paper is high quality or not, every author (even those with low quality articles) would respond that their paper is high quality. In order for a signal to credibly reveal the private information and be effective, it needs to meet two important conditions: (1) it needs to be costly; and (2) it needs to be more costly in a way that makes it unattractive for those with the low quality attributes to attempt. One way that signals could meet the second condition is if they are more costly to the authors of low quality articles.

By having idiosyncratic formatting requirements, it is clear that submitting to a journal is costly, so it meets the first condition. What about the second condition? I can see two ways that we could argue that costly re-formatting is more costly for authors of low-quality articles than for authors of high-quality articles.

First, authors of low-quality articles will realise that their submission has a lower chance of acceptance than a high-quality article does. That means that they can anticipate having to go through the re-formatting and submission process more than once, leading to a higher cost. To avoid this higher cost, they may avoid submitting to high quality journals (thereby revealing that their paper is low quality). Authors of high-quality articles know their submission is high quality and has a higher chance of being accepted, so they know they face a lower cost of re-formatting, and will be more likely to submit to a high-quality journal.

Second, authors of high-quality articles are more likely to be high-quality academics, who are well supported by their institutions, have research grants, and may have research assistants who can handle the re-formatting at relatively low (salary, or monetary) cost. Authors of low-quality articles are less likely to have this support, and have to handle the re-formatting themselves, at (presumably) higher salary (or monetary) cost.

So, perhaps the high cost of re-formatting journal articles for submission to journals is a signalling mechanism that acts as a way of sifting out the low-quality journal submissions before they start tying up editor (and peer reviewer) time? Would moving to a system where the authors can submit in any sensible format for the initial submission actually be an improvement, or would it tie up more resources in unnecessary editorial and peer review? It would be interesting to see some analysis of the experience of journals that have adopted a more open formatting approach for first submission, in terms of the quality of submissions (and the quality of published articles).

The high cost of re-formatting papers for submission to journals

One of the worst tasks in research is formatting papers ready for submission to an academic journal. Every journal has its own idiosyncratic requirements, in terms of formatting, referencing, word limits, abstract length, keywords, and so on. None of the time spent on formatting is productive time, and most of it is time wasted, given that more than half of the time your article submission is going to be rejected (and will need to be formatted again for submission to the next journal).

How much time is wasted on the task of re-formatting? This 2019 paper by Yan Jiang (Stanford University) and co-authors collected data from authors of articles in:

twelve journals from the InCites Journal Citation Reports (JCR) database in each of eight broad scientific (biology, biochemistry and molecular biology, microbiology, immunology, and cell biology) and clinical fields (cardiology, gastroenterology, oncology).

In total, they had 206 responses (out of the 288 authors they approached). They found that:

When asked how much time was needed for reformatting to all journals to which the paper was resubmitted to, the majority of authors (77/118, 65%...) reported that they spent 1–3 days or more (one day of effort was defined to the respondent as meaning eight hours). This did not include time spent on improving the scientific content or waiting for reviewer comments. Time spent on reformatting alone delayed resubmissions by over two weeks in most instances (60/118, 51%...).

I'd suggest that 1-3 days on re-formatting is probably an overestimate, based on my experience. One day perhaps, but only if the journal you are submitting to is incredibly idiosyncratic in terms of referencing (e.g. there are some journals that require the first names of authors in the reference list, and that take forever to compile). Anyway, based on the survey responses, Jiang et al. estimate the total cost of time spent on re-formatting:

Based on our data of 57.3% of articles needing resubmission, the time spent on reformatting... and prior data of 2.3 million annual scientific articles published... we estimate that first or corresponding authors spend about 23.8 million hours reformatting worldwide every year. Using the average first year postdoctoral researcher salary of $48,432... we roughly estimate costs of reformatting to be around $550 million dollars yearly worldwide for the first or corresponding author. When taking into account the time spent by the entire research team... the costs are estimated to be $1.1 billion dollars.

Yikes! The formatting requirements of journals cost US$1.1 billion per year. We need more journals to adopt a process where the authors can submit in any sensible format for the initial submission. Jiang et al. note how rare this is in their sample:

At the time of our review, only 4/96 (4%) of journals offered fully format-free initial submission.

Finally, I found this bit from the start of the introduction kind of quaint:

The process of publishing peer-reviewed research can be slow and onerous... It is not uncommon for manuscript reviews to take three months and the overall time from submission to publication to take between seven to nine months...

I think there would be plenty of economists who would dream of a process that takes seven to nine months from submission to publication, rather than periods up to several years at some top journals.

[HT: Marginal Revolution, back in 2019]

Tax incentives can encourage older people to delay retirement and work longer

Developed countries are facing a problem. Increasing life expectancy, coupled with low fertility, is leading to a rapidly ageing population. Countries that have publicly funded old age pensions are likely going to face challenges to their continuing affordability, because there will be fewer working age taxpayers for each pension recipient (what economists refer to as a lower 'support ratio'). The options available to policy makers include increasing the age of eligibility for pensions (as several countries have done in recent years), decreasing the real value of pensions (such as by not adjusting them for inflation), or shifting from universal pensions to means-tested pensions (where older people with high income or wealth would not be eligible to receive the pension).

All of these changes are politically tricky to implement, because as the population ages, older people (and those soon to become eligible for the pension) become an even larger share of the voting population. Also, reducing the real value of pensions (or delaying eligibility for them) may lead to increases in poverty among older people. Another alternative that may reduce these poverty concerns, is to encourage older people to delay retirement, working until they are older and, depending on the pension rules, potentially delaying their receipt of pension benefits (even when the age of eligibility has not changed). One way to encourage people to work more is to allow them to keep more of their labour earnings, such as by lowering the tax rate on labour income.

A reasonable question, then, is how much difference can a tax change make to the labour market behaviour of older people? This 2017 article by Lisa Laun (Institute for Evaluation of Labour Market and Education Policy, Sweden), published in the Journal of Public Economics (open access) provides some indication. Laun uses linked Swedish data from the "Income and Tax Register (IoT), the Longitudinal Database on Education, Income and Employment (LOUISE) and the Employment Register", which allows her to track nearly 190,000 people who turned 65 years old within three months either side of the year end, between 2001 and 2010. Importantly, there were two changes in the tax regime that occurred at the start of 2007, as Laun explains:

The first labor tax credit studied in this paper is an earned income tax credit that reduced the personal income tax on labor income only. It was introduced on 1 January 2007 for workers of all ages, with the purpose of increasing the returns from working relative to collecting public transfers. Motivated by the particular importance of encouraging older workers to remain in the labor force, the tax credit is substantially larger for workers aged 65 or above at the beginning of the tax year...

The second labor tax credit studied in this paper is a payroll tax credit for workers aged 65 or above at the beginning of the tax year. Like the earned income tax credit, it was introduced on 1 January 2007... The payroll tax rate for workers above age 65... was reduced from 26.37% in 2006 to 10.21% in 2007. Since then, it only includes pension contributions. The payroll tax credit thus reduced the payroll tax rate for older workers by 16.16 percentage points.

Laun evaluates the effect of the combination of these two tax rate changes on the labour market participation of older people. Specifically, she looks at the impact on the 'extensive margin' -whether older people work or not (as opposed to the 'intensive margin' - how many hours they work, if they are working). She essentially compares workers who are aged similarly, but on either side of the January date on which their tax rate changes. She finds that there is:

...a participation elasticity with respect to the net-of-participation-tax rate of about 0.22 for individuals who were working four years earlier.

In other words, a one percentage point decrease in the tax rate increases labour force participation by 0.22 percentage points. Given that the employment rate just before age 65 appears to be about 63 percent, and the tax rates dropped by around 20 percentage points, the effect of the Swedish tax change amounts to about 4.4 percentage points of additional labour force participation, or an increase of about 7 percent. The results are robust to various other specifications, and are similar to results from other countries in other contexts not related to retirement. Laun also shows that the retirement hazard (essentially similar to the probability of retirement) decreases by a statistically significant amount as a result of the tax change.

However, pension receipt does not change - people are just as likely to receive the pension after the tax change as before. Interestingly, in Sweden pension receipt is voluntary (but universal and not tied to  whether or not an older person is working or to their earnings, similar to the case in New Zealand), and delaying the pension allows a higher amount to be claimed later (a feature of pension systems that many countries have, but New Zealand does not). So, if working longer led to a delay in eligibility for pensions, you can bet that the effect of the tax change would be much smaller (and potentially zero).

The take-away from this paper is that incentives do matter. It is possible to incentivise older people to work longer, even when they remain eligible for the old age pension. However, this sort of change isn't going to make pensions any more affordable unless the value of pensions in real terms is reduced as well. If people are working more, then the pension could potentially be less generous without substantially increasing poverty among older people. However, that doesn't make any changes in this space any easier to introduce politically.