An important note of caution on meta-analysis

I've written a number of posts that mention meta-analysis (most recently here), where the coefficient estimates from many studies are combined quantitatively to arrive at an overall measure of the relationship between variables. Meta-analysis has the advantage that, while any one study can give an incorrect impression of the relationship simply by chance, that problem is much less likely when you look at a whole lot of studies. At least, that problem is much less likely when there is no publication bias (which occurs when studies that show statistically significant effects, often in an expected direction, are much more likely to be published than studies that show statistically insignificant effects). Fortunately, there are methods for identifying and correcting for publication bias in meta-analyses.

However, a recent post on the DataColada blog by Uri Simonsohn, Leif Nelson and Joe Simmons (the first of a series of posts), raises what appear to be two fundamental issues with meta-analysis:

Meta-analysis has many problems... But in this series, we will focus our attention on only two of the many problems: (1) lack of quality control, and (2) the averaging of incommensurable results.

In relation to the first problem:

Some studies in the scientific literature have clean designs and provide valid tests of the meta-analytic hypothesis. Many studies, however, do not, as they suffer from confounds, demand effects, invalid statistical analyses, reporting errors, data fraud, etc. (see, e.g., many papers that you have reviewed). In addition, some studies provide valid tests of a specific hypothesis, but not a valid test of the hypothesis being investigated in the meta-analysis...

When we average valid with invalid studies, the resulting average is invalid.

And in relation to the second problem:

Averaging results from very similar studies – e.g., studies with identical operationalizations of the independent and dependent variables – may yield a meaningful (and more precise) estimate of the effect size of interest. But in some literatures the studies are quite different, with different manipulations, populations, dependent variables and even research questions. What is the meaning of an average effect in such cases? What is being estimated? 

Both of these problems had me quite concerned, not least because I currently have a PhD student working on a meta-analysis of the factors associated with demand for healthcare in developing countries. However, I've been reflecting on this over the last couple of weeks, and I'm feeling a bit more relaxed now.

That's because the first problem is relatively easily addressed. The initial step in a meta-analysis is to identify all of the studies that could be included in the meta-analysis. That might include some invalid studies. However, if as a second step we subject all the identified studies to a quality check, and exclude studies that do not meet a minimum quality standard, I think we probably eliminate most of the problems. Of course, some studies with reporting errors, or outright fraudulent results, might sneak through, but poorly designed studies, which fail on basic statistical or sampling criteria, will be excluded. That is the approach that my PhD student has adopted.

The second problem may not be as bad as Simonsohn et al. suggest. Their example relates to experimental research, where the experimental treatment varies across studies, such that averaging the effect of the treatment makes little sense. However, not all meta-analyses are focused on experimental treatments. Some are combining the results of many observational or quasi-experimental studies, where the variable of interest is much more similar. For instance, looking at the effect of income on health services demand, we need to worry about how income (and health services demand) are measured. However, if we use standardised effects in the meta-analysis (so that all estimates are measuring the effect of a one-standard-deviation change in income on health services demand, measured in standard deviations), then I think we deal with most problems here as well. Again, that is the approach that my PhD student has adopted.

None of this is to say that all (or most, or even perhaps many) meta-analyses are bulletproof. It's just that the critiques of Simonsohn et al. may be overplayed. However, it is important to keep these issues in mind. I recommend reading the other posts in the DataColada series on this topic, of which there is just one so far, on meta-analyses of  'nudging', with a promise of more to come.

In the meantime, I think my PhD student can rest a little uneasily, but secure that so far their work is addressing these critiques. But with more to come, I reserve the right to change my mind. Simonsohn et al. are usually quite persuasive, so I'm awaiting a stronger case against meta-analysis from them in the rest of the series. On the other hand, I'm hoping not!

[HT: David McKenzie at Development Impact, among others]

A 30-year waitlist for Kobe beef croquettes, and counting

The next time you find yourself on the waitlist for some product or service, spare a thought for the Japanese consumers waiting 30 years for Kobe beef croquettes. As reported on CNN last week:

If you order a box of frozen Kobe beef croquettes from Asahiya, a family-run butcher shop in Takasago City in western Japan's Hyogo Prefecture, it'll take another 30 years before you receive your order.

That isn't a typo. Thirty. Years.

Founded in 1926, Asahiya sold meat products from Hyogo prefecture -- Kobe beef included -- for decades before adding beef croquettes to the shelf in the years following WWII.

But it wasn't until the early 2000s that these deep-fried potato and beef dumplings became an internet sensation, resulting in the ridiculously long wait buyers now face.

When there is a shortage of some good or service, we usually expect the price to go up (for example, see here). Not only is that not happening in the case of Asahiya beef croquettes, the low price that leads to the shortage is a purposeful business strategy:

"We sold Extreme Croquettes at the price of JPY270 ($1.8) per piece... The beef in them alone costs about JPY400 ($2.7) per piece," says Nitta.

"We made affordable and tasty croquettes that demonstrate the concept of our shop as a strategy to have customers enjoy the croquettes and then hope that they would buy our Kobe beef after the first try."

To limit the financial loss in the beginning, Asahiya only produced 200 croquettes in their own kitchen next to their shop each week.

So, not only are the croquettes being sold at a price that generates a shortage of them, they are being sold at below the cost of production. There are a number of reasons why firms may sell some of their products at below cost, but generally it is because they are a loss leader - the firm sells that product at a loss, and uses it to generate additional customers who then buy other products, which are more profitable. It appears that is the strategy that Asahiya has adopted:

"We hear that we should hire more people and make croquettes more quickly, but I think there is no shop owner who hires employees and produce more to make more deficit... I feel sorry for having them wait. I do want to make croquettes quickly and send them as soon as possible, but if I do, the shop will go bankrupt."

Fortunately, [the business owner, Shigeru] Nitta says that about half of the people who try the croquettes end up ordering their Kobe beef, so it's a sound marketing strategy.

Is it a sound marketing strategy though? You can generate a lot of buzz about your products without creating such a shortage that your customers are waiting 30 years for their purchase. I mean, even now:

Customers receiving croquettes these days placed their orders about 10 years ago.

Surely, they could reduce the size of the waiting list for croquettes from 30 years back to 10 years, or back to one year, and rapidly increase profits? On the other hand, perhaps that would entail a loss of quality, as the article notes:

The cheap price tag of the Extreme Croquettes flies in the face of the quality of the ingredients. They're made fresh daily with no preservatives. Ingredients include three-year-old female A5-ranked Kobe beef and potatoes sourced from a local ranch.

I guess there is a limit to how much you can ramp up production when you use very specific ingredients. But surely, there is scope for the ranch to increase beef and potato production? While loss leading can be a very profitable strategy, there is just so much scope for increasing profits in this case that I'm not convinced that this is a profit maximising business strategy at all. It even makes me wonder, how many other profit-maximisation failures are contributing to the decades of underperformance of the Japanese economy?

Audiobooks and supplier lock-in

In my ECONS101 class, we cover customer lock-in - where firms lock their customers into buying from them, because the costs of switching to an alternative supplier are high. We don't usually discuss the opposite effect - supplier lock-in. So, I was interested to read this recent article in The Conversation by Rebecca Giblin (University of Melbourne) and Cory Doctorow (Open University):

Amazon openly admits to doing everything it can to lock in its customers. That’s why Audible encourages book returns: its generous offer only applies to ongoing subscribers. Audible wants the money from monthly subscribers and wants the fact that they are subscribed to prevent them from shopping elsewhere...

Another way Audible locks customers in is by ensuring the books it sells are protected by digital rights management (DRM) which means they are encrypted, and can only be read by software with the decryption key...

Once customers are locked in, suppliers (authors and publishers) are locked in too. It’s incredibly difficult to reach audiobook buyers unless you’re on Audible. When the suppliers are locked in, they can be shaken down for an ever-greater share of what the buyers hand over.

Notice that there is lock-in on both sides of this market. That can be a characteristic of platform markets, of which Audible provides a key example. Amazon (through Audible) provides a platform where creators and readers connect. Creators want their books to be read (and, importantly, purchased), and they know that readers will look on Audible for audiobooks. Readers want to read audiobooks, and they know that creators put their audiobooks on Audible. Everyone wins. Creators don't want to go elsewhere, because they know that the readers are looking on Audible. Readers don't want to go elsewhere, because they know that the audiobooks are on Audible. Amazon's power as a middleman, due to controlling the Audible platform, gives them an immense amount of market power. 

Giblin and Doctorow refer to Amazon as a 'chokepoint':

The problem isn’t with middlemen as such: book shops, record labels, book and music publishers, agents and myriad others provide valuable services that help keep creative wheels turning.

The problem arises when these middlemen grow powerful enough to bend markets into hourglass shapes, with audiences at one end, masses of creators at the other, and themselves operating as a chokepoint in the middle.

Since everyone has to go through them, they’re able to control the terms on which creative goods and services are exchanged - and extract more than their fair share of value.

A platform provider (Amazon) could conceivably charge both sides of the market for access to the platform. In this case, Amazon charges readers for every audiobook they buy (and don't return - for details on this, read Giblin and Doctorow's article). They don't directly charge creators for making their audiobooks available, but nevertheless Amazon can exploit its market power to reduce the share of the profits that goes to creators. And it appears that that is exactly what they have been doing (and in particularly shady ways, such as encouraging readers to return audiobooks for a refund after they have been read).

Giblin and Doctorow have a new book out, Chokepoint Capitalism, which is the point of their article. I'm looking forward to reading it, as they highlight in the article that:

The whole second half is devoted to detailed proposals for widening these chokepoints out – such as transparency rights, among others...

And we need reforms to contract law to level the playing field in negotiations, interoperability rights to prevent lock-in to platforms, copyrights being better secured to creators rather than publishers, and minimum wages for creative work.

It will be interesting to see how they justify those policy proposals, and how workable they may be. You can expect a book review some time in the future (but given the backlog of my reading, it may be a while!). 

Cultural distance matters more for the spread of democracy than geographical distance

Over the last year, I've been working towards some exciting new projects using measures of cultural distance. So, I was interested to read this new article by Thanos Kyritsis (University of Auckland), Luke Matthews (RAND Corporation), David Welch and Quentin Atkinson (both University of Auckland), published in the journal Evolutionary Human Sciences (open access, with a non-technical summary available on The Conversation). In the article, Kyritsis et al. look at how differences in democracy between countries are related to the cultural and geographical distance between them, and whether changes in democracy over time in one country are related to the level of democracy in other countries that are culturally close and geographically close to the first country.

Kyritsis et al. develop measures of cultural distance based on linguistic distance and religious distance, and use three different measures of democracy: Polity 5 (covering 1800-2018), Vanhanen’s Index of Democracy (covering 1810-2012), and the Freedom in the World index (covering 1972-2020). Overall, their dataset covers 220 years, with 221 modern and historical nations, and 41,638 observations in total. They also distinguish between different waves of democracy:

...an initial ‘slow’ wave beginning in the US and culminating in the emergence of several European democracies at the end of the First World War (1828–1926); a second wave linked to the process of decolonisation following the end of the Second World War (1945–1962); and a third wave comprising a succession of transitions in Western Europe, Latin America, the Pacific, Eastern Europe after the fall of communism, and sub-Saharan Africa (1974 to present)...

In the first part of their analysis, Kyritsis et al. regress the difference in democracy between two countries on the distances between them (linguistic, religious, and geographical). They run this analysis cross-sectionally for each year in their dataset, and find that there were:

...independent effects of all three predictors on each democracy indicator over the time period covered by our data... While the effects of linguistic and religious ancestry were slightly attenuated in this combined model, when present, they remained generally better predictors of all three democracy indicators than geography. Over the last 50 years, linguistic and religious ancestry accounts for up to 12.3% and 17.4% of variance in pairwise differences in democracy, respectively, compared with 1.2% for geographical proximity...

Looking at how the cross-sectional relationships have changed over time, they find that:

...across all three democracy indicators, linguistic ancestry is an increasingly important predictor beginning mid-way through the second wave (circa 1955) and plateauing (or, in the case of Polity 5, declining somewhat) in the third wave from about 1990 to the present. Likewise, religious ancestry becomes an increasingly important predictor of similarity in all three democracy measures from approximately the beginning of the third wave, circa 1975, plateauing and then declining somewhat from circa 2000 to the present... 

The relationships between geographical distance and differences in democracy were not as large as for the two measures of cultural distance, and the relationship was inconsistent over time. Overall, this suggests a stronger effect of cultural distance on democracy than the effect of geographical distance.

Turning to their second analysis, of changes in democracy over time, Kyritsis et al. find that:

The democratic status of nations’ linguistic relatives is the only effect to show a consistently positive trend across all three democratic outcome measures for the duration of the time series. The language ancestry effect is strongest, and statistically significant for most of the third wave of democratisation across all outcome measures. Unsurprisingly, since democratic status tends to persist, most of the variation in nations’ democracy indicators at T2 is explained by their democracy at T1, but linguistic ancestry accounts for a non-trivial component of the remaining variation, explaining up to 17.4% of variation across outcome measures in the third wave. The democratic status of nations’ religious relatives shows no consistent effect until the third wave, when we see a sustained positive trend across all outcome measures, consistent with our cross-sectional analyses, with religious ancestry explaining up to 11.1% of the variation in democratic outcomes during this period. Also in accordance with our cross-sectional analyses, the effects of a nation’s geographical neighbours on its democratic outcomes tend to be positive, although these geographical effects show more variation through time and across outcome measures.

Essentially, the longitudinal analysis results confirm what they found in the cross-sectional analysis over time. Democracy appears to diffuse more readily between cultural 'neighbours' than between geographical neighbours. My future research will build on similar themes, where cultural distance appears to matter more than geographical distance. I look forward to sharing some of that in a future post.

[HT: The Conversation]