A/B testing vs. long shots, and funding of research

One of the things we discuss in the ECON100 topic on pricing strategy is A/B testing (yes, it isn't strictly limited to pricing, but we cover it in that topic nonetheless). A/B testing occurs where you provide different versions (of a website, an advertisement, a letter, etc.) to different people, then evaluate how those different versions affect people's interactions with you. In a recent blog post (about slowing productivity growth), Tim Harford provides a couple of examples:

But the same basic approach — using quick-and-dirty experiments, or “A/B testing” — has paid dividends elsewhere. David Cameron’s Behavioural Insight Team, known unofficially as the “nudge unit”, has used simple randomised trials to improve the wording of tax demands and the advice given to job seekers. Google tested 41 shades of blue for its advertising hyperlinks. Designers rolled their eyes — then Google claimed that the experiment had netted an extra $200m in annual revenue. As Mr Haldane says, marginal improvements can add up.

Of course, notwithstanding the eye-popping gains claimed about Google's blue links experiment, the gains from A/B testing are marginal. Offering different versions of a website can add to a firm's profits, but it isn't really a feasible way to test out a radical new idea. Harford writes:

An alternative view is that what’s really lacking is a different kind of innovation: the long shot. Unlike marginal gains, long shots usually fail, but can pay off spectacularly enough to overlook 100 failures. The marginal gain is a heated pair of overshorts, the long shot is the Fosbury Flop. If the marginal gain is a text message nudging you to finish a course of antibiotics, the long shot is the development of penicillin. Marginal gains give us zippier web pages; long shots gave us the internet.

These two types of innovation complement each other. Long shot innovations open up new territories; marginal improvements colonise them. The 1870s saw revolutionary breakthroughs in electricity generation and distribution but the dynamo didn’t make much impact on productivity until the 1920s. To take advantage of electric motors, manufacturers needed to rework production lines, redesign factories and retrain workers. Without these marginal improvements the technological breakthrough was of little use.

For productivity gains, we need a mix of both the marginal gains (such as from A/B testing) and the long-shot gains from radical new ideas. Harford only notes in passing that a lot of the long shots of the past arose from research that was either funded by, or at least well supported by, government. Consider as one example all of the spinoff technologies from NASA. Or the Internet, without which Google couldn't have run its blue links experiment at all.

This hasn't always been the case. As this excellent history of research funding notes, prior to World War II a lot of research was funded by private philanthropy. But since WWII, governments have provided the bulk of research funding, particularly for basic research (i.e. research that may not have an immediate application). Many argue that firms aren't interested in basic research, because it doesn't add to their bottom line. Even a lot of applied research is risky for firms to engage in, since it involves often large up-front costs with no certainty of a payoff at the end (see for example my earlier post on the economics of drug development). Which would explain why firms are happier to engage in A/B testing to capture marginal gains, than to chase long shots.

Although that might all be changing, as this article from Science earlier this year notes:

For the first time in the post–World War II era, the federal government no longer funds a majority of the basic research carried out in the United States. Data from ongoing surveys by the National Science Foundation (NSF) show that federal agencies provided only 44% of the $86 billion spent on basic research in 2015. The federal share, which topped 70% throughout the 1960s and ’70s, stood at 61% as recently as 2004 before falling below 50% in 2013.

The sharp drop in recent years is the result of two contrasting trends—a flattening of federal spending on basic research over the past decade and a significant rise in corporate funding of fundamental science since 2012. The first is a familiar story to most academic scientists, who face stiffening competition for federal grants.

Much of the privately-funded basic research is in pharmaceuticals and biotechnology. So, maybe that's where we should expect the next generation of long-shot gains to arise? And if we want long-shot gains in other areas, perhaps it is incumbent on the government to steer some funding in those other directions.