A/B testing in the wild

In ECON100 we briefly discuss A/B testing - where you provide different versions (of a website, an advertisement, a letter, etc.) to different people, then evaluate how those different versions affect people's interactions with you. A common example is online shopping, where different pictures of products on the website might be more effective at inducing sales. Or different formats for a direct mail letter might induce more donations for an NGO. And so on. It's hard to say how prevalent A/B testing is, but I would imagine that most online businesses should be engaging in it at least at some level. It's a pretty low-cost and evidence-based way of increasing sales, revenues, profits, etc.

This week, I was pointed to this blog piece by Dillon Reisman about A/B testing in the wild. He describes using a web crawler to unveil how websites are making use of A/B testing (or to be more specific, how websites using the A/B testing provider Optimizely are using A/B testing). Here's one example:

A widespread use of Optimizely among news publishers is “headline testing.” To use an actual recent example from the nytimes.com, a link to an article headlined:

“Turkey’s Prime Minister Quits in Rift With President”

…to a different user might appear as…

“Premier to Quit Amid Turkey’s Authoritarian Turn.”

The second headline suggests a much less neutral take on the news than the first. That sort of difference can paint a user’s perception of the article before they’ve read a single word. We found other examples of similarly politically-sensitive headlines changing, like the following from pjmedia.com:

“Judge Rules Sandy Hook Families Can Proceed with Lawsuit Against Remington”

…could appear to some users as…

“Second Amendment Under Assault by Sandy Hook Judge.”

 Here's another:

Many websites target users based on IP and geolocation. But when IP/geolocation are combined with notions of money the result is surprising. The website of a popular fitness tracker targets users that originate from a list of six hard-coded IP addresses labelled “IP addresses Spending more than $1000.” Two of the IP addresses appear to be larger enterprise customers — a medical research institute a prominent news magazine. Three belong to unidentified Comcast customers. These big-spending IP addresses were targeted in the past with an experiment presented the user a button that either suggested the user “learn more” about a particular product or “buy now.”

Connectify, a large vendor of networking software, uses geolocation on a coarser level — they label visitors from the US, Australia, UK, Canada, Netherlands, Switzerland, Denmark, and New Zealand as coming from “Countries that are Likely to Pay.”

Non-profit websites also experiment with money. charity: water (charitywater.org) and the Human Rights Campaign (hrc.org) both have experiments defined to change the default donation amount a user might see in a pre-filled text box.

I guess the take-away is that we should probably assume that all websites are experimenting on us, pretty much all the time. And it's not just limited to Facebook's secret psychological experiments.

[HT: David McKenzie at Development Impact]