Wednesday, 8 June 2016

A/B testing in the wild

In ECON100 we briefly discuss A/B testing - where you provide different versions (of a website, an advertisement, a letter, etc.) to different people, then evaluate how those different versions affect people's interactions with you. A common example is online shopping, where different pictures of products on the website might be more effective at inducing sales. Or different formats for a direct mail letter might induce more donations for an NGO. And so on. It's hard to say how prevalent A/B testing is, but I would imagine that most online businesses should be engaging in it at least at some level. It's a pretty low-cost and evidence-based way of increasing sales, revenues, profits, etc.

This week, I was pointed to this blog piece by Dillon Reisman about A/B testing in the wild. He describes using a web crawler to unveil how websites are making use of A/B testing (or to be more specific, how websites using the A/B testing provider Optimizely are using A/B testing). Here's one example:
A widespread use of Optimizely among news publishers is “headline testing.” To use an actual recent example from the, a link to an article headlined:
“Turkey’s Prime Minister Quits in Rift With President”
…to a different user might appear as…
“Premier to Quit Amid Turkey’s Authoritarian Turn.”
The second headline suggests a much less neutral take on the news than the first. That sort of difference can paint a user’s perception of the article before they’ve read a single word. We found other examples of similarly politically-sensitive headlines changing, like the following from
“Judge Rules Sandy Hook Families Can Proceed with Lawsuit Against Remington”
…could appear to some users as…
“Second Amendment Under Assault by Sandy Hook Judge.”
 Here's another:
Many websites target users based on IP and geolocation. But when IP/geolocation are combined with notions of money the result is surprising. The website of a popular fitness tracker targets users that originate from a list of six hard-coded IP addresses labelled “IP addresses Spending more than $1000.” Two of the IP addresses appear to be larger enterprise customers — a medical research institute a prominent news magazine. Three belong to unidentified Comcast customers. These big-spending IP addresses were targeted in the past with an experiment presented the user a button that either suggested the user “learn more” about a particular product or “buy now.”
Connectify, a large vendor of networking software, uses geolocation on a coarser level — they label visitors from the US, Australia, UK, Canada, Netherlands, Switzerland, Denmark, and New Zealand as coming from “Countries that are Likely to Pay.”
Non-profit websites also experiment with money. charity: water ( and the Human Rights Campaign ( both have experiments defined to change the default donation amount a user might see in a pre-filled text box.
I guess the take-away is that we should probably assume that all websites are experimenting on us, pretty much all the time. And it's not just limited to Facebook's secret psychological experiments.

[HT: David McKenzie at Development Impact]

No comments:

Post a Comment