In God we trust; all others, bring data

Several months back, Wired posted an essay on the rise of A/B testing within tech companies. Nothing groundbreaking for analytics practitioners, but hearing war stories—both successes and failures—never gets old.

On culture and HiPPOs:

This was the culture that he had come from at Google, what you might call a democracy of data… Siroker explains, “if an engineer had an idea and had the data to back it up, it didn’t matter that they weren’t the VP of some business unit…” Once adopted, that approach will beat the HiPPOs every time, he says. “A/B will empower a whole class of businesses to say, ‘We want to do it the way Google does it. We want to do it the way Amazon does it.’”

On local maxima and balancing incrementalism:

Google’s Scott Huffman cites this as one of the greatest dangers of a testing-oriented mentality: “One thing we spend a lot of time talking about is how we can guard against incrementalism when bigger changes are needed. It’s tough, because these testing tools can really motivate the engineering team, but they also can wind up giving them huge incentives to try only small changes.”

On letting data speak for itself:

At the gaming network IGN, for example, executives found that crisp, clear prose was outperforming hyped-up buzzwords (like free and exclusive) on certain parts of the homepage. But in previous years, the opposite had been true. Why? They talked and talked about it, but no one could figure it out. Soon they realized that it simply didn’t matter. A/B would guide them at ground level, so there was no need to worry about why users behaved in one way or another.

On the role of intuition:

It’s a false dichotomy, of course, to pose vision against data, lofty genius against head-down experimentation, as if companies are forced to choose between the two. Every firm ought to test the small stuff, at least; and no firm should (or does) use A/B for everything. Google doesn’t test things at random but relies on intuition and, yes, vision to narrow down the infinite number of possible changes to a finite group of testable candidates.

A/B testing is the latest in a long-line of buzzwords dating back to 1958, when Hans Peter Luhn coined the term “business intelligence” and the rise of decision support systems in the 1960s. Reading data warehouse literature, it’s easy to be struck by the recurring themes. Whatever the latest fad in analytics—bandit algorithms have been hyped and de-hyped lately—the challenges remain the same: culture, personalities and politics.