Measuring marketing value through incrementality experiments

Last week Google released CausalImpact, an open-source r package that both improves and simplifies incrementality analysis of time series data. For marketers who do not always have the luxury of A/B tests but still want to be data-driven, this package should be a core part of the toolkit.

Based on original research, the model establishes what’s called a counterfactual market response, i.e. what would have happened had the treatment never happened. This counterfactual is created by establishing a baseline relationship between treatment and control markets which can then be compared to actual data to estimate incremental impact.

Establishing causality is challenging when randomized experiments are not possible. For example:

  • How do television ads affect website visits?
  • How do digital display ads affect retail sales?
  • How do generic search ads affect brand queries?

These are tough attribution problems that are usually ignored (in small companies) or addressed through slow and political media mix modeling engagements (in large companies). Complementing top-down models with an approach such as counterfactual modeling can give marketers an edge by increasing their team’s velocity to test new ideas, reward winners, and discard losers.