Fascinating paper, well worth reading in depth. The summary is that it is pretty well impossible for most companies to run an effective experiment to test whether advertising spend works or not.
Classical theories of the firm assume access to reliable signals to measure the causal impact of choice variables on profit. For advertising expenditure we show, using twenty-five online field experiments (representing $2.8 million) with major U.S. retailers and brokerages, that this assumption typically does not hold.
This makes advertising another casino business – you can make and lose a lot of money just like casinos and investment banking. Interestingly Google’s scale is a solution to one part of this problem, in that they have the sheer scale to run experiments, but they don’t necessarily have the consumer purchase activity to complete the analysis (unless our subsequent behaviour is easily seen: search for camera on Google, purchase in real life, search for camera manual on Google).
Interesting also that this research was carried out by Google and Microsoft researchers.
It reminded me of an earlier paper on the causal effects of advertising (adfx) that I had forgotten that showed that people who saw an ad on a particular day were more likely to do related things (like purchase) but that these activities were not related to seeing the ad. Instead they were doing more that day than they usually did so were more likely to buy.
… in our examples, we see that on some days, a user does more of everything online, and other days, she does less of everything online.
This overall phenomenon, which we call activity bias, leads to severe overestimates of adfx when using observational or quasi-experimental methods.