Quant Online Pre-Testing: Lessons from Warren Buffett

Posted on by
Share

Quant online pre-testing is very popular; in the week 25th to 31st January 2012 I was invited to take part in at least 29 pieces of research from just 3 research panels:

  • 4 Ipsos Surveys
  • 7 MySurveys (Millward Brown)
  • More than 18 Survey Network surveys

I don’t have the time to take part in these studies as each one takes around 45 minutes to complete but I did have some down time between Christmas and New Year and did a few:

I concept tested a product innovation in butter that is simply brilliant, I hope makes it to market as genuine innovation in a category that is over 3000 years old is truly impressive! I also pre-tested an animatic for a 2012 Christmas ad for a popular soft drink that was distinctly reminiscent of Ghostbusters.

The anamatic for a possible 2012 Christmas ad for a popular soft drink brand was reminiscent of this scene from Ghostbusters.

Aside from the confidentiality issues of using online panels to pre-test concepts and ads there are other significant problems:

So Why is Online Quant Pre-Testing Popular?

I’m reading The Warren Buffett Way and a single sentence made sense of why:

“An investor needs to do very few things right as long as he or she avoids big mistakes.”

Warren Buffett in The Warren Buffett Way

On reading this sentence it all became clear: Confidentiality and effectiveness are being sacrificed to minimise the risk for shareholders. Quant online pre-testing exist to serve the shareholder’s needs, not the marketing manager’s.

There is certain irony in the fact that the investment bankers and pension fund managers who benefit most from these tests are not even aware of their existence or the problems they cause.

So What?

The rules have changed, if you want to make advantage of ‘free’ online impressions you need a big idea. By their very nature big ideas are inherently risky and are culled by online quantitative pre-testing.

So What Now?

We combine eye tracking and facial coding with depth interviews to find out what people really do, NOT what they think they do. Our behavioral economics approach is big idea friendly and produces actionable insights:

…my learning from working with THiNK Eye Tracking is that eye tracking is different, it can lead to very specific actionable results that lead to business benefits.

Dan White, CMO Millward Brown

 

  • Rory Sutherland

    I think the approach you should take to risk in advertising creative depends on a few factors.

    If you are Tide in the US, have 70% of the detergent market, and plan to outspend your competition 3:1 then a reasonably low-risk approach makes sense. Your potential upside is fairly small and your potential downside is huge.

    If you are Old Spice, the opposite applies. You may disappear or be delisted unless you do something remarkable. On the other hand the possible upside is enormous. There you should be far braver.

    The other approach is the barbell approach Taleb recommends. If you are Tide, spend 80% of your money very conservatively and experiment with 20% of it on higher risk ideas……

  • https://www.thinkeyetracking.com Robert Stevens

    Hi Rory,

    Quant online pre-testing is especially valuable to category captain such Tide and Tropicana, indeed I’m interested to know if the Tropicana fiasco was pre-tested as I expect it was not: If it had been it *SHOULD* have been weeded out and saved shareholders $100’s millions of lost value. This is where Millward Brown reigns supreme; Link testing is the best solution for circumstances where the downside is enormous and the upside is small.

    I believe quant online pre-testing is less useful for brands that are failing, in danger of being delisted and in need a big idea, such as Old Spice and Hovis. But these cases are unusual. There are many more brands that are simply treading water, Cadbury’s Dairy Milk for example. Sales were flat lining in in 2007, there was an over reliance on price promotion and a big pressure on price. As we all know the solution was Gorilla that delivered an ROI of £4.88 for every £1 spend. However It took six months to convince internal stakeholders to run the campaign, this wasn’t helped by the fact that the ad scored low on Millward Brown’s Link test branding score. When it was released the ad achieved the highest ever branded recall ever with a score of 96%. The IPA have the case study here: http://vimeo.com/31794837

    Millward Brown has a great position in the market as risk mitigators for many of the world’s best brands, but Link is not the best solution for creative work that challenges the boundaries of advertising: Millward Brown had ample opportunity to put the case for Link testing supporting creative ads in an article in an article in Admap Nov 2011. Instead of making a compelling case (when given editorial free reign) an interpretation of the data presented is that Award Winning Adverts are Irrelevant and Incomprehensible?