Quant Online Pre-Testing: Lessons from Warren Buffett


Quant online pre-testing is very popular; in the week 25th to 31st January 2012 I was invited to take part in at least 29 pieces of research from just 3 research panels:

  • 4 Ipsos Surveys
  • 7 MySurveys (Millward Brown)
  • More than 18 Survey Network surveys

I don’t have the time to take part in these studies as each one takes around 45 minutes to complete but I did have some down time between Christmas and New Year and did a few:

I concept tested a product innovation in butter that is simply brilliant, I hope makes it to market as genuine innovation in a category that is over 3000 years old is truly impressive! I also pre-tested an animatic for a 2012 Christmas ad for a popular soft drink that was distinctly reminiscent of Ghostbusters.

The anamatic for a possible 2012 Christmas ad for a popular soft drink brand was reminiscent of this scene from Ghostbusters.

Aside from the confidentiality issues of using online panels to pre-test concepts and ads there are other significant problems:

So Why is Online Quant Pre-Testing Popular?

I’m reading The Warren Buffett Way and a single sentence made sense of why:

“An investor needs to do very few things right as long as he or she avoids big mistakes.”

Warren Buffett in The Warren Buffett Way

On reading this sentence it all became clear: Confidentiality and effectiveness are being sacrificed to minimise the risk for shareholders. Quant online pre-testing exist to serve the shareholder’s needs, not the marketing manager’s.

There is certain irony in the fact that the investment bankers and pension fund managers who benefit most from these tests are not even aware of their existence or the problems they cause.

So What?

The rules have changed, if you want to make advantage of ‘free’ online impressions you need a big idea. By their very nature big ideas are inherently risky and are culled by online quantitative pre-testing.

So What Now?

We combine eye tracking and facial coding with depth interviews to find out what people really do, NOT what they think they do. Our behavioral economics approach is big idea friendly and produces actionable insights:

…my learning from working with THiNK Eye Tracking is that eye tracking is different, it can lead to very specific actionable results that lead to business benefits.

Dan White, CMO Millward Brown


Swimming: Lessons Beyond Drowning


Too many children were drowning in Western Australia; the solution was called ‘Every Pupil a Swimmer’ provided training for all children to swim to a basic level of survivability. The solution worked in that the number of children drowning decreased but the solution had a cost, the number of Olympic swimmers for Western Australia dropped significantly. In setting a basic minimum standard there had been an unforeseen consequence of curtailing those with the potential to be Olympians.


I believe the same thing is happening in the world of advertising for major brands: Traditional quantitative online pre-testing of adverts set minimum standards that reject the best creative ideas. This has led to adverts from P&G, Unilever, J&J and other major brands to be survivors rather than Olympians. I’m not alone in this belief, in a meeting with Rory Sutherland he told me:

“(Millward Brown’s) Link testing is effective at getting rid of the worst 20% of adverts but in doing so it also culls the best 20%.”

Rory Sutherland Chairman Ogilvy

Our beliefs are supported by one of the largest studies of creative work in the world ‘Marketing in the Era of Accountability’. In which the respected econometrician found that:

“Campaigns that are quantitatively pre-tested seem to be about half as effective in business terms that those that aren’t.”


Les Binet Marketing in the Era of Accountability

This lack of effectiveness may have played a part in 1,600 P&G staff from non-manufacturing roles recently losing their jobs. P&G CEO Robert McDonald told Wall Street analysts in an interview:

“In the digital space, with things like Facebook and Google and others, we find that return on investment of the advertising when properly designed, when the big idea is there, can be much more efficient.”

Robert McDonald CEO P&G

There are two problems with McDonald’s statement:

Research from the IPA and Deloitte proves that online works best when used in conjunction with TV, indeed TV + the Internet is recognised as a dream marketing combination, not least by the host of online brands who advertise on TV.

Traditional quantitative online pre-testing culls big ideas: In a recent AdMap article Dominic Twose (Global Head of Knowledge Management for Millward Brown) was only able to make a tenuous link between creativity and Millward Brown’s Link test despite give editorial free reign to do so. I’d bet my last £1 that if there was any substantial link between Link testing and creativity that Dominic would have made it clear in the article. Instead one interpretation of the data presented is that award winning adverts are irrelevant and incomprehensible.


Traditional online quantitative pre-testing inadvertently culls big ideas resulting in ads that are 50% less effective. I believe that is why Olympic quality brands from P&G, Unilever, J&J and many more are surviving rather than winning and without winners more people will lose their livelihoods.

The Pitch

Thanks for reading, now the pitch! If you have ever suffered the pain of failing a quantitative pre-test for a campaign that you just knew was in the top 20% you’ll find working with Bunnyfoot a pleasure: Our research methods are based in the behavioural economics of what people really do rather than what they think and are well suited to pre-testing the top 20% of ideas.
We have completed over 1,300 research projects and Rory Sutherland describes us as “Brilliant!”

Award Winning Adverts Irrelevant and Incomprehensible?


It may come as a surprise but this insight comes from Millward Brown in the recent article “Creative Effectiveness” (ADMAP, Nov, 2012).

In the article Millward Brown writes that ads which won IPA, Effie and Cannes Lions awards score highly on three of13 criteria that are part of a Millward Brown Link test:

  • Enjoyment
  • Involvement
  • Different to other adverts

The data presented by Millward Brown appears to support this. Continue reading

Gamification of Market Research


In a The Emperor’s New Sneakers I noted that sampling bias and Subject Expectancy effect causes erroneous results in online panel research. There is another problem with traditional question-based market research, Rory Sutherland, the Chairman of Ogilvy, summed it up nicely in an interview June 2011:

“The conscious, rational brain isn’t the Oval Office. It isn’t there making executive decisions in our minds. It is actually the press office, issuing explanations for actions we’ve already taken.” Continue reading

The Emperor’s New Sneakers


EyeTrackShop claim to be able to eye track a home based panel of participants using standard low resolution webcams. It’s a big claim that could revolutionise market research.
We were interested to find out if EyeTrackShop really works so we replicated an EyeTrackShop study with two fundamental differences:

  • We used specialist eye tracking equipment costing over $20,000 each, as used by thousands of academics so we know the eye tracking results are accurate.
  • We recruited participants using an in-street intercept rather than a self-selected online panel. Continue reading

Discovery Has No Merit Unless It Can Be Explained To A Barmaid


Ernest Rutherford won a Nobel Prize in Chemistry in 1908 before becoming the first person to split the atom in 1917, he went on to discover and name the proton. The chemical element Rutherfordium (element 104) was named for him in 1997.

There is no doubt Rutherford was a smart man, he was also an outspoken one. My favourite quote by him is:

“An alleged scientific discovery has no merit unless it can be explained to a barmaid.”

We want to apply the Rutherford test to neuro market research and invite companies in the field of EEG, FMRI and similar technologies to join us in an experiment at The Cherry Tree in Steventon near The Rutherford Appleton Laboratory in Oxfordshire UK.

Continue reading

Is Market Research Precisely Wrong?


There is a fundamental problem with most market research. David Ogilvy, the ‘Father of Advertising’, recognised it:


“People don’t do what they say, don’t say what they think, and don’t think how they feel.”

Traditional methods of market research focus on what can be gleaned from the conscious mind largely because until recently the tools to investigate the subconscious mind were not readily available.

THiNK Eyetracking has recently completed a comparison study of an award wining national advertising campaign that clearly shows the difference in conscious and unconscious responses to advertising. Continue reading