The case for!moderation!!A view on pre-testing research!!!James Hurman!Head of Planning, Colenso BBDO, New Zealand!Author of The Case for Creativity
Disclaimer…This isn’t an attempt to persuade you to abandon pre-testing research!It’s a discussion about how best to interpret and apply the conclusionsand recommendations that come out of pre-testing research.
Why are great brands so skeptical of pre-testing? “We don’t ask “We do no market “We never pretested “Geoﬀ believes consumers what research. We ﬁgure anything we did at Nike, research is a blunt they want. They out what we want. none of the ads. Dan instrument that don’t know. Instead And I think we’re Wieden (the founder of bludgeons good we apply our brain pretty good at Nike’s agency Wieden & ideas to death. He power to what they having the right Kennedy) and I had an was determined that need and will want discipline to think agreement that as long 42 Below would and make sure we’re through whether a as our hearts beat, we never be subjected there, ready.” lot of other people would never pretest a to pre-testing.” are going to want it word of copy. It makes - Akio Morita, Sony too. That’s what we you dull. It makes you - Justine Troy on founder get paid to do.” predictable.” Geoﬀ Ross, founder of 42 - Steve Jobs, former - Scott Bedbury, Nike’s Below Vodka Apple CEO former worldwide advertising director
We humans aren’t great at picking successes…“It’s a great invention, but who would want to use it?”- US President Rutherford HayesSix years later, as the telephone was transforming America…“The Americans have need of the telephone, but we do not. We haveplenty of messenger boys.”- British Post Ofﬁce’s Chief Engineer
We humans aren’t great at picking successes…“Everyone acquainted with the subject willrecognize it (the light bulb) as a conspicuousfailure.”- The President of Stevens Institute of Technology,notable for producing several Nobel Prize winners,1880
We humans aren’t great at picking successes…“We don’t like their sound, and guitarmusic is on the way out anyway.”Decca Records’ leading A&R manexplaining his rejection of theBeatles, 1962
We humans aren’t great at picking successes…“The market researchers concluded that no other product had everperformed so poorly in consumer testing: the look, taste and mouth-feelwere regarded as ‘disgusting’ and the idea that it ‘stimulates mind andbody’ didn’t persuade anyone that the taste was worth tolerating.”- Philip Graves on Red Bull in his book Consumer.ologyIn the two decades that followed, Red Bull sold over three billion cans ofits ‘disgusting’ drink, achieving sales of €2.6B.Source: Consumer.ology by Philip Graves
We humans aren’t great at picking successes…“Americans aren’t interested in Swedish vodka, with many peopleunaware of where Sweden even is.”- Conclusion of Manhattan’s Carillon Importers $80,000 AbsolutVodka pre-testing research projectAbsolut went on to sell over 70 million litres of vodka to the USannually.Source: Consumer.ology by Philip Graves
A history of pre-testing research• Conceived in the 1950’s by American psychologist Horace Schwerin• He created a research product he called ‘persuasion testing’ and sold it to advertisers as a way to measure the potential sales impact of an advertisement• The method was analysed by university researchers in 1965 and found to be barely more reliable than ﬂipping a coin.Source: Excellence in advertising: the IPA guide tobest practice by Leslie Butterﬁeld, p17
A history of pre-testing research• In the 1990s, Beecham (now GSK), carried out a long term global review of advertising testing methods.• They concluded “It ought to be emphasised that no reliable pre-testing technique exists for assessing the sales effectiveness of a speciﬁc advertisement.” • In 2004, researchers from the London Business School noted that there was still “no evidence in the public domain that pre-testing is predictive.”Source: Why Pre-Testing is Obselete by Tim Broadbent,published in Admap Magazine, October 2004
A history of pre-testing research Source: “Marketing in the Era of Accountability” by Les Binet & Peter Field• In 2007, the UK’s Institute of Practitioners in Advertising produced the largest ever study of historical marketing effectiveness case studies (880 in total). They compared pre-tested campaigns with those that weren’t pre-tested.• “Beware of pre-testing”, the study concluded. “If pre-testing really did lead to more effective campaigns, then one would expect cases that reported favourable pre-testing outcomes to show bigger effects than those that did not. In fact the reverse is true. Cases that reported favourable pre-testing results actually did signiﬁcantly worse than those that did not.”• This shows that the judgement of marketers is in fact much more reliable than positive pre-testing outcomes
A history of pre-testing research “I can never get a positive result. No matter how I cut the data, no matter how I stack the odds in favour of pretesting by doing ﬁne cuts of the data, I can never get a result that says that pre-tested campaigns are more effective than non-pretested campaigns.” “If pre-testing really did work, we should at least get some positive correlations, but we only ever get negative ones. After a while you think, well, there’s an obvious conclusion to draw from all of that.” - Peter Field, author of ‘Marketing in the Era of Accountability’ Source: Interview with the author
Equally, there are all sorts of pre-testing successesAmong others, Cadbury Gorilla and Old Spice ﬂewthrough pre-testing, and went on to become highlyeffective campaigns.It isn’t that pre-testing always gets it wrong.It’s just very difﬁcult to predict whether the pre-testingconclusions are right or wrong.So why is pre-testing so ﬁckle?
We are biased toward the familiarAmerican social psychologist Robert Zajonc studied what he called ‘theexposure effect’ in the 1970s. His experiments showed that simplyexposing subjects to a familiar stimulus led them to rate it morepositively than other, similar stimuli that had not been previouslypresented. In one experiment, people were shown a random sample of squiggledrawings. Some time later, they were shown the same sample, but thistime the squiggles were placed randomly among a selection of other,similar squiggles. The subjects were asked whether they couldremember which of the squiggles were the ones they were previouslyshown. As you’d expect, they had a hard time with the exercise andrarely chose correctly. Then they were asked to show the interviewerwhich squiggles they preferred. They found this test considerablyeasier, and unbeknownst to them, chose the squiggles that they’d seenthe ﬁrst time around.Zajonc’s work concluded that people tend to develop a preference for things merely because they’re familiar with them. Source: Affective Discrimination of Stimuli That Cannot Be Recognized”, Kunst-Wilson & Zajonc, published in Science, Vol. 207
We tend to be wrong about what we think we wantGoogle asked customers how many results theywanted the search engine to throw back on the ﬁrstpage.“Since conventional wisdom says more is always better,people naturally said ‘more’. When Google tripled thenumber of results, however, it found that trafﬁc actuallydeclined. Not only did the results take a fraction of asecond longer to load, but having more options ledpeople to click on links that were less relevant. Therespondents in Google’s research didn’t intentionallylead researchers down the wrong path; they just didn’tunderstand the real-world implications of their choices.”- Steve McKee, BusinessWeek, 2010.
Often things we dislike ‘grow on us’There’s an example of this effect that most of us are familiar with. Upon initially listening to a new album, we prefercertain songs to others. On subsequent listens this preference usually changes, and our long term favourite songstend not to be the ones we liked at ﬁrst."It is easy to slip into the comfortable belief that I like it comments after the ﬁrst exposure of a new execution are amust. In researching Levis executions over the years, it has become abundantly clear that such ﬁndings should betreated with extreme caution.”- Kirsty Fuller, Managing Director of RDS International Research, in 1995.Source: Walking the creative tightrope: the research challenge”, Kirsty Fuller, published in Admap Magazine, March 1995
Often things we dislike ‘grow on us’eg Levi’s ‘Swimmer’"In pre-launch qualitative research, response from the consumer on ﬁrst exposureto Swimmer was one of stunned silence. The heros status was initially seen to beseriously undermined - he did little to earn his colours. Moreover the slow musicdid not have the immediate appeal of previous executions. Perhaps the most ﬁttingdescription of response was disappointment. Swimmer broke the mould of thecampaign to date, and consumers claimed not to like it.”“At this stage the weight of negative reactions was strong. Then two months afterairing, research uncovered a marked shift in response. Swimmer had become atalking point: new, different, challenging. A further four months later and Swimmerwas being widely described as one of the best ever Levis ads, destined to liveamong the greats, such as the universally acclaimed Launderette.”“Research must therefore seek to evaluate the potential of an execution, not itsimmediate impact on one viewing. Challenging advertising is not necessarily eitherimmediately liked or fully understood. It may however, be rich and long-lasting.”
So…• A long history of pre-testing research being studied and proven unreliable• Pre-testing often gets it right, but it’s very difﬁcult to predict when that will be the case• Pre-testing is hampered by a few inconvenient realities…• We’re biased toward the familiar, not the effective• We often think (and will report) we want things that we actually don’t• Pre-testing only offers a ‘ﬁrst impression’ whereas new ideas tend to ‘grow on us’• The numbers show that marketers’ judgment is signiﬁcantly more reliable than positive pre-testing outcomes
The case for moderationAlcohol has positive and negative effects.When we use it moderately, it’s great.When we use it immoderately, we get into trouble.Pre-testing is the same.It’s useful when used as part of a wider decision anddevelopment processBut when used ‘immoderately’ – as a decision maker –it’s dangerous.Let’s be moderate in how we use the outcomes fromour pre-testing research.