Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Predictive Pre-Testing: A New Model for Ad Pre-Testing Based on Prediction Markets


Published on

Abstract: In the advertising industry, there's widespread agreement that ad pre-testing (or copy testing) is a creativity killer. Current research methodologies using focus groups to test innovative ideas and products have poor predictive ability, often throwing up lazy groupthink, post-rationalisations and risk-averse generalisations. This paper argues for a revamped model of ad pre-testing that harnesses collective intelligence to improve forecasts of real world response by a significant factor. Drawing insights from a range of business implementations of 'wisdom of crowds', the paper presents prediction markets as a tool to effectively aggregate the responses of a research group that is characterised by diversity of opinion, independent thinking and decentralisation. The updated model takes care to align the interests of research respondents with those commissioning it by incorporating gamification techniques to draw out deeply-held private information about their cohorts and the world they inhabit. The paper also suggests additional areas of exploration and improvement while implementing the concept into research practice.

Published in: Marketing
  • Be the first to comment

  • Be the first to like this

Predictive Pre-Testing: A New Model for Ad Pre-Testing Based on Prediction Markets

  1. 1. Predictive Pre-testing: A New Model for Ad Pre-Testing Based on Prediction Markets Iqbal Mohammed Twitter: @misentropy | Web: Abstract: In the advertising industry, there's widespread agreement that ad pre-testing (or copy testing) is a creativity killer. Current research methodologies using focus groups to test innovative ideas and products have poor predictive ability, often throwing up lazy groupthink, post-rationalisations and risk-averse generalisations. This paper argues for a revamped model of ad pre-testing that harnesses collective intelligence to improve forecasts of real world response by a significant factor. Drawing insights from a range of business implementations of 'wisdom of crowds', the paper presents prediction markets as a tool to effectively aggregate the responses of a research group that is characterised by diversity of opinion, independent thinking and decentralisation. The updated model takes care to align the interests of research respondents with those commissioning it by incorporating gamification techniques to draw out deeply-held private information about their cohorts and the world they inhabit. The paper also suggests additional areas of exploration and improvement while implementing the concept into research practice. Introduction Few suspected culprits are as routinely prosecuted and condemned in public as advertising pre-testing is. If the conversations in the plannersphere (the collection of blogs by advertising planners) are anything to go by, ad pre-testing has assumed Rasputin-like proportions. A scoundrel that everyone detests and one who routinely encounters every known killing device – poison, bullets, lynching, drowning, hanging, etc. - only to survive on. Famous men have been called in to provide the deathblow with their guillotine-edged words – Akio Morita, Aldous Huxley, Henry Ford and David Ogilvy. But to no avail. Which is not to say, ad pre-testing doesn’t have its defenders. Research agencies, who provide the service, parade its benefits – while also pointing out its limitations. The latter only serve to emphasize how their own ‘much improved’ version of the ad pre- testing model overcomes those limitations. And then there are the clients. The reason why – it seems – the discipline of advertising testing itself exists in the first place. Despite the thundering words of some of their ilk (“Nike never pre-tested any of its campaigns, and we took the responsibility of what we were creating rather than passing the buck” – Scott Bedbury) – in popular perception, clients are only too willing to forgo their own prerogative to decide and pass it instead to a group of respondents with nothing to do on a working day afternoon. While the slugfest between its defenders and opponents continues, the discipline itself - and the widely used model of advertising pre-testing - remains ignored. While research agencies routinely add new trinkets to their services, the heart of the contraption itself has undergone very little change. In fact, in the words of the Advertising Research Foundation – the industry body for market research in the US – “For the most part, there's been no wide scale significant innovation in copy testing and tracking (except maybe data collection methods) in 50 years." This paper argues that the time is ripe for the thinking behind advertising pre-testing to be re- visited. It presents an updated model incorporating new paradigms of collaborative thinking and decision-making that are currently gaining ground in business. But, first, here’s a round up of the common charges against ad pre-testing – as it is practiced now. Criticisms of current ad pre-testing models 1. The correlation between the stated intent of respondents and their actual behavior is very low. 2. Pre-testing is a rational exercise that’s fundamentally incapable of capturing emotional responses to advertising. 3. Cognitive research shows that a significant part of the human decision-making process takes place in the subconscious. If that’s the case, a conscious walkthrough of a consumer’s reactions to advertising hardly has any bearing to their ultimate actions. 4. Moderators often end up leading the respondents to express certain points of view. 5. Respondents have a propensity to express snap views and then post-rationalise. 6. Alpha respondents can often sway the opinions of the rest of the group. 7. It’s human tendency to criticize and most of the respondents are keen to do that – even if they have nothing worthwhile to say. 8. The motivations of the respondents aren’t in tune with those of the researchers – the respondents are there for the money or for the free food or, even worse, because they love the sound of their own voice.
  2. 2. A New Model for Ad Pre-Testing Based on Prediction Markets by Iqbal Mohammed 2 9. Respondents are often unable to visualize the presented concepts (animatics, storyboards) in finished form – a big concern when the ad being tested is execution-based. In the words of Simon Clift (global President of Marketing for Unilever - Home & Personal Care), “To me the excessive reliance on animatics is crazy – like choosing your wife from a stick drawing." 10. Ads that are simple to understand and the ones that stand out, tend do well. 11. By definition, new unfamiliar ideas are expected to cause discomfort in an audience. So any new thinking – which all advertising is expected to be – won’t test too well. Thereby defeating the very process of testing. 12. Ad pre-testing is reductive – trying to isolate variables, whereas the effects of advertising itself are emergent. 13. Pre-testing – instead of being used as a diagnostic/optimization tool – often ends up being used as a substitute for judgement. 14. The tendency among clients is to use pre-testing as a tool to tackle the fear of making a mistake – rather than to supplement the decision-making process. 15. Some of the world’s most successful advertising campaigns and products (Sony Walkman, Dyson Vacuum, Aeron Chair, Seinfeld, computer mouse) have failed research. (The research, in these cases, was wisely ignored.) 16. Advertising pre-testing doesn’t match the environment under which consumers watch advertising – as ad breaks between blocks of TV content. And in a synergistic rich real-life environment incorporating press, outdoor, PR, TV, radio, online, word of mouth, etc. Testing that embeds ads in TV content costs a lot more. 17. The use of one-size-fits-all pre-testing models for widely differing kinds of advertising. 18. Some studies claim that there’s a negative correlation between ads that have pre-tested well and those that have won IPA Effectiveness Awards. 19. The Advertising Research Foundation chimes in that “Because of emphasis on cognitive/rational measures, current copy testing techniques cause a regression to the mean, thereby reducing advertising effectiveness.” In short, ARF believes ad pre-testing reduces advertising effectiveness. 20. Research companies feel the need to show value to clients and thereby test the concepts to destruction – only to create a report that’s many times larger than the communication itself. While the list itself seems endless, the above criticisms about ad pre-testing fall under two broad categories. The reliability (or lack of reliability) of information gathering from respondents and the subsequent use of research data itself – not as something that informs the decision-making process but as something that substitutes it. Any attempt to create a better model of ad pre- testing should address these two broad criticisms – either tackle them effectively or define its usability with greater clarity. Wisdom of crowds? Although none of its opponents or its defenders have mentioned as much, but ad pre-testing is a particularly bad implementation of a prediction market using the ‘wisdom of crowds.’ Research agencies, in fact, have tended to de- emphasize ad pre-testing’s predictive abilities and instead focused on the diagnostic and optimization possibilities. But the overwhelming belief that ad pre- testing aims to provide predictive ability is difficult to shake off. Even its nay-sayers seem to think that ad pre-testing cannot predict market success – not that ad pre-testing isn’t designed to predict market success. If indeed ad pre-testing is being viewed and used as a predictive tool – then it’s particularly poorly designed to either predict market success, or to tap into the undeniable wisdom inherent in the crowd, even in a group of focus group respondents. James Surowiecki – who coined the phrase ‘Wisdom of Crowds’ and expanded on the concept in a book of the same name – lists 3 important elements that differentiate the wisdom from the folly of crowds. a. Diversity of opinion: If the group isn’t representing a diverse set of opinions, chances are that it’s going to have a one-sided take on everything. In particular, every person should bring his or her own unique ‘private’ or ‘tacit’ information to the group. This increases the width and depth of information held by the group – thereby maximizing its ability to accurately predict outcomes. b. Independence: The opinions of individuals should be independent and not easily influenced by those of others. In the absence of independence, we get information cascades which destroy the diversity of opinion within the group. c. Decentralization: A particular kind of decentralization, – where self-motivated individuals champion the information they possess, rather than co-operate and compromise explicitly – is also critical to the success of a ‘wisdom of crowds’ enterprise. In addition to the above, a successful ‘wisdom of crowds’ gathering should also incorporate some form of mechanism (like a stock market, for eg.) that aggregates the information within the group, crunches it and provides predictive data that then can be tested against a real outcome. (Contrary to popular belief, this aggregating mechanism need not always be overtly complicated like a stock market. In promotion contests where participants are asked to guess the number of balls in a container – the average of all participants is almost always better than the guess of the best performing participant. In this case, the mechanism of averaging serves to aggregate the collective knowledge of the
  3. 3. A New Model for Ad Pre-Testing Based on Prediction Markets by Iqbal Mohammed 3 group and provides a close-to-accurate result – provided the above 3 conditions are met.) The current models of ad pre-testing flout all the above 3 essential principles of the ‘wisdom of crowds.’ Recruits are often defined with a few paltry variables and end up resembling (and echoing) each other more often than not. Pushy moderators and the focus group itself (with the ubiquitous alpha male/female) create powerful ‘information cascades’ that squeeze out all traces of independent thinking in the group. And focus group dynamics doesn’t give enough incentive for all the participants to champion their own viewpoint passionately – resulting in centralization of thought and opinion, rather than the opposite. Added to that, the aggregation mechanism used in ad pre-testing is a poor collective tool. Instead of dynamically mashing all the information inherent in the group, it merely ends up representing passive data. Prediction markets Decision markets, much like stock markets, are a particularly elegant and well-designed means to aggregate the wisdom of crowds. By providing the means – diversity, independence and decentralization - for the crowd to be smarter than the individual, decision markets are finding application in a wide variety of business situations. Several companies have been putting decision markets to good use. Yahoo has instituted in-house decision markets to help it decide which technologies hold promise for the future. Arcelor – the largest steel producer in Europe and Latin America – uses prediction markets to accurately project quarterly variations in sales volume and prices of steel. When HP ran prediction markets among its employees to forecast computer workstation sales – in 6 out of 8 cases they were more accurate than the internal corporate forecasts. Over the years, Hollywood Stock Exchange ( - a multiplayer online stock market game trading in Hollywood movies and stars – has proved uncannily accurate in the prediction of Oscar winners and box office results. To date, there is no better indication of a movie’s first week loot than the price the movie is trading at on HSX. In fact, Hollywood studios use the information from HSX to make advertising and promotion decisions. Challenges to setting up an ad pre-testing decision market But before we rush to design a prediction market around ad pre-testing, there are many challenges to be considered and overcome. a. All decision markets rely on an infusion of information from the real world – to enable it to decide who the winners and losers are. For example, when employees of Arcelor predict variations in sales volume and price of steel, their predictions are judged against the real market figures when they come in. And the ones getting closest to the target are deemed the winners. But ad pre-testing has no provision for such an infusion of information from the real world. Since ad pre-testing is an exploration to seek what might work well, the ads in question might not even run – and most won’t run in the form they are in. So it’s hard to attribute success or failure to any prediction made during the course of an ad pre-testing session. And even if such information is forthcoming, it may take months to arrive. b. Most decision markets use a variation in wealth in the market as a key motivating factor for participants. Differing wealth (either in play money or in real money) creates differing risk profiles and thereby gives players a reason to dig deep within themselves to seek out the all the information they need to make the right decision. Different levels of wealth also give rise to wide range of strategies – rather than resorting to simple ones. Differences in wealth arise only when the game has been iteratively played over a period of time (most decision markets start with players getting a fixed amount of money.) Since ad pre- testing takes place over one session, it is difficult to introduce the notion of wealth (either with real money or play money) and an unequal distribution of it. More so, to use it to enable different investing strategies. c. All the examples of decision markets mentioned above are software-based and hosted online – and run over prolonged periods of time. They cost a fair bit in terms of investment, and by creating a market for information – they require that almost all information be shared with the participants. Which is the reason why, most corporate implementations of decision markets are usually in-house and for employees only. It seems, the only way to set up a massive multiplayer decision market based on ad pre- testing is to have companies (and often competitors) co-operating across the board and allowing their communication efforts to be visible to everyone. Surely, not something that will come to pass – now or ever. Upstream or downstream? Some of the augmentations to the ad pre-testing model have attempted going further downstream – ignoring what the individual is saying and listening instead to the biological cues like MRI, EEG, etc. By measuring a person’s biological responses rather than his consciously aired views (or by corroborating both), these models claim to capture reactions more accurately. These advances, of course, increase the cost phenomenally – which is why they have few takers. Plus, moving data collection further downstream doesn’t really solve the problem of aggregating it into a cohesive mashed-up whole.
  4. 4. A New Model for Ad Pre-Testing Based on Prediction Markets by Iqbal Mohammed 4 One alternative to current thinking is to move the respondent upstream - not to treat the respondent as a representative of his own opinion but as a barometer of other people’s opinions. By doing that, you’re immediately removing the introspectiveness, and associated issues, that tend to dog focus groups. Instead you are focusing the respondents’ energies on using all his knowledge about his peers and how they would respond to the ad in question. Carnie Mellon University professor Luis Von Ahn has put a similar trick to use to solve the intractable problem of how to label pictures on the net. Left to themselves, people could additionally label a picture of a rose as ‘pretty’, ‘red’, ‘flower’, ‘expensive’, ‘repulsive’ or anything else. Plus, they might make typos, use a different language or even a slang term – making the task of searching the picture back from an archive extremely difficult. If you didn’t know the mind of the labeler – then you would only be guessing at a picture-label combination. To counter this problem, Luis Von Ahn created a game called ‘The ESP Game.’ Two randomly paired online players are shown the same picture and asked to generate labels for it – but with a twist. They are both challenged to guess what labels the other anonymous player is thinking of – and score a point for every correct guess. So, personal preferences and idiosyncrasies go out of the window, and both players are now concentrating their energies on universally accepted attributes of a rose, or whatever else they are labeling. The game retains those labels that both players successfully guessed and throws out all labels that didn’t have a match – this also automatically throws out typos and other mistakes, unless both the players make the same typo, a miniscule probability. It helped that it was an addictive game. Within 4 months of debuting the game, as many as 13,000 players had accurately produced 1.3 million labels for some 300,000 images – all at no cost. Not only that, because of the clever way in which the data was collected, the labels tended to be subtle and nuanced – capturing not just what’s in the picture but the mood of it too. (A search for ‘funny’ returns pictures of Ronald McDonald being hauled away by police and Queen Elizabeth picking her nose.) A few months ago, Luis von Ahn demo’d the game to Google – who have now commercialized it as Google Image Labeler. It is now being put to use to make the company’s database of images better and smarter. The Upstream Advantage As it turns out, taking a leaf out Luis von Ahn’s book and turning the tables around solves many of the crucial issues and challenges that our enterprise encountered. a. Asking each one of the focus group to guess which particular ad the majority of the group will end up liking provides an unexpected bonus – the infusion of information from the real world that was missing earlier. Their votes not only provide their own individual guesses but when aggregated provide the overall guess of the group as a whole – which approximates for data from the real world and serves to decide if the individuals got it right or wrong. b. Asking the respondents to guess what the others in the group might end up liking (instead of what they themselves like) also solves one of the biggest criticisms of ad pre-testing. The authenticity of the respondents’ responses. When they are forthcoming on their own reactions about the advertising, we have no way to correlate that either their own subsequent behavior or with their own genuine but unstated reactions. But when the question is posed as a game (with an incentive thrown in), we have no reason to doubt that they are putting their minds and understanding of other consumers like them (friends, colleagues, relatives) to good use. c. One of the criticisms levelled against ad pre- testing is the inability of respondents to imagine what advertising in unfinished form - animatics, ripomatics, storyboards – might end up looking like. This, it’s rightly argued, results in skewed testing. One of the reasons why respondents are unable to build on advertising in unfinished form is because they don’t see themselves on the side of the creators. They see it as outsiders – thereby discounting all the potential and possibilities the idea has. But when involved in an ad pre-testing exercise as a game to guess other people’s reactions, most respondents are likely to imagine each unfinished ad in the best light – to be able to serve their own interests within the game. In short, they are likely to assess each ad as an insider – to see how other people might react to it, and gain from that. d. Another unexpected bonus of inverting the paradigm is that it multiplies the number of respondents immediately – though we still have the same number of people in the room. Each respondent is now bringing in information from about a dozen of his peers and friends – effectively widening the net for no increase in cost. e. Finally, the paradigm of a game where each one is trying to guess what the overall group is thinking also ensures that we don’t have respondents who are just there for the pocket money and free snacks. The snacks might still be free – but as we’ll see, they will have to work hard to get the money. An outline of an ad pre-testing model based a pari-mutuel prediction market We begin by assembling a group of 10 to 12 individuals based on the demographic requirements. But instead of merely ensuring the group conforms to the demographic requirements – we ensure that there’s also diversity (men, women, professions, ages,
  5. 5. A New Model for Ad Pre-Testing Based on Prediction Markets by Iqbal Mohammed 5 hobbies, etc.) within the defined set. The more diverse the group – the better the results will be. Each individual in the group is handed a sum of money, say $ 25 – real money that they need to wager during the course of the session. What they win, they take back with them. If they lose part or all of it, they return empty-handed. The session itself is broken into 5 rounds. Each round consists of viewing a clutter tape of 5 to 6 ads – after which the moderator asks all respondents to vote for the ad that the majority in the group will like the best. Along with the vote – written down on a piece of paper and handed over to the moderator – each respondent wagers $5 of his money on the outcome in every round. At the end of each round, the votes are tallied and one of the ads shown in the round emerges as a winner. All respondents who voted for the winner will then share the total money wagered in that round. For eg. if 6 people out of 10 voted for a particular ad – then they get to share the spoils (which in this round will be $ 50.) Each of the 6 will get $8.33 back – while the remaining 4 will lose the $ 5 they wagered. The first 2 to 3 rounds ideally are dummy rounds – to get the respondents warmed up and to ensure they get the hang of things. By the end of round 3, the respondents will have a fairly good idea of how the system works – not to vote for the one they like, but to vote for the one with the highest probability of the group’s approval. In doing the latter, they are considering a wider set of variables to judge the ads – and not just one’s own biases and opinions. The ad/ads that are being tested should ideally be introduced round 4 onwards. They could be tested in more than one round under differing criteria – which of the ads will the group like the most, which ad/product will the group most likely buy into or which of these ads will the group consider as one that stands out the most, etc. Or, alternatively they could be tested in more than one round for the same criteria, but surrounded by different sets of ads. At the end of all the rounds, the winners take home the cumulative money they have won in all the rounds. It is possible that some of the respondents will take home nothing, having lost each of the rounds. The results of the ad test rounds will indicate a fairly accurate picture of the groups’ collective reaction to the ads being tested. If all the hygiene factors have been observed until then, the results will indeed be predictive of how the ads will be received by the market at large – because the group in question is working together to arrive at that very same prediction. The results may often be unpalatable to the client and agency in question. And in the currently practiced form of ad pre-testing, that disappointment is usually wished away under a maze of questions, answers, interpretations – and doubts over authenticity of the reactions and the methodology involved. Those routes are unavailable in the prediction market model – the results are stark and clear more often than not (except in the case of close voting). And this is the biggest concern while implementing prediction markets – the ability to handle the truth that emerges. And because of clear and transparent methodology, there’s very little one can do to pad the truth. Additional factors to consider while implementing an ad pre-testing prediction market 1. It’s worth emphasizing once again that the data that turns up in a prediction market is very unlike the reams of data that traditional pre- testing throws up. It’s fairly terse and transparent (in a collection of a certain set of ads, the ad in question totaled so many votes) and aggregated by the dynamics of the game rather than by third party analysis. Layers can be added to data results from a prediction market by testing the same ad in more than one round for different criteria – emotional connect, ease of understanding, hip factor, or buying response. The difference from traditional pre-testing is that you ask the respondents which of these ads will score high on emotional connect (or whatever else) with the entire group – and not just in their personal estimation. 2. While the model presented above mentions only 10-12 participants – there’s no upper limit to the numbers one can recruit. An arbitrary limit, however, will have to be imposed by other factors – coordination and cost. From the point of the prediction market, the more the participants, the more accurately predictive the results will be (as long as they don’t compromise on the clause of diversity.) 3. Similarly, while the above model suggested 5 rounds of testing, one can theoretically add as many rounds as one needs – thereby testing the same ad on various parameters. The limiting factor in this case will be cost. Another factor to consider while using the same ad in multiple rounds is information leakage. If the results of the last round indicate one winner, then participants might lazily pick up the same one is subsequent rounds. One way to stop that happening is to withhold the results from the rounds with the same set of ads but different criteria – until all the rounds are played out. 4. It’s also a good idea to use the first 2 rounds to test ads that are already in market and for which market success numbers are already available. This yields valuable data and enables calibration of the group’s effectiveness and skew (if any), particularly useful when judging the group’s collective decisions on the ads being tested. 5. The first 2-3 rounds can also be used to create a mood for the later rounds where ads will be tested – either by featuring ads in the same
  6. 6. A New Model for Ad Pre-Testing Based on Prediction Markets by Iqbal Mohammed 6 product category or ads featuring similar kinds of execution. Animatics or other unfinished forms of advertising can also be introduced in the earlier rounds to enable the participants to warm up to those techniques. 6. One drawback of putting all the participants in one room is that it could lead to collusion and game-fixing between the participants. Ideally, the participants should be geographically dispersed – preferably each in a different room. Alternatively, 2 to 3 participants can be present in each room and given the chance to consult with each other before putting down their vote. The discussion will make their decisions more rounded, while the numbers will limit the possibility of an information cascade within the larger group. 7. An alternative way to set up the above prediction market is to eliminate the people who don’t get the right answers in each round. It then mandates that you have a sufficiently large number to begin with – to ensure there are more than a handful left for the final rounds. The advantage of eliminating people in each round is that you’ll have the clued-in bunch left behind – minus the nonclued-in ones to cloud the voting. The drawback, on the other hand, is that you run the risk of reducing the group to a homogenous non-diverse one, when the final rounds take place. 8. While the prediction market described above creates variation in wealth as the rounds progress (some participants are winning back more than the $5 they invested in each round and others are losing all of it) it doesn’t actively allow the differing levels of wealth to be leveraged during the rounds (the maximum and minimum one can bet on each round is till $5) or for it to affect tactics. More complicated schemes can be introduced, for eg. where each participant gets 3 votes per round – and he can vote for multiple ads, while also wagering different amounts of money on each vote. Such schemes, however, might end up confusing participants – and therefore making the results seems awry. 9. Unlike traditional ad pre-testing that also serves as a diagnostic and optimization tool, an ad pre- testing prediction market only provides predictive data about what the group thinks. However, post the session, the participants – especially the ones with the largest prize money, indicating the best performers – can also be quizzed in regular ad pre-testing format to enunciate what they think about the ad/ads being tested. After having spent a full session considering the advertising in question from all possible angles, they’ll be primed to provide useful information on what they think works and what doesn’t – and why. 10. In traditional pre-testing, the participants are led to indicate additional insights into what part of the advertising might not be working and why. In an ad pre-testing prediction market, the advertisers might have to do the hard work of throwing up all alternatives – in the same ad or as different concepts – right at the beginning. Prediction markets don’t excel at throwing up alternatives – but in choosing which one of the alternatives works the best. Some of the alternatives to consider – and test – are different ways the same plot unfolds or different call to action approaches, etc. Conclusion For a new ad pre-testing model to be considered as an advance, we isolated two key improvement areas. Enhance – by a huge quantum – the reliability of information collected and, two, prevent the misuse of results. The ad pre-testing prediction market described above scores very well on the first count. By inverting the paradigm of introspection and by adding the elements of a game, it provides the incentives (financial and otherwise) for participants to seek and arrive at the best predictions. Any skews the data might throw up are more likely the result of the group lacking diversity or numbers (or both) and are unlikely to be the outcome of an individual’s shortcomings. By clearly focusing only on predictive data about an ad’s likely market success – it also solves the second requirement – the prevention of pre-testing’s misuse. Clients and their agencies can choose to accept or ignore the prediction markets results – but with the process being completely transparent and clear, they can hardly use it to suit a pre-conceived agenda. Advertisers seeking diagnostic and optimization data for the ad campaign they are measuring have two options. They can either run the additional discussion sessions described in Point 9 of the section above or choose to run a traditional ad pre- testing exercise. The availability of a prediction market as a pre- testing option especially serves to clarify the objective of a pre-testing exercise, right in the beginning. Are we seeking predictive data to supersede our own judgement or qualitative data that can inform our decisions? This clarity in objective then dictates which one of the two methodologies we should pursue. It’s a choice that wasn’t available. Until now. ABOUT THE AUTHOR IQBAL MOHAMMED is a brand and marketing strategist whose area of specialization lies at the intersection of advertising, information systems and economics. He is the winner of the WPP Atticus Award 2006 for best original published writing in the 'Branding and Identity' category. To subscribe to email updates of his latest papers, visit
  7. 7. A New Model for Ad Pre-Testing Based on Prediction Markets by Iqbal Mohammed 7 REFERENCES (A linked references list can be found online at  James Surowiecki, 2004. The Wisdom of Crowds: Why the many are smarter than the few. Abacus Publishers.  Fredrik Sarnblad, 2006. Goodbye Innovation at 2006/05/09/ goodbye-innovation/  ‘Workers, Place Your Bets’ Businessweek, August 3rd 2006 at ug2006/tc20060803_012437.htm  ‘Hollywood Games People Play’ Businessweek, August 7th 2006 at ug2006/tc20060804_618481.htm  Leland Maschmeyer, 2007. Predictive markets better than ad testing? at 03_01_archive.html  Nigel Hollis, 2006. Is the Link pre-test the equivalent of the Smith & Wesson Magnum 500? at the-link-pre-test-the-equivalent-of-the-smith-wesson- magnum-500/  Jason Lonsdale, 2007. Great Quotes for Planners #22 - Huxley on pre-testing at 03/great_quote_for.html  Jason Lonsdale, 2007. Great Quotes for Planners #41 - Akio Morita on pre-testing at 12/great-quotes-fo.html  Ed Cotton, 2007. Arnold creatives show us just what they think of focus groups at old-creatives-show-us-just-what-they-think-of-focus- groups.html#comments  Jason Oke, 2007. Pre-testing at g.html  Nigel Hollis, 2007. In defense of pre-testing at defense-of-pre-testing/  Jason Oke, 2007. Pre-testing, part II at g_part.html  Clive Thompson, 2007. ‘The Human Advantage’, WIRED 15.07 at 07/ff_humancomp What do they know of cricket who only cricket know? - CLR James, 'Beyond A Boundary' ACKNOWLEDGEMENTS Thanks to Leland Maschmeyer for seeding the thought of advertising pre-testing using prediction markets via his blog post ‘Predictive markets better than ad testing?’ dated March 25, 2007 on his blog, Whistle Through Your Comb. Though initially skeptical that predictive markets would work in this context, I eventually changed my mind and this paper is the outcome. META Suggested Citation: Mohammed, Iqbal, Predictive Pre- Testing: An Outline for an Ad Pre-Testing Model Based on Prediction Markets (September 8, 2008). Available at: This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 2.5 License. To view a copy of this license, visit or send a letter to Creative Commons, 543 Howard Street, 5th Floor, San Francisco, California, 94105, USA.