For B2B Market Research Professionals               November 20, 2009              The Online Panel Quality Wars: All Mark...
The Online Panel Quality Wars: All Market Researchers Benefit — Especially B2B Tech                                     2F...
The Online Panel Quality Wars: All Market Researchers Benefit — Especially B2B Tech                                     3F...
The Online Panel Quality Wars: All Market Researchers Benefit — Especially B2B Tech                                     4F...
The Online Panel Quality Wars: All Market Researchers Benefit — Especially B2B Tech                                     5F...
The Online Panel Quality Wars: All Market Researchers Benefit — Especially B2B Tech                                       ...
The Online Panel Quality Wars: All Market Researchers Benefit — Especially B2B Tech                                       ...
Upcoming SlideShare
Loading in …5

Forrester Paper: The Online Panel Quality Wars: All Market Researchers Benefit — Especially B2B Tech


Published on

Forrester Paper discussing the merits of different data quality solutions for online Market Research.

Published in: Business, Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Forrester Paper: The Online Panel Quality Wars: All Market Researchers Benefit — Especially B2B Tech

  1. 1. For B2B Market Research Professionals November 20, 2009 The Online Panel Quality Wars: All Market Researchers Benefit — Especially B2B Tech by Brad Bortner with Ellen Daley and Chétina Muteba Executi v e S ummary At the same time that online quantitative research has taken off, dubiousness abounds about the representation of the online panel sample. Buyers ask: Are panels representative of the markets researchers use them for? What can we do about the moral hazard (greatly increased for B2B tech) that leads bad actors to take surveys inappropriately? A wide variety of approaches have been percolating in the market ranging from “it’s not really a problem” to various comprehensive solutions. Buyers are finally pushing back. Major buyers — such as Microsoft and Proctor & Gamble — are making very specific panel quality demands for their research vendors, and many of the major full-service research vendors and panel providers are taking note. The end result? Three years of equally competing solutions with no clear direction for buyers. This means market research professionals must be very specific about demanding high quality panel for all of your research needs, or risk having your findings undercut when your internal customers ask: “Is this analysis really representative of our market?” A Dodgy Sample is the unwelcome guest at the online market reseaRch party Online, panel-based research has quickly gone from the poor stepchild of market research to the dominant mode for quantitative research in the US and is now undergoing rapid growth in Europe.1 Why the rapid growth? It’s faster, cheaper, and delivers innovative new capabilities.2 The fly in the ointment of all of this growth is suspicion about how representative online panels really are. This is not just a matter of maintaining a representative panel; it is a matter of ensuring that respondents are who they say they are, answer accurately, and do not take a survey more than once — under multiple aliases. While the Internet greatly simplified the job of recruiting and managing panels, it also opened the door to potential problems based on the moral hazard, the so-called “professional survey taker” problem. What is going on? · Bad actors, in general, are incented to behave badly. Online surveys generally pay per survey. Those respondents who are financially motivated may attempt to take many surveys to make as much as possible. Some accomplish this goal by signing up for many different panels, perhaps under different identities. Others may disguise themselves to appear more attractive to a survey, or sign up for the same panel under multiple personas. The end result can be one person taking the same survey multiple times (especially when panels are mixed), creating invalid responses by people pretending to be what they are not. There are even cases of bad actors creating “survey bots” to harvest fees by responding to large numbers of surveys.3 The end result is corrupted insights. Headquarters Forrester Research, Inc., 400 Technology Square, Cambridge, MA 02139 USA Tel: +1 617.613.6000 • Fax: +1 617.613.5000 •
  2. 2. The Online Panel Quality Wars: All Market Researchers Benefit — Especially B2B Tech 2For B2B Market Research Professionals · The bad actor and moral hazard problem is worse for B2B. The incentives are much larger for B2B surveys — $2 for B2C versus $20 or more for B2B — this is even greater in B2B technology areas because there are fewer qualified potential respondents and a great desire to survey them by tech vendors, resulting in a greater potential of people claiming they have roles and responsibilities they do not to harvest the incentives.4Are these really problems? Yes.5 While vendors are making strong efforts to address the issue, it’s abuyer-beware market at this early stage. Since many buyers can’t see the effects of bad quality, whichare often subtle and hard to detect, they haven’t been informed shoppers and chosen — or told theirfull-service vendor to choose — panel X over panel Y.6However, major research buyers have recently been quite vocal about the need to fix this issue andare becoming savvier. In particular, large, sophisticated online research buyers’ demand for qualityis galvanizing research providers and giving a lift to third-party solutions for panel quality issues inthis market. In addition, industry groups are in the game as well. For example, ARF, ESOMAR andothers are trying to assess and improve this situation by providing best practice guidelines to panelvendors, and ARF has just released its own assessment to size the issue. 7A Broad Array Of Approaches Offer Solutions — None Dominate . . . YetMultiple solutions and approaches to the problem of panel quality have been developing over thepast few years. These include panel consolidation, widespread adoption of machine fingerprinting,panels with unique approaches to quality, and more comprehensive automated solutions. Whileup to now there has been little industry consensus as to which approach is best, each of these fourapproaches for buyers has been helping align the market behind enhanced quality: · Panel mergers allow use of ultra-large panels. As Forrester predicted would happen in 2008, in part because of the scale needed to invest in panel quality, several major panel merges have occurred, including Greenfield/Toluna and e-Rewards/Research Now.8 Really large panels have an advantage in that they don’t have to worry about de-duping against panel properties they don’t own, and can focus on cleaning a panel they wholly own. This helps, but does not eliminate the need to resolve issues around identity verification and assessing the validity of respondents’ answers. · Machine fingerprinting is gaining wide acceptance. Machine fingerprinting involves using multiple identifying marks on a computer to ensure that no computer signs up to take the same survey more than once. Many major panels are now using it, such as e-Rewards, Toluna, and SSI. This is certainly a good step to take; however, it does not resolve the issue that computers generally don’t take surveys, people do. It is possible for surveys to be taken from multiple computers, or to attempt to disguise machine fingerprints.9 Some of the major vendors in this space include Peanut Lab’s Optimus and RelevantID.November 20, 2009 © 2009, Forrester Research, Inc. Reproduction Prohibited
  3. 3. The Online Panel Quality Wars: All Market Researchers Benefit — Especially B2B Tech 3For B2B Market Research Professionals · The use of unique identity confirming practices. E-Rewards is a leader in using by invitation- only panel creation as opposed to open recruiting on the Internet and qualifying people as they ask to join the panel. Western Wats also uses a form of invitation-only recruit by using excess phone capacity to recruit. LinkedIn’s unique approach is to recruit B2B panel respondents from its highly profiled social network.10 The end result for these solutions should be panels whose members are much more verifiable. None, however, provide a saleable solution, especially when more samples are needed then they can provide. · Solutions that are more scalable have been expanding. MarketTools’ TrueSample has been advocating a three-part “unique, identifiable, and engaged” solution. This approach attempts to go beyond pure machine fingerprinting. The first of the three parts compares the sample against publically available databases to ensure that the respondent exists. They then use machine fingerprinting. Finally, they assess survey-taking behavior to see how engaged the respondent is. Their claim is that up to 30% of samples get scrubbed when they go through this process. This approach is salable, deployable against multiple panel sources, and works with many major online survey tools such as Confirmit and SPSS. Unfortunately, many have been resistant to work with a company they view as a research competitor.Why has consensus about which solution to back proven elusive? Two reasons: cost and competitiveforces. First, most panels have a disincentive to filter their sample for quality issues if it would reducetheir panel by up to 30%, and most buyers are not asking for it. Second, to the extent that solutions aremarketed by nominal competitors, major research firms are resistant to partnering with them.Buyer Demand is Forcing Movement toward Sample Quality solution consensusThe rumbling by major research buyers has been growing for a few years, but is now growing to afever pitch. Recent industry sessions about panel quality at ARF and other associations have beentumultuous and quite well attended.11 Recently, major sample buyers, who have a lot of clout in thismarket, are starting to insist on specific solutions or at least specific, verified high quality sampleoutcomes.Major Buyers Are Demanding, And Providers Are ReactingThe pressure from big research buyers is driving several reactions in the market. Full-service vendorsare concurring, or offering to come up with their own solutions, and panel vendors are doinglikewise. Even the machine fingerprinting solutions are starting to have a frontrunner, as panels seek anonthreatening partner to answer buyer’s needs for cleaner samples. Where are the lines being drawn? · Big buyers are forcible pushing for change, often to favor TrueSample. This includes companies such as ARSgroup, General Mills, Microsoft, and Proctor & Gamble. After extensive internal analysis, Microsoft has announced it will only do business with market research vendors that useNovember 20, 2009 © 2009, Forrester Research, Inc. Reproduction Prohibited
  4. 4. The Online Panel Quality Wars: All Market Researchers Benefit — Especially B2B Tech 4For B2B Market Research Professionals TrueSample. General Mills has taken a similar stand. P&G is not yet forcing all of its vendors to use TrueSample, but its criteria are such that that would be an obvious solution for its vendors, and Nielsen BASES has agreed to be a TrueSample certified vendor for P&G. ARSgroup has also decided to only use TrueSample for its massive online sample needs. Given that these sample buyers are quite large, their decisions could potentially start to coalesce the market. 12 · GfK Group, IPSOS, Kantar, and Synovate formed “The Consortium” to address panel quality. When P&G, after significant analysis, approached its major vendors (all which have major private online panels) and requested enhanced panel quality, it decided to team up to come up with its own solution. The results of this activity remain to be seen. On one hand, these are major full-service research houses that have very deep pockets and resources. On the other hand, they are competitors with each other. It is even possible that they may elect to sign on with one of the other panel quality initiatives. Only time will tell. · Panel providers and some full-service vendors are choosing sides. Toluna/Greenfield has committed to providing TrueSample-certified sample to clients that wish it. MRops, a major B2B research operations house, has standardized on Optimus. The bottom line for these, and other market intermediaries, is that they will ultimately adopt the measures that their clients demand. · RelevantID is emerging as a machine fingerprinting frontrunner for panel vendors. Peanut Labs was really a first mover in the machine fingerprinting market. There is a sense that it is being held back by panel companies’ perception that it is a competitor. RelevantID has made significant progress among panel vendors in the past year since it provides a similar solution and is not perceived as a competitor, while Peanut Lab’s strength is in different parts of the research ecosystem.The Next Three Years Will Have Equally Competitive SolutionsThe future holds better, higher quality sample! At the moment, TrueSample seems to have the windat its back, but the final resolution in the market will depend on part on what The Consortiumcomes up with. Can these competitors actually work together to build a better mousetrap, or willthey simply license an off-the-shelf solution such as TrueSample? Will more buyers demand sucha comprehensive solution, or will they be comfortable with panel vendors best efforts and machinefingerprinting? While Forrester believes ultimately one solution will at last dominate, that won’thappen for a while. Expect these solutions to equally compete for the next three years. What are thepossible scenarios after that time? · TrueSample triumphant! More and more buyers follow their peers and insist that their entire sample be TrueSample-certified so they take the discussion about panel quality off the table. At a certain point, it becomes the industry standard.November 20, 2009 © 2009, Forrester Research, Inc. Reproduction Prohibited
  5. 5. The Online Panel Quality Wars: All Market Researchers Benefit — Especially B2B Tech 5For B2B Market Research Professionals · The Consortium strikes back! The major full-service market research vendors come up with their own solution that is equal or superior to TrueSample, and this satisfies the requirements of major research buyers. Once such a solution exists, they could either set it up as a business in its own right or keep it as a competitive advantage. · Panel vendors go it alone, with machine fingerprinting-plus. It is still possible that panel vendors could assert that machine fingerprinting, combined with other internal quality initiatives, is good enough. Since the measures of quality sample are often up to argument, and many buyers can’t even decide if somewhat better sample actually drives better business decisions, this is a possible outcome if major buyers do not coalesce around one specific solution in their sample quality demands. R ecommendations this is not the time for complacency, demand good sample now! Now that transparency is possible, it is critical that buyers demand high-quality sample both from their panel providers (if they have in-sourced quantitative research) and the full-service vendors they are using. This is especially true for B2B sample in the tech industry, which is a rare and eagerly sought after commodity. Fundamentally, bad sample risks driving bad decisions, and that is something which no market researcher professional wants to be responsible for. · If you have a direct relationship with the panel provider, demand and test for quality. Find out how they are weeding out professional survey takers, de-duping when they merge panels, and ensuring that panelists are engaged. Be sure to trust, but verify, by actually insisting that there are tests in your surveys to catch respondents who are giving irrational answers. · If you are dealing with a full-service research provider, it’s not the time to play trust me. Insist that they provide all of the same quality approaches as described above. Have them detail how much sample is being scrubbed. If the answer is “almost none,” be suspicious. If it is a B2B survey in the technology space and they give that answer, find another vendor. · Keep an eye open for the next wave. All of the solutions in play are focused on fixing quality issues with online panels. As with many technology-driven advances, sometimes the market moves on to new paradigms just as the old one gets perfected. Keep an eye open to see if the market is moving on from panels to more of a river sample or other forms of surveying that are more focused on replicating the random sample experiences that phone can deliver offline in an online mode. Perhaps surprisingly, the CEO of one major panel vendor believes that such is the future, and is driven by a combination of ubiquitous Web access and the limit to the current populations who will agree to join panels.November 20, 2009 © 2009, Forrester Research, Inc. Reproduction Prohibited
  6. 6. The Online Panel Quality Wars: All Market Researchers Benefit — Especially B2B Tech 6For B2B Market Research ProfessionalsEndnotes1 Source: Inside Research, Issue 249, February 2009.2 Initially, because using online sample and tools reduced costs by up to 75% and cut the time to results by about one-third versus traditional phone and in-person quantitative data gathering modes. Additionally, the online tools that enabled surveys of online panels greatly simplified fielding complex survey designs, including skip patterns, complex analytics (such as conjoint), pictures, and multimedia. See the July 19, 2007, “Five Keys To Success With Online Panels” report and see the May 1, 2007, “Online Panels: Cheap And Effective, But Buyer Beware” report3 This type of activity is not uncommonly launched from certain developing nations.4 The major drivers of B2B tech sample quality issues are that it is a very scarce commodity, with a very high moral hazard of incenting bad actors. See the April 17, 2009, “Watch Out For The Landmine Of B2B Online Research” report.5 Add to this the innocent panel members who just happen to like to take a lot of surveys. They may well also sign up for multiple panels and be accidently swept into the same survey when panels are combined to reach large sample goals. If they are not de-duped properly, even these non-bad actors can end up having their responses counted more than once.6 The effects of bad actors on surveys can range from obvious illogical responses to an over-reporting of product control or utilization (to qualify for the survey) and averaged responses, since respondents are trying to intuit how they should answer if they actually had that role or used that product.7 ARF embarked on a detailed analysis of single panel bad sample, and found that they could indentify 20% corrupt sample in a typical panel. ESOMAR issued detailed panel management guidelines to try to abate the issue. comScore continues to stand by its assessment that a very small percentage of respondents are responsible for many of the responses for some panels. MarketTools has analysis that shows the problem is closer to 30%. Admittedly, MarketTools and comScore can be claimed to be self-interested in their findings, but there is a lot of industry activity for something that some panel providers would consider a nonissue.8 We made the prediction that panels are likely to consolidate in 2009. See the December 2, 2008 “Predictions 2009: What Happens In Market Research?” report.9 Many vendors of such solutions would argue that this is not that likely.10 LinkedIn’s entry into the B2B (especially tech) sample space has drawn a lot of attention. Since people do not join their community to take surveys, there should be far less moral hazard, and their sample is potentially very well profiled given the wealth of information members provide when they join. The open question about LinkedIn, however, is can it scale big enough to meet sample demand since providing B2B sample is not its core business.11 These have been notable for leaving participants unhappy in that industry players could not seem to agree on an approach.November 20, 2009 © 2009, Forrester Research, Inc. Reproduction Prohibited
  7. 7. The Online Panel Quality Wars: All Market Researchers Benefit — Especially B2B Tech 7For B2B Market Research Professionals12 Decision facilitated by version 3.0 of its platform, which works with major survey platforms, such as Confirmit, SPSS, and MarketTools, and can clean sample on the fly, even from non-TrueSample certified panels.Forrester Research, Inc. (Nasdaq: FORR) is an independent research company that provides pragmatic and forward-thinking advice to global leaders in businessand technology. Forrester works with professionals in 20 key roles at major companies providing proprietary research, customer insight, consulting, events, andpeer-to-peer executive programs. For more than 26 years, Forrester has been making IT, marketing, and technology industry leaders successful every day. Formore information, visit© 2009, Forrester Research, Inc. All rights reserved. Unauthorized reproduction is strictly prohibited. Information is based on best available resources. Opinionsreflect judgment at the time and are subject to change. Forrester®, Technographics®, Forrester Wave, RoleView, TechRadar, and Total Economic Impact aretrademarks of Forrester Research, Inc. All other trademarks are the property of their respective companies. To purchase reprints of this document, please For additional information, go to 55747