Esomar 26 Questions - FAQ for Epinion Online Panel


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Esomar 26 Questions - FAQ for Epinion Online Panel

  2. 2. ESOMAR is the global organization for pro-fessional market research agencies.They have challenged the industry by for-mulating 26 questions all research buyersshould ask when buying online surveys.Here are our answers.If you have further questions, don’t beafraid to challenge us.EPINION ONLINE PANEL FAQ 2
  3. 3. >3 What experience does your company sources which will provide us with the sort of 01 panelists that may at any given time be in have with providing online samples for market research? short supply in our pan we have created aYEARS IN OPERATION global quarantine system that ensures that no The Epinion sample only technology has been potential panelist wherever that person in operation in Denmark for more than 3 years, (browser) is located around the world gets and recently we passed our first milestone invited to our panel more than once pr. month. which was 2 million completed interviews on As a result, for each of our country panels, we>2.0MILLION INTERVIEWS WAS the system. We build our first online panel back in 2001 and has been refining and devel- oping our online methodologies and measure and define quality in three dimen- sions: approaches ever since. Current size of active panelistsCOMPLETED BY THIS SYSTEM Current monthly inflow of new panelists Current active recruitment channels Please describe and explain the types (we aim at 75 or more open channels pr. coun- 02 of source(s) for the online sample that try) you provide (are these databases, actively managed panels, direct mar- keting lists, web intercept sampling, What do you consider to be the primary 03 river sampling or other)? advantage of your sample over other sample sources in the marketplace? Our strategy is to resemble as closely as possible the principle of equal sample- The competitive advantages of the Epinion extraction probability known from traditional recruitment strategy in terms of delivering off line research. Our recruitment ensures that superior quality sample originate from two any online population over an extended period distinct traits of the business. of time is equally exposed to invitations to join First, we keep very close attention to maintain- the panel (in no cases, we allow people to opt ing a large number of different recruitment in to the panel without being invited). We do channels in each country as this is the only this by making sure that we have a large way to resemble the random probability sam- number of open recruitment channels at all pling techniques that is used in good quality times. Moreover, our system makes it possible offline interviewing. We use the term ‘resem- for us to direct our recruitment efforts to bling’ off line sampling, as it is obviously not inEPINION ONLINE PANEL FAQ 3
  4. 4. any real sense possible to create a known, If the sample source is a panel or data- 04 well defined, universe from which to draw base, is the panel or database used respondents. However, representativeness is solely for market research? If not, all about giving internet users an please explain? (approximated) equal probability to be exposed for an invitation to join the panel and later on The panel is only used for market research - to answer surveys. The Epinion approach nothing else. The sample source is a panel adheres strictly to this principle. consisting of panelists that have actively Secondly, the unique Epinion sample builder opted in to participate in the Epinion panel. makes it possible to stratify sample so precisely that it is rarely necessary to post- stratify data (weigh data) according to demo- How do you source groups that may be 05 graphics. This is so, because: hard-to-reach on the internet? Regardless of what combination of Soci- We source these groups by contacting sites odemographics that are used to define a that are visited primarily by hard to reach sample (say, women 15-29 years of age), the target groups. For example, in many countries system ensures that all the subsegments in young low-educated males are a hard to reach the underlying sample-matrix (region, educa- target group. In these countries, we make a tion, income etc.) are representatively distrib- special effort in reaching out to local football- uted based on the latest statistics from the sites where we know that this group is over- relevant National Bureau of Census. represented. This is so because the system automati- cally monitors whether the required number of What are people told when they are 06 interviews in each of the subsegments in the recruited? samplematrix are reached. If not, more invites are distributed to the specific subsegment in People are first asked a series of questions question. This repetitive process ensures mainly related to background demographics incredibly representative sample. and then they are faced with the following In combination, these advantages make for question: superior sample compared to traditional online Would you be interested in becoming a sampling. member of Epinions shared opinions panel?EPINION ONLINE PANEL FAQ 4
  5. 5. 8.0 Yes, I would like to join Please describe the opt-in process 08 No, thank you. The exact wording of the question is slightly In order to become a member of Epinion‘sINTERVIEWS ON AVERAGE OF different for each person, since we have long internet panel you have to be invited by Epin-A PANELIST LIFE TIME ago established that different types of argu- ion. We do not accept voluntarily signups at ments appeal to different socio demographic our website or at other places. All respondents groups. Young people should be recruited must complete a 4 step verification process to using a different approach than old people. become members:4STEPS TO OPT-IN 07 If the sample comes from a panel, When invited, first of all: 1. You have to accept our Terms & Condi- what is your annual panel tions, turnover/attrition/retention rate and 2. Then you have to accept to participate in> Accept Terms and Conditions how is it calculated? our establishment survey, 3. You need to provide us with full demo-> Accept Survey Participation We do not have an exact number for panel graphic background information in the estab- retention. Actually, we don´t see panel reten- lishment survey,> Provide demographic background tion as a goal in and of itself. We do not 4. Provide us with your e-mail address and believe in the panel retention strategy of many then one more time, double confirm that you in> Provide contact email address other online research companies. It may be deed wish to become a member of the panel. good for profits and it is even used as a sign of quality to build up a narrow pool of people Unless all of the above steps have been com- making good money out of answering ques- pleted using Epinion recruitment system (no tions. But data from professional panelists affiliate recruitment is allowed), we do not does not mirror the average consumer. There- consider the applicant to be a member of fore, we only use prize draws for small Epinion’s panel. amounts of money as monetary incentives. However, as we take care to give respondents good feedback opportunities, appealing invites and not too long surveys, we have many panelists that complete multiple surveys. The average lifetime number of completed surveys pr. panelist is approximately 8 surveys.EPINION ONLINE PANEL FAQ 5
  6. 6. Do you have a confirmation of identity difficult thing to handle when making cross- 09 procedure? Do you have procedures to country surveys). As a result, it is possible to detect fraudulent respondents at the make exact comparisons across different time of registration with the panel? If countries between for example: highly edu- so, please describe? cated vs. low educated or between occupa- tional categories. Since we do not pay individual monetary In the early days of our panel-buildup, we rewards to respondents, we can rely on e-mail profiled the panelists with lots of different identification as our main identification proce- variables: Car ownership, newspaper- dure. All invitations are distributed / linked to a subscription etc. However, it turned out that unique e-mail address. Without access to this out of perhaps 150 variables we only made e-mail address, you cannot participate. good use of a few. And often we found that As part of the survey process we monitor the variables were obsolete when we actually fraudulent respondents using a series of needed them. In the meantime, the panelist methods described in greater detail in ques- have bought a new car, changed newspaper- tion 24. Also, we select and invite panel mem- subscription etc. Therefore, we now take a bers, they don’t select us. Respondents more dynamic approach to profiling meaning cannot find ways to self recruit, thus making it that we in ad hoc surveys reach narrow target more difficult for fraudulent respondents to groups by use of a sophisticated screening create multiple accounts. system. When wording screeners we take care to avoid self-selection bias. For instance by using a wording like: What newspaper do What profile data is kept on panel you read? (and then present a range of news- 10 papers where one of them is the newspaper in members? For how many members is this data collected and how often is question). Instead of asking: “Do you read this data updated? newspaper A. Yes or No?”. Also, if clients want to create a specific subpanel, we profile The Respondents are profiled with socio- over a short period of time a large part of the demographic criteria. The exact wording of the panel with the requested profile. profile questions is developed carefully and is in line with the acclaimed World Values Survey. This makes sure that categories are compat- ible across different countries (elsewise a veryEPINION ONLINE PANEL FAQ 6
  7. 7. 211,000 What is the size and/or the capacity of Census Bureau. 11 the panel, based on active panel mem- The system automatically monitors whether bers on a given date? Can you provide the required number of interviews in each ofCURRENT ACTIVE PANEL an overview of active panelists by type the subsegments in the samplematrix areMEMBERS of source? reached. If not, more invites are distributed to At all times, one can see the current number the specific subsegment in question. This of active panelists, the current inflow of new repetitive process ensures incredibly repre- panelists and the number of active recruitment sentative sample. Vietnam 41 channels in all of our country panels at Sweden 24 An active panelist is defined in a rather narrow Explain how people are invited to take 13 way meaning that is a panelist that have part in a survey. What does a typical Finland 15 responded to a survey invitation within the invitation look like? Norway last 6 invitations. 21 People are invited to surveys by way of email- Ukraine 5 invitations. We work with acclaimed email Please describe your sampling process newsletter-type companies and consistently 12Denmark 105 including your exclusion procedures if work hard to maintain an industry-leading applicable. Can samples be deployed mail-reputation which is crucially important forMEMBERS OF ONLINE PANEL as batches/replicates, by time zones, mail deliverability among leading email-(IN THOUSAND) geography, etc? If so, how is this con- providers (Hotmail, Yahoo etc.). trolled? We operate with many different types of invitations depending on the type of survey. The sampling process is at the very heart of But under no circumstances can an invitation what makes Epinion very special in terms of be sent out without active decline and panel- online quality sampling. opt out links as well as the accept link itself. Our clients or we can on behalf of the client define a sample (say, women 15-29 years of age) and then the system will ensure that all the subsegments in the underlying sample- matrix (region, education, income etc.) are representatively distributed based on the latest statistics from the relevant NationalEPINION ONLINE PANEL FAQ 7
  8. 8. Please describe the nature of your How often are individual members 14 15 incentive system(s). How does this contacted for online surveys within a vary by length of interview, respondent given time period? Do you keep data characteristics, or other factors you on panelist participation history and may consider? are limits placed on the frequency that members are contacted and asked to Over the cause of the years, we have done participate in a survey? quite a bit of research on what incentives work the best. As a result, we have structured Yes, we apply very stringent respondent our incentive/motivational system around the quarantine practices. No respondent is invited following: to more than one survey every 12 days. They Monetary incentives: We almost always only only receive one reminder per survey invitation use prize draws. We draw frequent winners We save full statistics on invitations sent, but we award only small amounts of money at responses (accept, decline), mail-server statis- a time. We´ve found this works better than tics, participation records on all respondents infrequent, bigger, awards. Even more, our as long as they are members of the panel. findings suggests that micropayments to all participants do increase participation a little bit Is there a privacy policy in place? If so, 16 but first and foremost, it creates more flawed what does it state? Is the panel com- answers, more fraud, and above all a pool of pliant with all regional, national and professional panelists which in no meaningful local laws with respect to privacy, data way can be said to be representative for the protection and children e.g. EU Safe average consumer. Harbour, and COPPA in the US? What Feedback/Motivational incentives. After each other research industry standards do survey, each respondent are prompted to give you comply with e.g. ICC/ESOMAR immediate feedback both by ranking the International Code on Market and survey and by giving verbatim feedback. This Social Research, CASRO guidelines provides the project manager with valuable etc.? feedback, but even more importantly, it gives respondents an opportunity to vent whatever We adhere to national and local laws, and feelings they have about the survey. This adds beyond this, we follow ESOMAR´s International to our general “know-and-appreciate-your- Code on Market and Social Research. respondent” policy. In some instances, we offer the respondent access to selected results from the survey and other specifically targeted incentives.EPINION ONLINE PANEL FAQ 8
  9. 9. 30-45%PARTICIPATE RATE 17 What data protection/security meas- ures do you have in place? 19 Do you conduct online surveys with children and young people? If so, please describe the process for obtain- Epinion is concerned with safeguarding the ing permission information gathered through our Services. We employ a variety of measures designed to In line with the ESOMAR code of conduct, we protect your information from unauthorized do not commission surveys among children access. We work only with wellknown and unless we have asked their parents for1-25%RANGE OF DROP-OUT RATE professional database suppliers and only authorized personnel are allowed to conduct database operations. permission. Do you supplement your samples with 20 samples from other providers? How do Do you apply a quality management you select these partners? Is it your 18 system? Please describe it policy to notify a client in advance when using a third party provider? Do As most of our survey processes is embedded you de-duplicate the sample when10-15% in sophisticated IT-technology much of the quality control is embodied in the IT-development process itself. In this process, using multiple sample providers? Occasionally, we use some of our partners toAVERAGE DROP-OUT RATE a variety of techniques has been put to use in provide sample in countries where our own manual as well as automated testing. panels are too small or over-extended. We only In terms of work-procedures between different use Partners from a preferred supplier list. departments and individuals and with respect That is, trusted partners with proven quality to the content of deliveries designed and and service. As we do not split-sample produced by our consultants or other staff, between our own panel and partner-panels, we our processes and deliveries are internally do not need to deduplicate sample between audited every 12 months. two different panels.EPINION ONLINE PANEL FAQ 9
  10. 10. Do you have a policy regarding multi- gestions to improve the content of the survey 21 panel membership? What efforts do and we very often penalize surveys with high you undertake to ensure that survey drop-out rates. results are unbiased given that some It is our philosophy to maintain a healthy individuals belong to multiple panels? relationship with the respondents to ensure repeat business. In order to maintain that Our panel management system automatically relationship, we often reject long surveys (or controls that there are no duplicate emails. In insist that they broken down into several addition, in each survey, by way of browser- independent surveys), and evaluate the session, IP-address and cookie-tagging, we language used in the surveys using the make sure that no respondent can answer any WYMAT-principle (Would Your Mother Approve given survey (including tracking surveys) more This?). than once. Do you maintain individual level data 23 What are likely survey start rates, such as recent participation history, 22 drop-out and participation rates in date of entry, source, etc., on your connection with a provided sample? panelists? Are you able to supply your How are these computed? client with a per job analysis of such individual level data? Participation rate is usually between 30-45%. This rate depends greatly on a series of We maintain a full record of all individual parameters, like time of day the invitations are participants. Since we source only respond- distributed, the day of the week, the target ents from our own in-house fully profiled group selected, e-mail invitation text provided panel, we are able to retrieve a full log of all etc. actions performed by our system in regards to Drop-out rates are between 1-25%, with an a single respondent, as well as a full record of average around 10-15%. The dropout rates all the respondent actions in relation to our depend greatly on length of the survey, as survey invitations. well as the content of the survey. We continuously monitor drop-out rates, accept-rates etc. on each survey. If perfor- mance is below average, we introduce sug-EPINION ONLINE PANEL FAQ 10
  11. 11. Do you use data quality analysis and rewarded as ‘payment’ for each completed 24 validation techniques to identify inat- survey. tentive and fraudulent respondents? If We do not require that participants fit a spe- yes, what techniques are used and at cific profile in order to receive the reward – in what point in the process are they this case participate in the prize-draw. If you applied? spend time answering our surveys, you partici- pate in the prize-draws no matter if the partici- We use a mix of different techniques to pant fits the target group or not. identify fraudulent users. We combine ‘hard’ We have completely removed the incentive to data from the participant’s computer/browser provide false answers in order to gain an with behavioral data analysis in order to iden- advantage by earning money, points etc. tify fraudulent respondents. Hard data consist of: Session cookies, browser cookies, IP-address validation, e-mail validation (double opt-in required, no e-mail equals no survey) + a wide range of different browser plug-ins. We utilize a wide range of information about the browser used as well as the network connection used in order to verify the participant. All these hard data are com- bined with our ability to monitor deviations in behavior when the participant completes the surveys. We analyze features like: incoherent response-patterns, average time spend devia- tions, probability of completing open-text answers etc. Together, all these information’s create a credibility-index, which allows us to monitor suspicious behavior closely. But our most effective weapon against fraudu- lent behavior is to remove the monetary incen- tive to perform fraud. Epinion almost always offers price-draws as a reward for participa- tion. No cash rewards, rebates or points areEPINION ONLINE PANEL FAQ 11
  12. 12. Do you measure respondent satisfaction? survey or questions that are difficult to understand) that requires immedi-25 ate action is automatically forwarded to the project manager by e-mail. Yes, after every survey is completed, all respondents are motivated to give The feedback collected from the respondents is then combined with hard feedback. A typical feedback page looks like this (but will be modified to data about completion rates, average interview time etc. into an aggre- suit different brands and languages): gated easy to read feedback report format. We also use the invitation e-mails to motivate the respondents to provide us with feedback, also those respondents who fail to complete the survey. Epinion measures respondent satisfaction on five comparable parameters as well as collecting open verbatim feedback from the participants of each survey. This feedback goes directly to the client project manager. The project manager can always access the respondent feedback report which is updated in real time, as the respondent feedback comes in. Critical feedback from the participants (like technical problems with theEPINION ONLINE PANEL FAQ 12
  13. 13. What information do you provide to debrief your client after 26 the project has finished? Along with the respondent feedback report shown above, the client can at all times access the current completion report showing exactly the progression of the survey in its lifecycle. Dropout rates, participations rates, invitations rounds executed, success rates etc. are all supplied in an easy to read interface. Epinion also provides the project manager with a ‘General Statistics‘-report benchmarking the survey performance of the project manager with Epinion averages internationally. The same report evaluates not only current performance, but also past performance of the project manager’s surveys, and provides recommen- dations on how to improve future survey projects. See slide from benchmark report below format.EPINION ONLINE PANEL FAQ 13
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.