Panel duplication and recruitment: ESOMAR panel conference Orlando 2007

1,136 views

Published on

As the market research industry is gradually outsourcing more and more of its online data collection to specialized online panel providers and with research budgets still shifting towards online data collection, the topic of managing the quality of online access panels has become extremely important. With the statement “Garbage in, garbage out” more than ever relevant, we are witnessing a necessary evolution towards defining relevant norms for the industry.

As in any debate there are supporters and opponents who each make a case for their own right, while the simple truth is we do not know (yet). One of the key issues is whether duplication, i.e. respondents belonging to multiple panels, really makes a difference in terms of the quality of the research data. Another issue is whether online versus offline recruitment of access panel members leads to quality differences. In order to shed a better light on both of these issues, we have set up two different studies in cooperation with the XL Online Panels online research panel.

Published in: Business, Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,136
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
27
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Panel duplication and recruitment: ESOMAR panel conference Orlando 2007

  1. 1. DUPLICATION & MULTI-SOURCE ONLINE PANEL RECRUITMENT Real quality differences or idle rumours? Kristof De Wulf Sam Berteloot
  2. 2. Content <ul><li>Some major concerns ... </li></ul><ul><li>Research model & hypotheses </li></ul><ul><li>Effects of multiple membership </li></ul><ul><li>Effects of recruitment </li></ul><ul><li>Discussion & implications </li></ul>
  3. 3. Some major concerns ...
  4. 4. Some major concerns ...
  5. 5. Some major concerns ... “ 1% of panel members accounts for 34% of questionnaires completed” (CASRO 2006) “ It's been difficult to duplicate the results from one panel &quot;so that people in the business can have confidence that the results they're getting are valid and predictable” ( ClickZ Network 2007) “ The market research industry faces a crisis of ‘over-fishing’ and poor respondent quality” that demands solutions to “improve the quality of online samples and associated responses” (comScore 2004) “ In the past year, marketers and researchers alike have focused unprecedented attention on the accuracy of research – specifically, data from online panels, which can produce wildly varying results for the same questions within and across suppliers ” (American Marketing Association 2007) “ We need to stop this blind trust at the client side. And the industry needs to help clients understand what quality means, and why data quality needs to be a key differentiator ” (Research World April 2007)
  6. 6. Content <ul><li>Some major concerns ... </li></ul><ul><li>Research model & hypotheses </li></ul><ul><li>Effects of multiple membership </li></ul><ul><li>Effects of recruitment </li></ul><ul><li>Discussion & implications </li></ul>
  7. 7. Research model & hypotheses <ul><li>Telephone recruitment </li></ul><ul><li>Random digit dialing </li></ul><ul><li>Specialised telemarketing field agency </li></ul><ul><li>209 new panel members recruited </li></ul><ul><li>Street, event, mall intercept recruitment </li></ul><ul><li>Face-to-face recruitment </li></ul><ul><li>Several events & locations </li></ul><ul><li>Weekdays as well as during the day </li></ul><ul><li>199 new panel members recruited </li></ul><ul><li>E-mail recruitment </li></ul><ul><li>Permesso.be opt-in e-mail list </li></ul><ul><li>Subscribers receive targeted offers and discounts for various products and services </li></ul><ul><li>195 new panel members recruited </li></ul><ul><li>Online advertising recruitment </li></ul><ul><li>Online recruitment campaign via TradeDoubler </li></ul><ul><li>Targeted campaign </li></ul><ul><li>253 new panel members recruited </li></ul><ul><li>Pop-up recruitment after on-site survey </li></ul><ul><li>Network of 155 websites </li></ul><ul><li>Pop-up asking website visitors for their opinion on the website with opt-in at the end </li></ul><ul><li>386 new panel members recruited </li></ul><ul><li>Response status </li></ul><ul><li>Click rate </li></ul><ul><li>Participation rate </li></ul><ul><li>Drop-out rate </li></ul><ul><li>Fill-out time </li></ul><ul><li>Response quality </li></ul><ul><li>Item response consistency </li></ul><ul><li>Response styles </li></ul><ul><li>Data structure equivalency </li></ul><ul><li>Open answer detail </li></ul><ul><li>Self-reported quality of responses </li></ul><ul><li>Panel member feedback </li></ul><ul><li>Motivation to subscribe </li></ul><ul><li>Survey satisfaction </li></ul><ul><li>Panel member satisfaction </li></ul><ul><li>Multiple online panel membership </li></ul><ul><li>Yes </li></ul><ul><li>No </li></ul>
  8. 8. Research model & hypotheses <ul><li>6 data sources to test effects of type of recruitment vehicle (Belgian panel only) </li></ul><ul><ul><li>Data source 1: subscription database </li></ul></ul><ul><ul><li>Small survey containing questions about their socio-demographic and internet profile. </li></ul></ul><ul><ul><li>Data source 2: business-to-business study (wave 1) </li></ul></ul><ul><ul><li>Questionnaire about their professional profile 48 hours after subscription. </li></ul></ul><ul><ul><li>Data source 3: indirect quality test (wave 2) </li></ul></ul><ul><ul><li>Fake packaging test was developed: respondents were asked to give their opinion about the interface of two car radios. They also provided information about their socio-demographic profile and motivation to subscribe. </li></ul></ul><ul><ul><li>Data source 4: category involvement and adoption study (wave 3) </li></ul></ul><ul><ul><li>Questions about media profile, consumption of FMCG goods, technology adoption, brand sensitivity, social class, values, … </li></ul></ul><ul><ul><li>Data source 5: brand tracking study (wave 4) </li></ul></ul><ul><ul><li>Study assessing brand awareness, advertising recall, … for a brand of liquor. </li></ul></ul><ul><ul><li>Data source 6: panel member satisfaction study (wave 5) </li></ul></ul><ul><ul><li>Panel members were asked about their satisfaction with the XL Online Panels panel: the recruitment procedure, invitations, the way of asking questions, the incentive, and so on. Also data on quality of completion, loyalty, motivation to participate to surveys and socio-demographic information were collected. </li></ul></ul><ul><li>1 data source to test effects of multiple online panel membership (Belgian panel only) </li></ul><ul><ul><li>Data source : panel member satisfaction study </li></ul></ul><ul><ul><li>Panel members were asked about their satisfaction with the XL Online Panels panel: the recruitment procedure, invitations, the way of asking questions, the incentive, and so on. Also data on quality of completion, loyalty, motivation to participate to surveys and socio-demographic information were collected. </li></ul></ul>
  9. 9. Content <ul><li>Some major concerns ... </li></ul><ul><li>Research model & hypotheses </li></ul><ul><li>Effects of multiple membership </li></ul><ul><li>Effects of recruitment </li></ul><ul><li>Discussion & implications </li></ul>
  10. 10. Who are people being subscribed to multiple panels? People subscribed to multiple panels do not differ from people subscribed to one panel only in terms of education, professional situation, and monthly net family income. Significant difference between groups (95% confidence interval) Professional situation Monthly net family income Highest education
  11. 11. Who are people being subscribed to multiple panels? People subscribed to multiple panels are more internet savvy : they have been using the medium longer and use it more often and intensively. Significant difference between groups (95% confidence interval) Daily usage of internet (hours) Number of days using internet Usage history internet
  12. 12. Research model & hypotheses <ul><li>Telephone recruitment </li></ul><ul><li>Random digit dialing </li></ul><ul><li>Specialised telemarketing field agency </li></ul><ul><li>209 new panel members recruited </li></ul><ul><li>Street, event, mall intercept recruitment </li></ul><ul><li>Face-to-face recruitment </li></ul><ul><li>Several events & locations </li></ul><ul><li>Weekdays as well as during the day </li></ul><ul><li>199 new panel members recruited </li></ul><ul><li>E-mail recruitment </li></ul><ul><li>Permesso.be opt-in e-mail list </li></ul><ul><li>Subscribers receive targeted offers and discounts for various products and services </li></ul><ul><li>195 new panel members recruited </li></ul><ul><li>Online advertising recruitment </li></ul><ul><li>Online recruitment campaign via TradeDoubler </li></ul><ul><li>Targeted campaign </li></ul><ul><li>253 new panel members recruited </li></ul><ul><li>Pop-up recruitment after on-site survey </li></ul><ul><li>Network of 155 websites </li></ul><ul><li>Pop-up asking website visitors for their opinion on the website with opt-in at the end </li></ul><ul><li>386 new panel members recruited </li></ul><ul><li>Response status </li></ul><ul><li>Click rate </li></ul><ul><li>Participation rate </li></ul><ul><li>Drop-out rate </li></ul><ul><li>Fill-out time </li></ul><ul><li>Response quality </li></ul><ul><li>Item response consistency </li></ul><ul><li>Response styles </li></ul><ul><li>Data structure equivalency </li></ul><ul><li>Open answer detail </li></ul><ul><li>Self-reported quality of responses </li></ul><ul><li>Panel member feedback </li></ul><ul><li>Motivation to subscribe </li></ul><ul><li>Survey satisfaction </li></ul><ul><li>Panel member satisfaction </li></ul><ul><li>Multiple online panel membership </li></ul><ul><li>Yes </li></ul><ul><li>No </li></ul>
  13. 13. Effects of multiple membership No difference in response status : while the overall click rate is higher among subscribers to multiple panels, drop-out rates are higher, resulting in a comparable participation rate. Significant difference between groups (95% confidence interval)
  14. 14. Research model & hypotheses <ul><li>Telephone recruitment </li></ul><ul><li>Random digit dialing </li></ul><ul><li>Specialised telemarketing field agency </li></ul><ul><li>209 new panel members recruited </li></ul><ul><li>Street, event, mall intercept recruitment </li></ul><ul><li>Face-to-face recruitment </li></ul><ul><li>Several events & locations </li></ul><ul><li>Weekdays as well as during the day </li></ul><ul><li>199 new panel members recruited </li></ul><ul><li>E-mail recruitment </li></ul><ul><li>Permesso.be opt-in e-mail list </li></ul><ul><li>Subscribers receive targeted offers and discounts for various products and services </li></ul><ul><li>195 new panel members recruited </li></ul><ul><li>Online advertising recruitment </li></ul><ul><li>Online recruitment campaign via TradeDoubler </li></ul><ul><li>Targeted campaign </li></ul><ul><li>253 new panel members recruited </li></ul><ul><li>Pop-up recruitment after on-site survey </li></ul><ul><li>Network of 155 websites </li></ul><ul><li>Pop-up asking website visitors for their opinion on the website with opt-in at the end </li></ul><ul><li>386 new panel members recruited </li></ul><ul><li>Response status </li></ul><ul><li>Click rate </li></ul><ul><li>Participation rate </li></ul><ul><li>Drop-out rate </li></ul><ul><li>Fill-out time </li></ul><ul><li>Response quality </li></ul><ul><li>Item response consistency </li></ul><ul><li>Response styles </li></ul><ul><li>Data structure equivalency </li></ul><ul><li>Open answer detail </li></ul><ul><li>Self-reported quality of responses </li></ul><ul><li>Panel member feedback </li></ul><ul><li>Motivation to subscribe </li></ul><ul><li>Survey satisfaction </li></ul><ul><li>Panel member satisfaction </li></ul><ul><li>Multiple online panel membership </li></ul><ul><li>Yes </li></ul><ul><li>No </li></ul>
  15. 15. Effects of multiple membership Response quality is slightly better among subscribers to multiple panels : they indicate to fill out surveys more truthfully, to be better concentrated when answering questions, and to try harder while filling out surveys. Significant difference between groups (95% confidence interval) Self-reported quality of responses (5-pt scale)
  16. 16. Effects of multiple membership Response quality is better among subscribers to multiple panels : item response consistency between surveys is higher. Item response consistency Significant difference between groups (95% confidence interval)
  17. 17. Research model & hypotheses <ul><li>Telephone recruitment </li></ul><ul><li>Random digit dialing </li></ul><ul><li>Specialised telemarketing field agency </li></ul><ul><li>209 new panel members recruited </li></ul><ul><li>Street, event, mall intercept recruitment </li></ul><ul><li>Face-to-face recruitment </li></ul><ul><li>Several events & locations </li></ul><ul><li>Weekdays as well as during the day </li></ul><ul><li>199 new panel members recruited </li></ul><ul><li>E-mail recruitment </li></ul><ul><li>Permesso.be opt-in e-mail list </li></ul><ul><li>Subscribers receive targeted offers and discounts for various products and services </li></ul><ul><li>195 new panel members recruited </li></ul><ul><li>Online advertising recruitment </li></ul><ul><li>Online recruitment campaign via TradeDoubler </li></ul><ul><li>Targeted campaign </li></ul><ul><li>253 new panel members recruited </li></ul><ul><li>Pop-up recruitment after on-site survey </li></ul><ul><li>Network of 155 websites </li></ul><ul><li>Pop-up asking website visitors for their opinion on the website with opt-in at the end </li></ul><ul><li>386 new panel members recruited </li></ul><ul><li>Response status </li></ul><ul><li>Click rate </li></ul><ul><li>Participation rate </li></ul><ul><li>Drop-out rate </li></ul><ul><li>Fill-out time </li></ul><ul><li>Response quality </li></ul><ul><li>Item response consistency </li></ul><ul><li>Response styles </li></ul><ul><li>Data structure equivalency </li></ul><ul><li>Open answer detail </li></ul><ul><li>Self-reported quality of responses </li></ul><ul><li>Panel member feedback </li></ul><ul><li>Motivation to subscribe </li></ul><ul><li>Survey satisfaction </li></ul><ul><li>Panel member satisfaction </li></ul><ul><li>Multiple online panel membership </li></ul><ul><li>Yes </li></ul><ul><li>No </li></ul>
  18. 18. Effects of multiple membership Panel member feedback is more positive among subscribers to multiple panels : overall satisfaction, intention to participate, positive word-of-mouth and recommendation to others are higher. Significant difference between groups (95% confidence interval) Panel member satisfaction & intentions
  19. 19. Effects of multiple membership Intrinsic motivations to learn new things are higher among subscribers to multiple panels . Extrinsic motivations (receiving rewards) are less important and not different between both groups. Significant difference between groups (95% confidence interval) Reasons to subscribe
  20. 20. Content <ul><li>Some major concerns ... </li></ul><ul><li>Research model & hypotheses </li></ul><ul><li>Effects of multiple membership </li></ul><ul><li>Effects of recruitment </li></ul><ul><li>Discussion & implications </li></ul>
  21. 21. Research model & hypotheses <ul><li>Telephone recruitment </li></ul><ul><li>Random digit dialing </li></ul><ul><li>Specialised telemarketing field agency </li></ul><ul><li>209 new panel members recruited </li></ul><ul><li>Street, event, mall intercept recruitment </li></ul><ul><li>Face-to-face recruitment </li></ul><ul><li>Several events & locations </li></ul><ul><li>Weekdays as well as during the day </li></ul><ul><li>199 new panel members recruited </li></ul><ul><li>E-mail recruitment </li></ul><ul><li>Permesso.be opt-in e-mail list </li></ul><ul><li>Subscribers receive targeted offers and discounts for various products and services </li></ul><ul><li>195 new panel members recruited </li></ul><ul><li>Online advertising recruitment </li></ul><ul><li>Online recruitment campaign via TradeDoubler </li></ul><ul><li>Targeted campaign </li></ul><ul><li>253 new panel members recruited </li></ul><ul><li>Pop-up recruitment after on-site survey </li></ul><ul><li>Network of 155 websites </li></ul><ul><li>Pop-up asking website visitors for their opinion on the website with opt-in at the end </li></ul><ul><li>386 new panel members recruited </li></ul><ul><li>Response status </li></ul><ul><li>Click rate </li></ul><ul><li>Participation rate </li></ul><ul><li>Drop-out rate </li></ul><ul><li>Fill-out time </li></ul><ul><li>Response quality </li></ul><ul><li>Item response consistency </li></ul><ul><li>Response styles </li></ul><ul><li>Data structure equivalency </li></ul><ul><li>Open answer detail </li></ul><ul><li>Self-reported quality of responses </li></ul><ul><li>Panel member feedback </li></ul><ul><li>Motivation to subscribe </li></ul><ul><li>Survey satisfaction </li></ul><ul><li>Panel member satisfaction </li></ul><ul><li>Multiple online panel membership </li></ul><ul><li>Yes </li></ul><ul><li>No </li></ul>
  22. 22. Effects of recruitment Participation rate is highest among people recruited via e-mail and lowest among people recruited via intercept . There are no significant differences in fill-out time between modes of recruitment. Significant difference between groups (95% confidence interval) Average across 5 data collection waves
  23. 23. Effects of recruitment Participation rate is highest among people recruited via e-mail and lowest among people recruited via intercept . E-mail Online ad Telephone Pop-up survey Intercept Participation rate per data collection wave Significant difference between groups (95% confidence interval)
  24. 24. Research model & hypotheses <ul><li>Telephone recruitment </li></ul><ul><li>Random digit dialing </li></ul><ul><li>Specialised telemarketing field agency </li></ul><ul><li>209 new panel members recruited </li></ul><ul><li>Street, event, mall intercept recruitment </li></ul><ul><li>Face-to-face recruitment </li></ul><ul><li>Several events & locations </li></ul><ul><li>Weekdays as well as during the day </li></ul><ul><li>199 new panel members recruited </li></ul><ul><li>E-mail recruitment </li></ul><ul><li>Permesso.be opt-in e-mail list </li></ul><ul><li>Subscribers receive targeted offers and discounts for various products and services </li></ul><ul><li>195 new panel members recruited </li></ul><ul><li>Online advertising recruitment </li></ul><ul><li>Online recruitment campaign via TradeDoubler </li></ul><ul><li>Targeted campaign </li></ul><ul><li>253 new panel members recruited </li></ul><ul><li>Pop-up recruitment after on-site survey </li></ul><ul><li>Network of 155 websites </li></ul><ul><li>Pop-up asking website visitors for their opinion on the website with opt-in at the end </li></ul><ul><li>386 new panel members recruited </li></ul><ul><li>Response status </li></ul><ul><li>Click rate </li></ul><ul><li>Participation rate </li></ul><ul><li>Drop-out rate </li></ul><ul><li>Fill-out time </li></ul><ul><li>Response quality </li></ul><ul><li>Item response consistency </li></ul><ul><li>Response styles </li></ul><ul><li>Data structure equivalency </li></ul><ul><li>Open answer detail </li></ul><ul><li>Self-reported quality of responses </li></ul><ul><li>Panel member feedback </li></ul><ul><li>Motivation to subscribe </li></ul><ul><li>Survey satisfaction </li></ul><ul><li>Panel member satisfaction </li></ul><ul><li>Multiple online panel membership </li></ul><ul><li>Yes </li></ul><ul><li>No </li></ul>
  25. 25. Effects of recruitment Response styles are relatively more present among people recruited via online ads . Non-response to open ended questions is larger among people being recruited via e-mail . Significant difference between groups (95% confidence interval) Response style & open answer detail based on wave 3
  26. 26. Effects of recruitment Item response consistency is comparable between modes of recruitment. Significant difference between groups (95% confidence interval) Item response consistency based on wave 3
  27. 27. Effects of recruitment Self-reported response quality is comparable between modes of recruitment. Significant difference between groups (95% confidence interval) Self-reported quality of responses (5-pt scale) based on wave 5
  28. 28. Research model & hypotheses <ul><li>Telephone recruitment </li></ul><ul><li>Random digit dialing </li></ul><ul><li>Specialised telemarketing field agency </li></ul><ul><li>209 new panel members recruited </li></ul><ul><li>Street, event, mall intercept recruitment </li></ul><ul><li>Face-to-face recruitment </li></ul><ul><li>Several events & locations </li></ul><ul><li>Weekdays as well as during the day </li></ul><ul><li>199 new panel members recruited </li></ul><ul><li>E-mail recruitment </li></ul><ul><li>Permesso.be opt-in e-mail list </li></ul><ul><li>Subscribers receive targeted offers and discounts for various products and services </li></ul><ul><li>195 new panel members recruited </li></ul><ul><li>Online advertising recruitment </li></ul><ul><li>Online recruitment campaign via TradeDoubler </li></ul><ul><li>Targeted campaign </li></ul><ul><li>253 new panel members recruited </li></ul><ul><li>Pop-up recruitment after on-site survey </li></ul><ul><li>Network of 155 websites </li></ul><ul><li>Pop-up asking website visitors for their opinion on the website with opt-in at the end </li></ul><ul><li>386 new panel members recruited </li></ul><ul><li>Response status </li></ul><ul><li>Click rate </li></ul><ul><li>Participation rate </li></ul><ul><li>Drop-out rate </li></ul><ul><li>Fill-out time </li></ul><ul><li>Response quality </li></ul><ul><li>Item response consistency </li></ul><ul><li>Response styles </li></ul><ul><li>Data structure equivalency </li></ul><ul><li>Open answer detail </li></ul><ul><li>Self-reported quality of responses </li></ul><ul><li>Panel member feedback </li></ul><ul><li>Motivation to subscribe </li></ul><ul><li>Survey satisfaction </li></ul><ul><li>Panel member satisfaction </li></ul><ul><li>Multiple online panel membership </li></ul><ul><li>Yes </li></ul><ul><li>No </li></ul>
  29. 29. Effects of recruitment People recruited via online ads are most driven by intrinsic motivations : providing their opinion and liking to fill out surveys. People recruited via offline methods are most driven by ‘ helping others ’ (researchers and charity). Significant difference between groups (95% confidence interval) Reasons to subscribe based on wave 5 (5-pt scale)
  30. 30. Effects of recruitment Panel member satisfaction among people recruited via online ads is significantly higher . Significant difference between groups (95% confidence interval) Project satisfaction based on waves 4 and 5 (5-pt scale) Panel member satisfaction based on wave 5 (5-pt scale)
  31. 31. Content <ul><li>Some major concerns ... </li></ul><ul><li>Research model & hypotheses </li></ul><ul><li>Effects of multiple membership </li></ul><ul><li>Effects of recruitment </li></ul><ul><li>Discussion & implications </li></ul>
  32. 32. Discussion & implications <ul><li>We should not be too concerned about the fact that people join multiple panels as their quality is not inferior to people who have only joined one panel </li></ul><ul><ul><li>While they are more heavy users of the internet, they are positive towards the panel they have joined, reveal better click rates, show slightly better response quality, and attach a greater importance to extrinsic as well as intrinsic motivations </li></ul></ul><ul><li>Our results also indicate that we should abandon stereotyping people being subscribed to multiple panels </li></ul><ul><ul><li>As opposed to what many think, they are not lower social class, weakly educated housewives filling out surveys as a complementary source of funding </li></ul></ul><ul><li>Multi-method recruitment for building online panels is not a necessary condition for building a good quality panel </li></ul><ul><ul><li>Most recruitment methods have shown to recruit quite similar profiles </li></ul></ul><ul><ul><li>Online recruitment methods have proven to generate comparable results in terms of response status and slightly better results in terms of response quality than offline recruitment methods </li></ul></ul><ul><ul><li>Given that online recruitment is much more cost-effective, online recruitment method not only makes sense from a return-on-investment perspective, but also from a quality perspective </li></ul></ul>
  33. 33. Limitations & directions for future research <ul><li>Can the same results be replicated across more markets ? </li></ul><ul><ul><li>All research hypotheses were tested based on panel members from Belgium only </li></ul></ul><ul><ul><li>What is the impact of the following factors on our results? </li></ul></ul><ul><ul><li>Economic country situation </li></ul></ul><ul><ul><li>Internet penetration within country </li></ul></ul><ul><ul><li>Maturity of internet research in the country </li></ul></ul><ul><ul><li>Cultural differences between countries </li></ul></ul><ul><ul><li>… </li></ul></ul><ul><li>Can the same results be replicated over longer timeframe ? </li></ul><ul><ul><li>5 consecutive research studies were included as observations </li></ul></ul><ul><ul><li>More long-term research is needed in order to check whether the effects remain the same. </li></ul></ul><ul><li>Can the same results be replicated in other panels / across other recruitment channels ? </li></ul><ul><ul><li>Multiple panel membership </li></ul></ul><ul><ul><li>What is the effect of the number of memberships? </li></ul></ul><ul><ul><li>What is the effect of the type of membership? </li></ul></ul><ul><ul><li>Type of recruitment source </li></ul></ul><ul><ul><li>What are differences between different e-mail lists? </li></ul></ul><ul><ul><li>How can online advertising be optimized (creative execution, publishers used, …)? </li></ul></ul><ul><ul><li>… </li></ul></ul>
  34. 34. THANK YOU More information? Kristof De Wulf: [email_address] Sam Berteloot: [email_address]

×