Panel duplication and recruitment: ESOMAR panel conference Orlando 2007

DUPLICATION & MULTI-SOURCE ONLINE PANEL RECRUITMENT Real quality differences or idle rumours? Kristof De Wulf Sam Berteloot

Content

Some major concerns ...

Research model & hypotheses

Effects of multiple membership

Effects of recruitment

Discussion & implications

Some major concerns ... “ 1% of panel members accounts for 34% of questionnaires completed” (CASRO 2006) “ It's been difficult to duplicate the results from one panel "so that people in the business can have confidence that the results they're getting are valid and predictable” ( ClickZ Network 2007) “ The market research industry faces a crisis of ‘over-fishing’ and poor respondent quality” that demands solutions to “improve the quality of online samples and associated responses” (comScore 2004) “ In the past year, marketers and researchers alike have focused unprecedented attention on the accuracy of research – specifically, data from online panels, which can produce wildly varying results for the same questions within and across suppliers ” (American Marketing Association 2007) “ We need to stop this blind trust at the client side. And the industry needs to help clients understand what quality means, and why data quality needs to be a key differentiator ” (Research World April 2007)

Content

Telephone recruitment

Random digit dialing

Specialised telemarketing field agency

209 new panel members recruited

Street, event, mall intercept recruitment

Face-to-face recruitment

Several events & locations

Weekdays as well as during the day

E-mail recruitment

Permesso.be opt-in e-mail list

Subscribers receive targeted offers and discounts for various products and services

Online advertising recruitment

Online recruitment campaign via TradeDoubler

Targeted campaign

Pop-up recruitment after on-site survey

Network of 155 websites

Pop-up asking website visitors for their opinion on the website with opt-in at the end

Response status

Click rate

Participation rate

Drop-out rate

Fill-out time

Response quality

Item response consistency

Response styles

Data structure equivalency

Open answer detail

Self-reported quality of responses

Panel member feedback

Motivation to subscribe

Survey satisfaction

Panel member satisfaction

Multiple online panel membership

Yes

No

6 data sources to test effects of type of recruitment vehicle (Belgian panel only)

Data source 1: subscription database

Small survey containing questions about their socio-demographic and internet profile.

Data source 2: business-to-business study (wave 1)

Questionnaire about their professional profile 48 hours after subscription.

Data source 3: indirect quality test (wave 2)

Fake packaging test was developed: respondents were asked to give their opinion about the interface of two car radios. They also provided information about their socio-demographic profile and motivation to subscribe.

Data source 4: category involvement and adoption study (wave 3)

Questions about media profile, consumption of FMCG goods, technology adoption, brand sensitivity, social class, values, …

Data source 5: brand tracking study (wave 4)

Study assessing brand awareness, advertising recall, … for a brand of liquor.

Data source 6: panel member satisfaction study (wave 5)

Panel members were asked about their satisfaction with the XL Online Panels panel: the recruitment procedure, invitations, the way of asking questions, the incentive, and so on. Also data on quality of completion, loyalty, motivation to participate to surveys and socio-demographic information were collected.

1 data source to test effects of multiple online panel membership (Belgian panel only)

Data source : panel member satisfaction study

Panel members were asked about their satisfaction with the XL Online Panels panel: the recruitment procedure, invitations, the way of asking questions, the incentive, and so on. Also data on quality of completion, loyalty, motivation to participate to surveys and socio-demographic information were collected.

Content

Who are people being subscribed to multiple panels? People subscribed to multiple panels do not differ from people subscribed to one panel only in terms of education, professional situation, and monthly net family income. Significant difference between groups (95% confidence interval) Professional situation Monthly net family income Highest education

Who are people being subscribed to multiple panels? People subscribed to multiple panels are more internet savvy : they have been using the medium longer and use it more often and intensively. Significant difference between groups (95% confidence interval) Daily usage of internet (hours) Number of days using internet Usage history internet

E-mail recruitment

Targeted campaign

Response status

Click rate

Participation rate

Drop-out rate

Fill-out time

Response quality

Response styles

Open answer detail

Survey satisfaction

Yes

No

Effects of multiple membership No difference in response status : while the overall click rate is higher among subscribers to multiple panels, drop-out rates are higher, resulting in a comparable participation rate. Significant difference between groups (95% confidence interval)

E-mail recruitment

Targeted campaign

Response status

Click rate

Participation rate

Drop-out rate

Fill-out time

Response quality

Response styles

Open answer detail

Survey satisfaction

Yes

No

Effects of multiple membership Response quality is slightly better among subscribers to multiple panels : they indicate to fill out surveys more truthfully, to be better concentrated when answering questions, and to try harder while filling out surveys. Significant difference between groups (95% confidence interval) Self-reported quality of responses (5-pt scale)

Effects of multiple membership Response quality is better among subscribers to multiple panels : item response consistency between surveys is higher. Item response consistency Significant difference between groups (95% confidence interval)

E-mail recruitment

Targeted campaign

Response status

Click rate

Participation rate

Drop-out rate

Fill-out time

Response quality

Response styles

Open answer detail

Survey satisfaction

Yes

No

Effects of multiple membership Panel member feedback is more positive among subscribers to multiple panels : overall satisfaction, intention to participate, positive word-of-mouth and recommendation to others are higher. Significant difference between groups (95% confidence interval) Panel member satisfaction & intentions

Effects of multiple membership Intrinsic motivations to learn new things are higher among subscribers to multiple panels . Extrinsic motivations (receiving rewards) are less important and not different between both groups. Significant difference between groups (95% confidence interval) Reasons to subscribe

Content

E-mail recruitment

Targeted campaign

Response status

Click rate

Participation rate

Drop-out rate

Fill-out time

Response quality

Response styles

Open answer detail

Survey satisfaction

Yes

No

Effects of recruitment Participation rate is highest among people recruited via e-mail and lowest among people recruited via intercept . There are no significant differences in fill-out time between modes of recruitment. Significant difference between groups (95% confidence interval) Average across 5 data collection waves

Effects of recruitment Participation rate is highest among people recruited via e-mail and lowest among people recruited via intercept . E-mail Online ad Telephone Pop-up survey Intercept Participation rate per data collection wave Significant difference between groups (95% confidence interval)

E-mail recruitment

Targeted campaign

Response status

Click rate

Participation rate

Drop-out rate

Fill-out time

Response quality

Response styles

Open answer detail

Survey satisfaction

Yes

No

Effects of recruitment Response styles are relatively more present among people recruited via online ads . Non-response to open ended questions is larger among people being recruited via e-mail . Significant difference between groups (95% confidence interval) Response style & open answer detail based on wave 3

Effects of recruitment Item response consistency is comparable between modes of recruitment. Significant difference between groups (95% confidence interval) Item response consistency based on wave 3

Effects of recruitment Self-reported response quality is comparable between modes of recruitment. Significant difference between groups (95% confidence interval) Self-reported quality of responses (5-pt scale) based on wave 5

E-mail recruitment

Targeted campaign

Response status

Click rate

Participation rate

Drop-out rate

Fill-out time

Response quality

Response styles

Open answer detail

Survey satisfaction

Yes

No

Effects of recruitment People recruited via online ads are most driven by intrinsic motivations : providing their opinion and liking to fill out surveys. People recruited via offline methods are most driven by ‘ helping others ’ (researchers and charity). Significant difference between groups (95% confidence interval) Reasons to subscribe based on wave 5 (5-pt scale)

Effects of recruitment Panel member satisfaction among people recruited via online ads is significantly higher . Significant difference between groups (95% confidence interval) Project satisfaction based on waves 4 and 5 (5-pt scale) Panel member satisfaction based on wave 5 (5-pt scale)

Content

We should not be too concerned about the fact that people join multiple panels as their quality is not inferior to people who have only joined one panel

While they are more heavy users of the internet, they are positive towards the panel they have joined, reveal better click rates, show slightly better response quality, and attach a greater importance to extrinsic as well as intrinsic motivations

Our results also indicate that we should abandon stereotyping people being subscribed to multiple panels

As opposed to what many think, they are not lower social class, weakly educated housewives filling out surveys as a complementary source of funding

Multi-method recruitment for building online panels is not a necessary condition for building a good quality panel

Most recruitment methods have shown to recruit quite similar profiles

Online recruitment methods have proven to generate comparable results in terms of response status and slightly better results in terms of response quality than offline recruitment methods

Given that online recruitment is much more cost-effective, online recruitment method not only makes sense from a return-on-investment perspective, but also from a quality perspective

Limitations & directions for future research