Prof. Dr. Kristof De Wulf Dr. Mike Friedman Bert Borggreve Pavlov revisited Effects of panel method & conditioning on panel member quality & product ratings Prof. Dr. Kristof De Wulf Dr. Mike Friedman Bert Borggreve

Contents Background Research propositions Method Results Conclusions Future directions

Background

Research propositions

Method

Results

Conclusions

Future directions

Dedicated panels: the wave of the future? “ People are now demanding more two-way relationships with websites, brands, and with market researchers … The traditional online access panel is largely a one-way relationship, with its paradigm very much fixed in the old world” (Comley & Andersson, 2007) “ The online [dedicated] audience panel has enabled ITV to conduct a much wider range of projects across their different business areas… the sheer speed and accessibility of the panel has meant that the findings now form part of the daily discussions across ITV” (Connor & Scholes, 2007) “ The JD [dedicated] Panel serves as an efficient and effective means to gather customer input, while at the same time strengthening relationships with our key customers” (Dorsey, 2007)

“ People are now demanding more two-way relationships with websites, brands, and with market researchers … The traditional online access panel is largely a one-way relationship, with its paradigm very much fixed in the old world” (Comley & Andersson, 2007)

“ The online [dedicated] audience panel has enabled ITV to conduct a much wider range of projects across their different business areas… the sheer speed and accessibility of the panel has meant that the findings now form part of the daily discussions across ITV” (Connor & Scholes, 2007)

“ The JD [dedicated] Panel serves as an efficient and effective means to gather customer input, while at the same time strengthening relationships with our key customers” (Dorsey, 2007)

Each communication you have with a customer is also a door you open for them within your company. If you do not treat them with respect… they may not only decline to complete your survey, the may also take their business elsewhere. Dorsey 2007 Respect

Confidentiality

Fast and flexible

Contents Background Research propositions Method Results Conclusions Future directions

Background

Research propositions

Method

Results

Conclusions

Future directions

Research propositions Effects of panel group membership (XL Online Panels vs Heinz panel) Effects of conditioning (within Heinz panel) Panel member quality Response rate Open-ended questions answer rate Opend-ended number of words Survey satisfaction Product ratings

Effects of panel group membership (XL Online Panels vs Heinz panel)

Effects of conditioning

(within Heinz panel)

Panel member quality

Response rate

Open-ended questions answer rate

Opend-ended number of words

Survey satisfaction

Product ratings

Contents Background Research propositions Method Results Conclusions Future directions

Background

Research propositions

Method

Results

Conclusions

Future directions

Method Migration of 2.000 panel members from the online access panel of XL Online Panels to a dedicated customer Heinz panel Multi-branded Exclusively available for Heinz Similar incentive scheme as XL Online Panels Longitudinal research design 5 different measurements Concept screening objective Data collection from both panels at the same time

Migration of 2.000 panel members from the online access panel of XL Online Panels to a dedicated customer Heinz panel

Multi-branded

Exclusively available for Heinz

Similar incentive scheme as XL Online Panels

Longitudinal research design

5 different measurements

Concept screening objective

Data collection from both panels at the same time

Method Wave 1: Soup Wave 2: Fruit drink Wave 3: Cereal Wave 4: Soup Wave 5: Cereal Wave 1 Wave 2 Wave 3 Wave 4 Wave 5 Heinz total sample n = 697 n = 770 n = 811 n = 1.082 n = 1.210 Heinz fresh sample n = 697 n = 223 n = 221 n = 356 n = 262 Heinz repeat sample n = 697 (1x) n = 553 (2x) n = 384 (3x) n = 288 (4x) n = 213 (5x) XLOP fresh sample n = 301 n = 213 n = 816 n = 218 n = 253

Method Significance Effect greater than zero? Is it replicable? Sensitive to sample size Testing for effects: significance versus effect size Effect size Magnitude of effect? Is it meaningful? Not sensitive to sample size

Significance

Effect greater than zero?

Is it replicable?

Sensitive to sample size

Effect size

Magnitude of effect?

Is it meaningful?

Not sensitive to sample size

Contents Background Research propositions Method Results Conclusions Future directions

Background

Research propositions

Method

Results

Conclusions

Future directions

Testing panel method effects Effects of panel method (XL Online Panels vs Heinz panel) Effects of conditioning (within Heinz panel) Panel member quality Response rate Open-ended questions answer rate Opend-ended number of words Survey satisfaction Product ratings

Effects of panel method

(XL Online Panels vs Heinz panel)

Effects of conditioning

(within Heinz panel)

Panel member quality

Response rate

Open-ended questions answer rate

Opend-ended number of words

Survey satisfaction

Product ratings

Testing panel method effects Wave 1: Soup Wave 2: Fruit drink Wave 3: Cereal Wave 4: Soup Wave 5: Cereal Wave 1 Wave 2 Wave 3 Wave 4 Wave 5 Heinz total sample n = 697 n = 770 n = 811 n = 1.082 n = 1.210 Heinz fresh sample n = 697 n = 223 n = 221 n = 356 n = 262 Heinz repeat sample n = 697 (1x) n = 553 (2x) n = 384 (3x) n = 288 (4x) n = 213 (5x) XLOP fresh sample n = 301 n = 213 n = 816 n = 218 n = 253

Testing panel method effects – Product ratings Product Rating (Unpriced Buying Intention) Wave 1: Soup Wave 2: Fruit drink Wave 3: Cereal Wave 4: Soup Wave 5: Cereal Sig. difference (95%) Effect size R 2 = .01 R 2 = .01 R 2 = .04 R 2 = .03

Testing panel method effects – Panel member quality Overall satisfaction questionnaire Wave 1: Soup Wave 2: Fruit drink Wave 3: Cereal Wave 4: Soup Wave 5: Cereal Sig. difference (95%) Fraction that filled in open ended question (likes) R 2 = .01 R 2 = .01 R 2 = .04 R 2 = .03 R 2 = .01 R 2 = .01 R 2 = .02 R 2 = .06 R 2 = .03

Testing panel method effects – Panel member quality Response rates between groups

Testing conditioning effects Effects of panel group membership (XL Online Panels vs Heinz panel) Effects of conditioning (within Heinz panel) Panel member quality Response rate Open-ended questions answer rate Opend-ended number of words Survey satisfaction Product ratings

Effects of panel group membership (XL Online Panels vs Heinz panel)

Effects of conditioning

(within Heinz panel)

Panel member quality

Response rate

Open-ended questions answer rate

Opend-ended number of words

Survey satisfaction

Product ratings

Testing conditioning effects Wave 1: Soup Wave 2: Fruit drink Wave 3: Cereal Wave 4: Soup Wave 5: Cereal Wave 1 Wave 2 Wave 3 Wave 4 Wave 5 Heinz total sample n = 697 n = 770 n = 811 n = 1.082 n = 1.210 Heinz fresh sample n = 697 n = 223 n = 221 n = 356 n = 262 Heinz repeat sample n = 697 (1x) n = 553 (2x) n = 384 (3x) n = 288 (4x) n = 213 (5x) XLOP fresh sample n = 301 n = 213 n = 816 n = 218 n = 253

Testing conditioning effects – Product ratings Wave 1: Soup Wave 2: Fruit drink Wave 3: Cereal Wave 4: Soup Wave 5: Cereal Sig. difference (95%) Product Rating (Unpriced Buying Intention) R 2 = .01 R 2 = .01 R 2 = .03 Effect size

Testing conditioning effects – Product ratings – A zoom on all key scores of wave 5 Product Rating Wave 5: Cereal Sig. difference (95%) R 2 = .01 R 2 = .02 R 2 = .01 R 2 = .02 Effect size

Testing conditioning effects – Panel member quality Overall satisfaction questionnaire Wave 1: Soup Wave 2: Fruit drink Wave 3: Cereal Wave 4: Soup Wave 5: Cereal Sig. difference (95%) Fraction that filled in open ended question (likes) R 2 = .01 R 2 = .01 R 2 = .01 R 2 = .06

Contents Background Research propositions Method Results Conclusions Future directions

Background

Research propositions

Method

Results

Conclusions

Future directions

Effects of panel method

Effects of panel method Panel member quality Much higher response rate in Heinz panel Clearly higher satisfaction scores in Heinz panel More response to open ended questions in Heinz panel Product ratings Frequent and consistent differences between panel types Heinz panel members are more favorable than XLOP members This effect is small; there is variation between waves

Panel member quality

Much higher response rate in Heinz panel

Clearly higher satisfaction scores in Heinz panel

More response to open ended questions in Heinz panel

Product ratings

Frequent and consistent differences between panel types

Heinz panel members are more favorable than XLOP members

This effect is small; there is variation between waves

Effects of conditioning

Effects of conditioning Panel member quality Very similar results in terms of overall questionnaire satisfaction. Frequent participation does not lead to diverging satisfaction scores Decreasing number of open ended responses over time Product ratings Frequent participation leads to diverging product ratings. New panel members are more favorable than repeat members There is variation between waves

Panel member quality

Very similar results in terms of overall questionnaire satisfaction. Frequent participation does not lead to diverging satisfaction scores

Decreasing number of open ended responses over time

Product ratings

Frequent participation leads to diverging product ratings.

New panel members are more favorable than repeat members

There is variation between waves

Contents Background Research propositions Method Results Conclusions Future directions

Background

Research propositions

Method

Results

Conclusions

Future directions

Future directions