Mixing the Right Sample Ingredients


Published on

At the Advertising Research Foundation’s (ARF) 2011 annual re:think convention, a key issues forum presentation was held entitled Mixing the Right Sample Ingredients. The presentation was given by Jackie Lorch, VP of Global Knowledge Management for Survey Sampling International. The presentation discussed which factors to blend and emphasized the importance of implementing multi-source testing.

Published in: Business, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Mixing the Right Sample Ingredients

  1. 1. Mixing the Right SampleIngredientsA New Source Recipe Jackie Lorch VP, Global Knowledge Management Survey Sampling International
  2. 2. What is the ProblemWe Are Trying to Solve?1. A systematic problem with online access panels
  3. 3. What is the ProblemWe Are Trying to Solve?2. A scarce resource
  4. 4. Many Reasons People Are Not in Panels Why have you not joined an Why do you no longer belong to an online research panel?1 online research panel?2I don’t want any more emails 7% Too many surveys 4%I think they are scams 6% Not enough surveys 27%It’s a waste of time 3% I never qualified for a survey 44%Not enough time to take surveys 11% Poor respondent experience 4%Never heard of an online research panel 20% Not enough time to take surveys 17%Don’t want to commit to a panel 22% Didn’t want to commit to a panel anymore 8%Never had any opportunities to join a panel 37% Other 8%Other 6% 1. Asked of people online who have never joined a research panel 2. Asked of people online, on a panel previously, not currently Multiple answers allowed
  5. 5. What is the ProblemWe Are Trying to Solve?
  6. 6. What is the ProblemWe Are Trying to Solve?
  7. 7. A Better Breed of Online Sample
  8. 8. Real Differences in BehaviorOwn an MP3 player
  9. 9. Why Not Use Socio-Demographics?• Demographics may not be the most helpful or relevant stabilizers• Demographic quotas work if the stratification chosen is relevant to questionnaire topic• A product may be liked across all demographics
  10. 10. Which are the right quotas? Sample 1 Sample 2 Balanced on: Balanced on: Age Ethnicity Gender Region
  11. 11. From Sources to People• Using thousands of sources• Metric must be at “person” level
  12. 12. Beyond Socio-DemographicsWith effective stabilization variables, no need tomaintain same blend of sources project-to-project
  13. 13. Factors for the BlendBroad set of factors defining groups of people,spanning multiple disciplinesPersonality traits Need for cognition (Cacioppo et al)Music preferences (Rentfrow) Neurographics measuresCognitive ability Propensity factors e.g., to participate,(Kahneman and Frederick) share, risk averse, attitude to privacyGeographic / personality alignment Chronotypes, “lark” or “owl”(Gosling and Rentfrow) (Roenneberg)Social Values. Schwarz has international Disruption/orienting reflex measure,benchmarks habituation disruption
  14. 14. Factors for the Blend• Questions created around 162 variables• Tested factor performance against questions on technology, hobbies, interests, brand preference, loyalty, ad awareness• Variables tested iteratively on power to “move the needle” on dependent variables• 14 variables identified as explaining more variance than socio-demographics alone; from these created 8 clusters to control
  15. 15. Multi-Source Testing• Broad set of factors defining groups of people, spanning multiple disciplines• Factors included such topics as attitudes to trust, gender roles, online behavior, personality traits, ideas and beliefs• Results varied by source
  16. 16. Results Closer to “Truth”On measures where benchmarks are available,blend comes closer to it Social Media iPod/MP3 Smartphone (Facebook or Ownership Ownership MySpace)SSI SurveySpot panel 37 16 60Benchmark 43 c.20-22 69SSI Blend 43 20 68Benchmark sources: Pew Research; comScore; Harris Interactive
  17. 17. Implementing the Blend
  18. 18. Taming the Stream
  19. 19. Lock and Control the Stream
  20. 20. Simulation Without ControlMP3 ownership – uncontrolled
  21. 21. Simulation Holding Clusters ConstantMP3 ownership – controlled
  22. 22. Practical Steps When Blending• Know your sources• Use a consistent blend for the entire project• Use benchmarks and calibrate• Understand blending techniques being used
  23. 23. In Summary• Multi-sourcing is the future, offering superior sample• We can keep multi-sourced samples consistent by pre-profiling participants• Standard socio-demographics aren’t enough• A concise list of variables can be used for a broad range of research subjects• Participant experience is key