Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

QuestionPro Audience Webinar - How to Improve Data Quality For Your Research


Published on

With millions of people interacting online it only makes sense for market researchers to conduct research with online communities. The internet allows increased access to a wide range of diverse audiences, but gathering quality data from these participants can be challenging. This webinar will help you understand how to improve your data collection practices through better survey design methodology, tips to avoid response bias, variations in question styles and optimal data analysis. John Barrett, CEO of Priority Metrics Group, will share strategies regarding design methodologies that will keep your respondents engaged. With over 20 years experience working with Fortune 500 companies, John Barrett knows how to optimize results.

Published in: Data & Analytics
  • Be the first to comment

QuestionPro Audience Webinar - How to Improve Data Quality For Your Research

  1. 1. John Barrett Rudly Raphael President, QuestionPro AudienceCEO, Priority Metrics Group
  2. 2. QuestionPro Audience 20 Million Respondents B2B and B2C Solutions Mobile Research Panels 10 Specialty Panels • Global Scalability • Quality Recruitment • Ongoing quality control • Panel Respondents • Business Panels • Customer Panels • Real time results • Mobile community • Robust platform • Veterinarians • Building Contractors • Pet Owners • Registered Voters • And more!
  3. 3. #DataQualitytips
  4. 4. Agenda 1. Data quality 2. Data quality problems 3. Conversion 4. Interviewer training 5. Reducing non-response bias 6. Questionnaire design 7. Question Construction
  5. 5. Improving Data Quality
  6. 6. Importance of Data Quality
  7. 7. Quality Problems – Total Survey Error Groves (1989) identifies three categories of error: 1. Coverage – some members of the population under study do not have a known nonzero chance of being included in the sample. 2. Measurement effect – the instrument or items on the instrument are constructed in such a way to produce unreliable or invalid data. 3. Non-Response effect – nonrespondents in the survey sample differ from respondents in ways that are germane to the objectives of the survey.
  8. 8. Total Survey Error … for the rest of us Measurement Effect Non-Response Effect
  9. 9. Data Quality Problems 1.Sampling 2.Non-Response 3.Questionnaire Design 4.Execution/Methodology 5.Quality Control 6.Analysis/Reporting 7.Costs Poll: Which of the following has a direct effect on data quality, as it relates to your research? Please select the top 3. a. Poor sample b. Poor questionnaire design c. Poor execution/methodology d. Poor quality control procedures e. Poor survey incentives f. Poor open-end responses g. Limited analysis/reporting tools h. Limited research budget
  10. 10. Non-Response
  11. 11. Non-Response Documentation • Document non-response using AAPOR categories: ❏Interview ❏Eligible, non-interview ❏Unknown eligibility, non-interview ❏Not eligible • Identify correlates of non-response
  12. 12. Non-Response vs. Maximizing Response Rates Survey of healthcare leaders: web-based, self-report, four follow-ups • Overall 95% response rate, examined results in each of five waves Response Wave and Evaluative Attitudes Assessed Using a Multi-Item Scale: Mean and 95 Percent Confidence Intervals for the Alignment and Commitment Scale (ACS). Conclusion: Although high response rates are desirable because of their effect on precision and power, absolute thresholds representing “adequate” survey response rates may not be accurate. Source: “Response Rates, Nonresponse Bias, and Data Quality: Results from a National Survey of Senior Healthcare Leaders, “Meterko et al, Public Opinion Quarterly, January 27, 2015
  13. 13. Conversion: Non-Respondent Sees the Light! ✓ Different versions of survey introduction ✓ Study contact rules ✓ Incentives ✓ Interviewer style and training Research literature reports telephone conversion rates of 5% to 40%.
  14. 14. Interviewer Training – The Often Overlooked Nugget ✓ Interviewer hiring practice ✓ Thorough interview training ❏Survey background ❏Questionnaire familiarity ❏5-second rule: Introduction, survey purpose ❏Timing of contact ❏Personality of interviewer ✓ Multiple contacts ❏May increase response rates dramatically The right person makes a difference.
  15. 15. Reducing Non-Response Bias • Good survey design increases response rates • Good survey communication increases response rates Incentives can increase response rates Clever Foreshadow ✓ Do not cross “coercive threshold” ✓ Donation to charity (business) ✓ Prepaid (vs. promised) incentive (consumer) ✓ Differences decrease with follow-up ✓ May lead to lengthier open-end answers Source: “National Survey of Physicians to Determine the Effect of Unconditional Incentives on Response Rates of Physician Postal Surveys,”Abdulaziz et al, BMJ Open, Volume 5, Issue 2
  16. 16. Reducing Non-Response Bias Interviewers can gather information about non-respondents • Observable characteristics • Evaluations of engagement, honesty, ability • Visual and verbal clues • Observable characteristics • Evaluations of engagement, honesty, ability • Verbal clues
  17. 17. Questionnaire Design
  18. 18. Questionnaire Design Starts with Clear Objectives Start End
  19. 19. Good Design Practices ✓ Promise (or at least offer) anonymity or confidentiality ✓ Some form of relationship (known brand or existing customer) increases response rate ✓ Simplify the perceived task • Preserve white space • Design a logical flow • Group common items together – similarity and proximity • Don’t ask unnecessary questions • Minimize task difficulty ✓ Avoid use of jargon and notation ✓ Include relevant questions using question and page logic based on previous answers
  20. 20. Good Design Practices Improvements Explanation of task Question wording Confidentiality Positive Elements Clean, simple Logical flow Comment space
  21. 21. Visual Display ✓ No more than four visual elements ✓ Identify new information with distinct colors and/or sizes ✓ Bold type used in favor of italics, underline, or upper case ✓ Use highlights and graphics sparingly ✓ Differentiate important elements with large, bright or distinctive colors
  22. 22. Visual Display Positive Elements Logical flow Improvements Jargon throughout • Contract • Self-discover • Influencing style 37 visual elements Everything is bright Everything is new Everything is important
  23. 23. Question Construction ✓ Use conversational norms where possible ✓ Use common words with single primary definition • Few letters and syllables • Easy to pronounce • Avoid abbreviations ✓ Avoid asking certain types of questions: • Opinions held at prior times • Explain prior behavior or thoughts ✓ Balance use of closed- and open-end questions appropriately • Use open-end for numeric answers and categorical questions with unknown breadth of possible answers
  24. 24. Question Construction – Grice’s Maxims The maxim of quantity, where one tries to be as informative as one possibly can, and gives as much information as is needed, and no more. The maxim of quality, where one tries to be truthful, and does not give information that is false or that is not supported by evidence. The maxim of relation, where one tries to be relevant, and says things that are pertinent to the discussion. The maxim of manner, when one tries to be as clear, as brief, and as orderly as one can in what one says, and where one avoids obscurity and ambiguity.
  25. 25. Question Construction – Response Process (Tourangeau Model) Response based on recall or educated guess from cues or inferences, often requires choice of answers to report and how to report (agree or strongly agree) Comprehension how respondents understand the questions and infer the question’s point Judgment assessment of completeness or sufficiency of information/opinion, appropriateness and manner of using inferences, and how to transform retrieved information into appropriate answer Retrieval recalling information from long-term memory (behavioral questions) or a preformed opinion (attitude questions), but retrieval is rarely complete
  26. 26. Question Construction – Bad Examples 1. What is your frequency of utilization of retail travel agents? 2. Are you against the restrictive House Bill 935? 3. Have you visited ___ in the past and did you enjoy your visit? 4. Please rank order the following with 1 being the most important and 7 being the least important 5. When you purchased ____ what other options did you consider? 6. Did you use the features on our website? 7. In order to speed up your shopping experience, have you used the self check-out lanes? 8. How likely are you willing to pay $50 for a ____?
  27. 27. Rating Scales ✓ Label options with words, not numbers ✓ Ensure range of options covers all points on continuum ✓ For bipolar constructs use 7-point scale ✓ For single constructs use 5-point scale ✓ Don’t offer “Don’t Know” response ✓ Rotate order of response on categorical questions and wherever else it is practical How satisfied were you with the speed of check-out?
  28. 28. Communication Plan ✓ Invitations and advance/introductory letters ✓ Self-administered questionnaires ✓ Reminder letters/emails
  29. 29. Self- Administered Interviewer- Administered Computer- Administered Importance of Pre-Test Traditional pre-test – match survey method to complexity Cognitive pre-test – think-aloud
  30. 30. Conclusions – Part 1 To address non-response bias: 1. Document and categorize non-responses 2. Look for correlations 3. Don’t be fooled by high response rate 4. Convert! 5. Hire the best interviewers and train them well 6. Use well-designed questionnaires 7. Use incentives thoughtfully 8. Gather information from interviewers about non-respondents (back to #2)
  31. 31. Conclusions – Part 2 To build the best questionnaires: 1. Start with the end in mind 2. Follow GDPs 3. Establish relationship 4. Use visual elements in limits 5. Craft questions that are clear and direct 6. Test against Grice’s Maxims 7. Think like a respondent (Tourangeau Model) 8. Benefit from research learnings about rating scales 9. Develop and implement a communication plan 10. Pretest, and if necessary, pre-test again
  32. 32. Poll Results Poll: Which of the following has a direct effect on data quality, as it relates to your research?
  33. 33. Thank You!!! Questions?  @questionpro