Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

11. RTI International Presentation

1,936 views

Published on

Presentation for Pew Research Center's Methodological Workshop on Public Opinion Research in Sub-Saharan Africa

Published in: Data & Analytics
  • Hi there! Get Your Professional Job-Winning Resume Here - Check our website! http://bit.ly/resumpro
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

11. RTI International Presentation

  1. 1. www.rti.orgRTI International is a registered trademark and a trade name of Research Triangle Institute. What is the Optimal Mode? A Comparison of CATI, IVR, SMS Pew Research Center Workshop on Public Opinion in Africa, 30 November 2017 Charles Q. Lau, PhD, MS 1
  2. 2. What is the Optimal Mode? 2 SMS & IVR & Web & CATI: Rapid Data for Public Health Surveillance (10 countries) SMS: Tracking Vocational School Graduates SMS: Panel Surveys of Youth SMS & Web: Panel for Job Accelerator Trainees SMS: Feasibility of SMS Surveys IVR: Best Practices in IVR Introductions CATI: Technology Adoption CATI & SMS: Health Worker Panel Survey
  3. 3. Mobile Survey Modes 3 CATI IVR SMS Other Modes: Web and Chatbots
  4. 4. Errors in Mobile Phone Surveys 4 Coverage Error CATI, IVR, SMS all suffer from same coverage error (exclude people without mobiles)
  5. 5. Errors in Mobile Phone Surveys 5 Coverage Error Non- Response Error CATI, IVR, SMS all suffer from same coverage error (exclude people without mobiles) CATI, IVR, SMS share common non-response errors … -No in-person interviewer -Short intros -Poor network connection -Phones turned off
  6. 6. Errors in Mobile Phone Surveys 6 Coverage Error Non- Response Error CATI, IVR, SMS all suffer from same coverage error (exclude people without mobiles) CATI, IVR, SMS share common non-response errors … ... but CATI, IVR, SMS differ in non-response errors -No in-person interviewer -Short intros -Poor network connection -Phones turned off -SMS, IVR lack interviewer -SMS, IVR require tech familiarity -SMS requires literacy -Concern about cost for SMS -But: Do SMS at leisure
  7. 7. Errors in Mobile Phone Surveys 7 Coverage Error Non- Response Error Measurement Error CATI, IVR, SMS all suffer from same coverage error (exclude people without mobiles) CATI, IVR, SMS share common non-response errors … ... but CATI, IVR, SMS differ in non-response errors CATI allows for probing and more complicated questions … … but IVR and SMS may be better for sensitive topics
  8. 8. Previous Research Key Insights  Response rates are low, especially for IVR and SMS  Cross-sectional surveys with a single mode (SMS, IVR, CATI) over-represent young, men, urban, educated  Panel approach with face-to-face recruitment is promising  Mixed mode approaches can improve quality  Weighting might help reduce bias Knowledge Gaps  Limited apples-to-apples comparisons  How biased are survey estimates? Can weighting help?  How do modes differ with regard to cost and time? 8
  9. 9. Research Questions 9 1. How do response rates differ among CATI, IVR, SMS, and FTF surveys? 2. How representative are respondents from CATI, IVR, and SMS surveys? 3. Can IVR and SMS provide an unbiased estimate of voting behavior? If there is bias, can weights reduce bias? 4. How does the cost and data collection time differ across survey modes?
  10. 10. Method  Mobile Phone Surveys – Modes: CATI, IVR, SMS – All used RDD sampling – IVR and SMS conducted in 2017; CATI in 2016 – IVR, SMS used $1 incentive. CATI had experiment ($0 vs. $1) – IVR and SMS asked same 12 questions; CATI asked 85 questions  FTF: 2013 Nigeria Demographic and Health Survey (paper-and- pencil, area probability sample with HH listing, HH quex used)  Target population for all surveys is general population age 18-64 (though frames differ)  Approach: Compare estimates from CATI, IVR, SMS to FTF (and to each other) 10
  11. 11. RQ1. Response Rates [VALUE]% [VALUE]% [VALUE]% CATI IVR SMS Response Rates by Mode 11 Response Rate for FTF is 99%
  12. 12. RQ2. Representativeness 12 40% 54% 62% 65% 25% 27% 27% 23% 18% 12% 8% 9% 17% 7% 2% 3% FTF CATI IVR SMS Age by Mode 18-29 30-39 40-49 50-64 53% 33% 28% 36% FTF CATI IVR SMS Female by Mode
  13. 13. RQ2. Representativeness 13 39% 5% 8% 3% 26% 9% 11% 6% 22% 46% 32% 41% 13% 40% 50% 51% FTF CATI IVR SMS Education by Mode No school Primary Secondary Post-secondary
  14. 14. RQ. Representativeness 14 71% 46% 34% FTF IVR SMS Married by Mode 57% 24% 21% FTF IVR SMS Village by Mode 70% 79% 71% 69% FTF CATI IVR SMS Radio by Mode 81% 85% IVR SMS Read Well by Mode
  15. 15. RQ3. Bias 15 31% 68% 59% Estimates of Voting in 2015 Presidential Election SMS IVR True Preliminary analysis: Weighting reduces some bias, but does not eliminate bias. Weighting by some factors (e.g., age) increases bias.
  16. 16. RQ4. Cost and Time Data Collection Cost - SMS is 52% the cost of IVR. - In this study, IVR is 43% and SMS is 22% the cost of CATI. However, IVR and SMS had 12 questions, CATI had 85. - CATI is cheaper than IVR and SMS for long (85 qx) surveys -IVR and SMS is cheaper than CATI for short (12 qx) surveys 16 Data Collection Time Data collection days  CATI: 43 days  IVR: 42 days  SMS: 16 days Completes per day  CATI: 88  IVR: 43  SMS: 172
  17. 17. Takeaways and Discussion 1. Don’t be fooled by incremental improvements in response rates 2. CATI, IVR, SMS can’t match representativeness of FTF surveys 3. Voting estimates are biased; weighting can’t save the day 4. IVR respondents are slightly more representative than SMS (age, education, village). But are small differences worth the cost? 5. CATI has edge (age, education) over IVR/SMS. 6. For short, simple surveys, IVR & SMS may be best modes if you’re willing to sacrifice a bit of representativeness 7. For complex or larger surveys (questions, N), CATI is your best bet. Unresolved issue: Measurement differences between modes Limitations: Nigeria only; cross-sectional surveys; may be topic specific; can’t separate coverage, non-response, measurement; house effects 17
  18. 18. Tool: Surveda for Mixed Mode Surveys 18 As part of Bloomberg Philanthropies Data for Health project, RTI and InSTEDD have built an open-source survey tool for mixed mode (SMS, IVR, Web) surveys: https://surveda.instedd.org
  19. 19. More Information Charles Q. Lau, PhD, MS Survey Methodologist +1.919.541.8798 clau@rti.org 19

×