This document compares the CATI, IVR, and SMS survey modes. It finds that: 1. Response rates were lowest for IVR and SMS surveys at [VALUE]% and [VALUE]%, compared to [VALUE]% for CATI surveys. Face-to-face surveys had nearly perfect response of 99%. 2. IVR and SMS surveys over-represented young, male, urban, and educated respondents compared to face-to-face surveys. CATI surveys were more representative than IVR and SMS but still showed some biases. 3. Estimates of voting behavior from IVR and SMS surveys were biased compared to true rates, and weighting only partially reduced the bias. So while

1. www.rti.orgRTI International is a registered trademark and a trade name of Research Triangle Institute.
What is the Optimal Mode?
A Comparison of CATI, IVR, SMS
Pew Research Center Workshop on Public
Opinion in Africa, 30 November 2017
Charles Q. Lau, PhD, MS
1

2. What is the Optimal Mode?
2
SMS & IVR & Web & CATI: Rapid Data for
Public Health Surveillance (10 countries)
SMS: Tracking Vocational
School Graduates
SMS: Panel Surveys of Youth
SMS & Web: Panel for
Job Accelerator Trainees
SMS: Feasibility of SMS Surveys
IVR: Best Practices in IVR
Introductions
CATI: Technology Adoption
CATI & SMS: Health
Worker Panel Survey

3. Mobile Survey Modes
3
CATI
IVR SMS
Other Modes: Web and Chatbots

4. Errors in Mobile Phone Surveys
4
Coverage
Error
CATI, IVR, SMS all
suffer from same
coverage error
(exclude people
without mobiles)

5. Errors in Mobile Phone Surveys
5
Coverage
Error
Non-
Response
Error
CATI, IVR, SMS all
suffer from same
coverage error
(exclude people
without mobiles)
CATI, IVR, SMS share
common non-response
errors …
-No in-person
interviewer
-Short intros
-Poor network
connection
-Phones turned off

6. Errors in Mobile Phone Surveys
6
Coverage
Error
Non-
Response
Error
CATI, IVR, SMS all
suffer from same
coverage error
(exclude people
without mobiles)
CATI, IVR, SMS share
common non-response
errors …
... but CATI, IVR, SMS
differ in non-response
errors
-No in-person
interviewer
-Short intros
-Poor network
connection
-Phones turned off
-SMS, IVR lack
interviewer
-SMS, IVR require
tech familiarity
-SMS requires
literacy
-Concern about
cost for SMS
-But: Do SMS at
leisure

7. Errors in Mobile Phone Surveys
7
Coverage
Error
Non-
Response
Error
Measurement
Error
CATI, IVR, SMS all
suffer from same
coverage error
(exclude people
without mobiles)
CATI, IVR, SMS share
common non-response
errors …
... but CATI, IVR, SMS
differ in non-response
errors
CATI allows for
probing and more
complicated
questions …
… but IVR and
SMS may be better
for sensitive topics

8. Previous Research
Key Insights
 Response rates are low, especially for IVR and SMS
 Cross-sectional surveys with a single mode (SMS, IVR,
CATI) over-represent young, men, urban, educated
 Panel approach with face-to-face recruitment is promising
 Mixed mode approaches can improve quality
 Weighting might help reduce bias
Knowledge Gaps
 Limited apples-to-apples comparisons
 How biased are survey estimates? Can weighting help?
 How do modes differ with regard to cost and time?
8

9. Research Questions
9
1. How do response rates
differ among CATI, IVR,
SMS, and FTF surveys?
2. How representative are
respondents from CATI,
IVR, and SMS surveys?
3. Can IVR and SMS
provide an unbiased
estimate of voting
behavior? If there is bias,
can weights reduce bias?
4. How does the cost and
data collection time differ
across survey modes?

10. Method
 Mobile Phone Surveys
– Modes: CATI, IVR, SMS
– All used RDD sampling
– IVR and SMS conducted in 2017; CATI in 2016
– IVR, SMS used $1 incentive. CATI had experiment ($0 vs. $1)
– IVR and SMS asked same 12 questions; CATI asked 85 questions
 FTF: 2013 Nigeria Demographic and Health Survey (paper-and-
pencil, area probability sample with HH listing, HH quex used)
 Target population for all surveys is general population age 18-64
(though frames differ)
 Approach: Compare estimates from CATI, IVR, SMS to FTF (and to
each other)
10

11. RQ1. Response Rates
[VALUE]%
[VALUE]%
[VALUE]%
CATI IVR SMS
Response Rates by Mode
11
Response
Rate for FTF
is 99%

12. RQ2. Representativeness
12
40%
54%
62%
65%
25%
27% 27%
23%
18%
12%
8% 9%
17%
7%
2% 3%
FTF CATI IVR SMS
Age by Mode
18-29 30-39 40-49 50-64
53%
33%
28%
36%
FTF CATI IVR SMS
Female by Mode

13. RQ2. Representativeness
13
39%
5%
8%
3%
26%
9%
11%
6%
22%
46%
32%
41%
13%
40%
50% 51%
FTF CATI IVR SMS
Education by Mode
No school Primary
Secondary Post-secondary

14. RQ. Representativeness
14
71%
46%
34%
FTF IVR SMS
Married by Mode
57%
24% 21%
FTF IVR SMS
Village by Mode
70%
79%
71% 69%
FTF CATI IVR SMS
Radio by Mode
81% 85%
IVR SMS
Read Well by Mode

15. RQ3. Bias
15
31%
68%
59%
Estimates of Voting in 2015 Presidential
Election
SMS IVR True
Preliminary
analysis:
Weighting
reduces some
bias, but does not
eliminate bias.
Weighting by
some factors
(e.g., age)
increases bias.

16. RQ4. Cost and Time
Data Collection Cost
- SMS is 52% the cost of IVR.
- In this study, IVR is 43% and
SMS is 22% the cost of CATI.
However, IVR and SMS had 12
questions, CATI had 85.
- CATI is cheaper than IVR and
SMS for long (85 qx) surveys
-IVR and SMS is cheaper than
CATI for short (12 qx) surveys
16
Data Collection Time
Data collection days
 CATI: 43 days
 IVR: 42 days
 SMS: 16 days
Completes per day
 CATI: 88
 IVR: 43
 SMS: 172

17. Takeaways and Discussion
1. Don’t be fooled by incremental improvements in response rates
2. CATI, IVR, SMS can’t match representativeness of FTF surveys
3. Voting estimates are biased; weighting can’t save the day
4. IVR respondents are slightly more representative than SMS (age,
education, village). But are small differences worth the cost?
5. CATI has edge (age, education) over IVR/SMS.
6. For short, simple surveys, IVR & SMS may be best modes if you’re
willing to sacrifice a bit of representativeness
7. For complex or larger surveys (questions, N), CATI is your best bet.
Unresolved issue: Measurement differences between modes
Limitations: Nigeria only; cross-sectional surveys; may be topic specific;
can’t separate coverage, non-response, measurement; house effects
17

18. Tool: Surveda for Mixed Mode Surveys
18
As part of Bloomberg Philanthropies Data for Health project, RTI and InSTEDD
have built an open-source survey tool for mixed mode (SMS, IVR, Web)
surveys: https://surveda.instedd.org

19. More Information
Charles Q. Lau, PhD, MS
Survey Methodologist
+1.919.541.8798
clau@rti.org
19