Presentation for Pew Research Center's Methodological Workshop on Public Opinion Research in Sub-Saharan Africa

1. Using CAPI to assess interviewer
skills during training:
Presentation at the Pew Research Center Workshop
on data collection in Sub-Saharan Africa, Washington,
D.C.
advice from Kenya, Ghana, Uganda and
Senegal
Sarah M. Hughes
November 30, 2017

2. 22
“Collecting good
data is one of the
most powerful tools
to end extreme
poverty.”
Jim Yong Kim 2015
source: World Bank, 2015

3. 33
Outline of the presentation
• Why focus on interviewer training?
• Introducing 3 training tools
– Round robin
– Mock script
– Mock dataset
• Assessing interviewers
• Pros and cons of the method

4. 44
Outline of the presentation
• Why focus on interviewer training?
• Introducing 3 training tools
– Round robin
– Mock script
– Mock dataset
• Assessing interviewers
• Pros and cons of the method

5. 55
Uneven statistical capacity
• Limited training for
statistical capacity
• Use of paper
questionnaires
• Few private
organizations with
high-quality survey
methodology
capabilities
• Pay and
employment
structure for field
teams
AfDB ranking statistical capacity for collecting agriculture data
African Development Bank, 2014

6. 66
Outline of the presentation
• Why focus on interviewer training?
• Introducing 3 training tools
– Round robin
– Mock script
– Mock dataset
• Assessing interviewers
• Pros and cons of the method

7. 77
Interviewer
1
Iboughtit
Interviewer
2
Interviewer
3
Howdidyou
obtainthisplot?
Trainer /
respondent
Round Robin: interviewers have tablets

8. 88
Outline of the presentation
• Why focus on interviewer training?
• Introducing 3 training tools
– Round robin
– Mock script
– Mock dataset
• Assessing interviewers
• Pros and cons of the method

9. 99
Using the mock script
SIMPLE MOCK: "MA DIOP" TRAINER: INSTRUCT INTERVIEWERS TO SELECT RESPONDENT MA DIOP FROM
THEIR TRAINING SAMPLE CASES
Q # VARNAME QUESTION TEXT
TRAINER'S
VERBATIM
RESPONSE
EXPECTED
INTERVIEWER
ACTION
STOP AND EXPLAIN:
TRAINER NOTES
B.10 HHNUMCF I would like to confirm
that the total number of
people living in your
household is %textfill%.
Well, there's me, my
husband, our two kids
and my mother in law.
PROBE: So that would
be how many? (or) So,
that would be five?
Respondents sometimes have a hard time
adding up numbers and you might make a
mistake recording the number. So,
remember to probe and to repeat
numbers
Interviewer uses CAPI tablet
to pose the question

10. 1010
Using the mock script
SIMPLE MOCK: "MA DIOP" TRAINER: INSTRUCT INTERVIEWERS TO SELECT RESPONDENT MA DIOP FROM
THEIR TRAINING SAMPLE CASES
Q # VARNAME QUESTION TEXT
TRAINER'S
VERBATIM
RESPONSE
EXPECTED
INTERVIEWER
ACTION
STOP AND EXPLAIN:
TRAINER NOTES
B.10 HHNUMCF I would like to confirm
that the total number of
people living in your
household is %textfill%.
Well, there's me, my
husband, our two
kids and my mother
in law.
PROBE: So that would
be how many? (or) So,
that would be five?
Respondents sometimes have a hard time
adding up numbers and you might make a
mistake recording the number. So,
remember to probe and to repeat
numbers
Trainer reads aloud exactly as written

11. 1111
Using the mock script
SIMPLE MOCK: "MA DIOP" TRAINER: INSTRUCT INTERVIEWERS TO SELECT RESPONDENT MA DIOP FROM
THEIR TRAINING SAMPLE CASES
Q # VARNAME QUESTION TEXT
TRAINER'S
VERBATIM
RESPONSE
EXPECTED
INTERVIEWER
ACTION
STOP AND EXPLAIN:
TRAINER NOTES
B.10 HHNUMCF I would like to confirm
that the total number of
people living in your
household is %textfill%.
Well, there's me, my
husband, our two kids
and my mother in law.
PROBE: So that would
be how many? (or) So,
that would be five?
Respondents sometimes have a hard time
adding up numbers and you might make a
mistake recording the number. So,
remember to probe and to repeat
numbers
Trainers observe how interviewer probes

12. 1212
Using the mock script
SIMPLE MOCK: "MA DIOP" TRAINER: INSTRUCT INTERVIEWERS TO SELECT RESPONDENT MA DIOP FROM
THEIR TRAINING SAMPLE CASES
Q # VARNAME QUESTION TEXT
TRAINER'S
VERBATIM
RESPONSE
EXPECTED
INTERVIEWER
ACTION
STOP AND EXPLAIN:
TRAINER NOTES
B.10 HHNUMCF I would like to confirm
that the total number of
people living in your
household is %textfill%.
Well, there's me, my
husband, our two kids
and my mother in law.
PROBE: So that would
be how many? (or) So,
that would be five?
Respondents sometimes have a hard time
adding up numbers and you might make
a mistake recording the number. So,
remember to probe and to repeat
numbers.
Trainer pauses and explains
to improve interviewer
behavior and understanding

13. 1313
Mock script: only for the trainer!
SIMPLE MOCK: "MA DIOP" TRAINER: INSTRUCT INTERVIEWERS TO SELECT RESPONDENT MA DIOP FROM
THEIR TRAINING SAMPLE CASES
Q # VARNAME QUESTION TEXT
TRAINER'S
VERBATIM
RESPONSE
EXPECTED
INTERVIEWER
ACTION
STOP AND EXPLAIN:
TRAINER NOTES
B.10 HHNUMCF I would like to confirm
that the total number of
people living in your
household is %textfill%.
Well, there's me, my
husband, our two kids
and my mother in law.
PROBE: So that would
be how many? (or) So,
that would be five?
Respondents sometimes have a hard time
adding up numbers and you might make a
mistake recording the number. So,
remember to probe and to repeat
numbers
G.15 PLOTOWN How did you obtain this
plot? 1-Purchase , 2-
Inherit , 3-Rent for cash or
kind , 4-Sharecrop , 5-
Allocated free, 6-Other,
specify
I bought it Select 1-Purchase Be sure to read and know the response
options so you select from the precoded
responses.
G.16 SEEDS$ How much did you spend
on seeds for this plot?
5,000 CFA REPEAT RESPONSE,
Enter 5,000
It is easy to make mistakes when entering
numbers on a tablet-read the number you
have entered out loud to the respondent
to make sure you have entered it correctly
L.1 PHONE1 What phone number
should we use to contact
you for follow up in 2-3
months?
77-328-44-91 REPEAT RESPONSE,
enter 77-328-44-91
We check phone numbers for accuracy
and follow up, so be sure this number is
correct.

14. 1414
Interviewer
1
Iboughtit
Interviewer
2
Interviewer
3
Howdidyou
obtainthisplot?
Trainer /
respondent/
Ma Diop
Round Robin: interviewers have tablets

15. 1515
Outline of the presentation
• Why focus on interviewer training?
• Introducing 3 training tools
– Mock script
– Round robin
– Mock dataset
• Assessing interviewers
• Pros and cons of the method

16. 1616
Create excel version of data
Filter columns

17. 1717
Identify errors
Identify values that do not
match the Mock script

18. 1818
Highlight for ease of presentation
Highlight wrong values

19. 1919
Identify data quality errors by interviewer
This interviewer has made multiple errors

20. 2020
Outline of the presentation
• Why focus on interviewer training?
• Introducing 3 training tools
– Mock script
– Round robin
– Mock dataset
• Assessing interviewers
• Pros and cons of the method

21. 2121
Interviewer assessment
• Expect errors during training!
• Mock script with data review allows assessment of:
– reading
– listening
– recording
• Does NOT assess ability to select correct respondent or to
gain cooperation
• Does NOT replace supervisor observation or back-checks

22. 2222
Outline of the presentation
• Why focus on interviewer training?
• Introducing 3 training tools
– Mock script
– Round robin
– Mock dataset
• Assessing interviewers
• Pros and cons of the method

23. 2323
Pros and cons of the method
Pros Cons
Identify trainees for additional training before
hiring
Time consuming for material
development
Trains on content, basic interviewer skills (active
listening, probing, correctly entering numeric
values), tablet/CAPI function
Requires new approach to
classroom training for local team
Forces local team to become very familiar with
questionnaire, programming
Requires Training of Trainers
Adds objectivity to interviewer assessment

24. 2424
Better training = better data
“Data are the lifeblood of
decision-making and the raw
material for accountability.
Without high-quality data
providing the right information
on the right things at the right
time; designing, monitoring
and evaluating effective
policies becomes almost
impossible.”
United Nations, A World That
Counts, 2014

25. 2525
For More Information
• Sarah Hughes
– SHughes@mathematica-mpr.com