The document outlines strategies for developing high-quality survey instruments to gauge support for controversial topics, specifically violence and public health initiatives. It emphasizes the use of simple questions, experimental measures, and the need to account for biases and comprehension challenges among respondents. The focus is on evaluating overt and covert measures of support, such as public opinions on vaccination plans influenced by extremist groups.

1. Strategies for Developing High-Quality
Survey Instruments
John F. McCauley
Associate Professor, University of Maryland
Pew Research Center
November 30, 2017

3. Focus:
Innovations in Survey Experimental Questionnaires

4. Overt Measures; Stated Support for VE:
1. “Violence against civilians to defend religion can
be justified.”
(2. Violence in the name of Islam can be justified)
(3. Violence can be justified to solve problems)

5. Covert Measures; True Support for VE:
“The government is considering a national
polio vaccination. Would you say that you
are in favor or against this initiative?”
In favor
Against
Don’t know/no answer
“The government is considering a national
polio vaccination, which Al Qaeda IM’s
leaders likely oppose. Would you say that
you are in favor or against this initiative?”
In favor
Against
Don’t know/no answer
Control Treatment

6. Advantages:
1. Causal Inference

7. Support for
Vaccination
Plan

8. Advantages:
1. Causal Inference
2. Exposes Social Desirability Bias

10. Advantages:
1. Causal Inference
2. Exposes Social Desirability Bias
3. Respondent Engagement

12. Challenges:
1. Basic Comprehension Levels
2. Multiple Local Languages
3. Consistency bias, order effects
4. Assuming the General from the Specific
5. Enumerator Fallibility

13. Strategies:
1. Short, simple questions

14. Covert Measures; True Support for VE:
“The government is considering a national
polio vaccination. Would you say that you
are in favor or against this initiative?”
In favor
Against
Don’t know/no answer
“The government is considering a national
polio vaccination, which Al Qaeda IM’s
leaders likely oppose. Would you say that
you are in favor or against this initiative?”
In favor
Against
Don’t know/no answer
Control Treatment

15. Alternative:
“The government is considering shutting
down schools in some areas of the North.
Would you say that you are in favor or
against this plan?”
In favor
Against
Don’t know/no answer
“The government is considering shutting
down schools in some areas of the North,
which Al Qaeda IM’s leaders likely
support. Would you say that you are in
favor or against this plan?”
In favor
Against
Don’t know/no answer
Control Treatment

16. Alternative:
“The government is considering shutting
down schools in some areas of the North.
Would you say that you are in favor or
against this plan?”
In favor
Against
Don’t know/no answer
“The government is considering shutting
down schools in some areas of the North,
which Al Qaeda IM’s leaders likely
support. Would you say that you are in
favor or against this plan?”
In favor
Against
Don’t know/no answer
Control Treatment
In favor of policy =
In favor of terrorist group

17. Strategies:
1. Short, simple questions
2. Include overt and experimental measures

18. Strategies:
1. Short, simple questions
2. Include overt and experimental measures
3. Include multiple measures and vary placement

19. Strategies:
1. Short, simple questions
2. Include overt and experimental measures
3. Include multiple measures and vary placement
4. Vary treatment (endorsement)

20. Covert Measures; True Support for VE:
“The government is considering a national
polio vaccination. Would you say that you
are in favor or against this initiative?”
In favor
Against
Don’t know/no answer
“The government is considering a national
polio vaccination, which Al Qaeda IM’s
leaders likely oppose. Would you say that
you are in favor or against this initiative?”
In favor
Against
Don’t know/no answer
Control Treatment

21. Alternative 2:
“The government is considering a national
polio vaccination. Would you say that you
are in favor or against this initiative?”
In favor
Against
Don’t know/no answer
“The government is considering a national
polio vaccination, which Boko Haram’s
leaders likely oppose. Would you say that
you are in favor or against this initiative?”
In favor
Against
Don’t know/no answer
Control Treatment

22. “The government is considering a national
polio vaccination. Would you say that you
are in favor or against this initiative?”
In favor
Against
Don’t know/no answer
“The government is considering a national
polio vaccination, which Boko Haram’s
leaders likely oppose. Would you say that
you are in favor or against this initiative?”
In favor
Against
Don’t know/no answer
Control Treatment
Controls for specific
endorser effect
Alternative 2: