Description of an experiment using LST to determine who is more likely to participate in a survey based on survey topic. Specifically will incentive of $25 or $50 lead to an increase in participation, will increasing survey duration impact participation, do incentive and duration predict willingness to participate, and do the main effects vary across topics and importance.

1. icf.com
Designing, implementing, and
analyzing Leverage Saliency
Theory experiments
Wed, April 17, 2018
FedCASIC 2019 | Washington, DC
Matt Jans, Julia Sumner, Rachael Forando, John
Boyle, James Dayton, Lew Berman
ICF

2. ICF proprietary and confidential. Do not copy, distribute, or disclose.
What is Leverage Salience(y) Theory?
24/17/19 - FedCASIC 2019 - BLS

3. ICF proprietary and confidential. Do not copy, distribute, or disclose.
Leverage Saliency Theory (LST) Principals
(Groves, Singer, & Corning, 2000)
 Participation decisions are a combination of three things
1. How salient a specific survey design feature is (i.e., whether it’s noticed…kind of odd use of “salience”)
2. How much weight a potential respondent puts on that feature (i.e., how important it is to them)
3. And the valence of that weight (i.e., does it nudge them toward or away from participation?)
34/17/19 - FedCASIC 2019 - BLS

4. ICF proprietary and confidential. Do not copy, distribute, or disclose.
Salience and Leverage for a Respondent and
Nonrespondent
44/17/19 - FedCASIC 2019 - BLS

5. ICF proprietary and confidential. Do not copy, distribute, or disclose.
LST Example
 Respondents who are interested in birding will be more likely to
participate in a survey than people who are not
 And more likely when the survey is titled “Birders’ Survey” than “Survey of Hobbies”
54/17/19 - FedCASIC 2019 - BLS

6. ICF proprietary and confidential. Do not copy, distribute, or disclose.
LST Example
 Respondents who are interested in birding will be more likely to
participate in a survey than people who are not
 And more likely when the survey is titled “Birders’ Survey” than “Survey of Hobbies”
 The stated survey topic (a design feature) influences participation and
thus, birding activities (and related characteristics) are over-represented
64/17/19 - FedCASIC 2019 - BLS

7. ICF proprietary and confidential. Do not copy, distribute, or disclose.
LST Example
 Respondents who are interested in birding will be more likely to
participate in a survey than people who are not
 And more likely when the survey is titled “Birders’ Survey” than “Survey of Hobbies”
 The stated survey topic (a design feature) influences participation and
thus, birding activities (and related characteristics) are over-represented
 Further, incentives will have less of an effect of people with high topic
interest…more of an effect on people with low topic interest
 Thus incentives help minimize nonresponse bias by increasing response from those with low topic interest
74/17/19 - FedCASIC 2019 - BLS

8. ICF proprietary and confidential. Do not copy, distribute, or disclose.
LST Example
 Respondents who are interested in birding will be more likely to
participate in a survey than people who are not
 And more likely when the survey is titled “Birders’ Survey” than “Survey of Hobbies”
 The stated survey topic (a design feature) influences participation and
thus, birding activities (and related characteristics) are over-represented
 Further, incentives will have less of an effect of people with high topic
interest…more of an effect on people with low topic interest
 Thus incentives help minimize nonresponse bias by increasing response from those with low topic interest
 Interest in a topic leads people to think it’s important that they respond
84/17/19 - FedCASIC 2019 - BLS

9. ICF proprietary and confidential. Do not copy, distribute, or disclose.
Our Research Questions
94/17/19 - FedCASIC 2019 - BLS

10. ICF proprietary and confidential. Do not copy, distribute, or disclose.
LST Research Questions in the Dimensions of
Survey Participation Study
RQ #1: Incentive: Increasing incentive amount from $25 to $50 will lead to an
increase in stated willingness to participate?
104/17/19 - FedCASIC 2019 - BLS

11. ICF proprietary and confidential. Do not copy, distribute, or disclose.
LST Research Questions in the Dimensions of
Survey Participation Study
RQ #1: Incentive: Increasing incentive amount from $25 to $50 will lead to an
increase in stated willingness to participate?
RQ #2: Duration: Increasing duration form 1 hour to 2 hours will reduce stated
willingness to participate?
114/17/19 - FedCASIC 2019 - BLS

12. ICF proprietary and confidential. Do not copy, distribute, or disclose.
LST Research Questions in the Dimensions of
Survey Participation Study
RQ #1: Incentive: Increasing incentive amount from $25 to $50 will lead to an
increase in stated willingness to participate?
RQ #2: Duration: Increasing duration form 1 hour to 2 hours will reduce stated
willingness to participate?
RQ #3: Interactions (moderation): Do incentive and duration interact to predict
participation willingness?
124/17/19 - FedCASIC 2019 - BLS

13. ICF proprietary and confidential. Do not copy, distribute, or disclose.
LST Research Questions in the Dimensions of
Survey Participation Study
RQ #1: Incentive: Increasing incentive amount from $25 to $50 will lead to an
increase in stated willingness to participate?
RQ #2: Duration: Increasing duration form 1 hour to 2 hours will reduce stated
willingness to participate?
RQ #3: Interactions (moderation): Do incentive and duration interact to predict
participation willingness?
RQ #4: Generalizability of effects (LST): Do these main effects vary across …
4a: Survey topics and topic importance
4b: Types of respondents
- Demographics
- Survey experience
- Civic involvement and general interests
134/17/19 - FedCASIC 2019 - BLS

14. ICF proprietary and confidential. Do not copy, distribute, or disclose.
Principle #1: Start with problem/question/theory-
based research questions
 In this case, LST is the theory
 Write out problem statement (i.e., research questions) clearly
 State as hypotheses if possible
 Directional if possible
144/17/19 - FedCASIC 2019 - BLS

15. ICF proprietary and confidential. Do not copy, distribute, or disclose.
Designing our Experiment
154/17/19 - FedCASIC 2019 - BLS

16. ICF proprietary and confidential. Do not copy, distribute, or disclose.
Principal #2: Experimental factors flow from
research questions
 All respondents asked about…
 Importance/Unimportance of several topics (RQ #4b)
 Respondents randomized factorially to…
 Incentive level $25 v. $50 (RQ #1)
 Interview duration 1 hr. v. 2 hrs (RQ #2)
 Tabular example of the design
 Unimportant survey topic (RQ #4b)
 Interesting survey topic (RQ #4b)
164/17/19 - FedCASIC 2019 - BLS
$25 $50
1 hour % willing to participate % willing to participate
2 hours % willing to participate % willing to participate
$25 $50
1 hour % willing to participate % willing to participate
2 hours % willing to participate % willing to participate

17. ICF proprietary and confidential. Do not copy, distribute, or disclose.
Topic Importance
 Q39 “If you were asked to participate in government surveys on the
following topics, how important do you think it would be for you to
participate?”
 Very important
 Somewhat important
 Not too important
 Not important at all
Topics
 Education and schools
 Child care and problems of parents
 Medicare and aging
 Voting and elections
 Health and disease prevalence
 Nutrition and physical activity
 Issues facing the nation
174/17/19 - FedCASIC 2019 - BLS

18. ICF proprietary and confidential. Do not copy, distribute, or disclose.
Participation likelihood/willingness
 “Imagine you were contacted this week to participate in a government survey
about [INSERT “VERY IMPORTANT” OR “SOMEWHAT IMPORTANT” TOPIC
FROM Q39] that was conducted face to face in your home at a time of your
choice, took one hour [OR two hours], and you received $25 [OR $50],
 How likely would you be to participate in a survey like this?”
 Definitely would
 Probably would
 Might or might not
 Probably would not
 Definitely would not
184/17/19 - FedCASIC 2019 - BLS

19. ICF proprietary and confidential. Do not copy, distribute, or disclose.
Participation likelihood/willingness
 “Imagine you were contacted this week to participate in a government survey
about [INSERT “NOT TOO IMPORTANT” OR “NOT IMPORTANT” TOPIC
FROM Q39] that was conducted face to face in your home at a time of your
choice, took one hour [OR two hours], and you received $25 [OR $50],
 How likely would you be to participate in a survey like this?”
 Definitely would
 Probably would
 Might or might not
 Probably would not
 Definitely would not
194/17/19 - FedCASIC 2019 - BLS
Previous question with unimportant
topics filled.

20. ICF proprietary and confidential. Do not copy, distribute, or disclose.
Analyzing Results
204/17/19 - FedCASIC 2019 - BLS

21. ICF proprietary and confidential. Do not copy, distribute, or disclose.
Principal #3: Consider (i.e., plan) the analysis
approach before collecting data
 Easy for experimental studies
214/17/19 - FedCASIC 2019 - BLS

22. ICF proprietary and confidential. Do not copy, distribute, or disclose.
Principal #3: Consider (i.e., plan) the analysis
approach before collecting data
 Easy for experimental studies
 First explore experimental factors through ANOVA or linear regression
(continuous outcome) or logistic regression (binary outcome)
 Code multi-category responses into binary ones (e.g., combining very likely and somewhat likely)
 Conduct statistical tests
– Significance of main effects and interaction
 Presentation visualization
– Bar charts of %s (binary outcome)
– Box and whisker plots (continuous outcomes)
– For interactions, “cross-over plots” are VERY helpful
224/17/19 - FedCASIC 2019 - BLS

23. ICF proprietary and confidential. Do not copy, distribute, or disclose.
Principal #3: Consider (i.e., plan) the analysis
approach before collecting data
 Easy for experimental studies
 First explore experimental factors through ANOVA or linear regression
(continuous outcome) or logistic regression (binary outcome)
 Code multi-category responses into binary ones (e.g., combining very likely and somewhat likely)
 Conduct statistical tests
– Significance of main effects and interaction
 Presentation visualization
– Bar charts of %s (binary outcome)
– Box and whisker plots (continuous outcomes)
– For interactions, “cross-over plots” are VERY helpful
 Second, compare significance and form of interactions between key
groups
 Respondents interested in topic should be less influenced by incentive and duration than those who aren’t
234/17/19 - FedCASIC 2019 - BLS

24. ICF proprietary and confidential. Do not copy, distribute, or disclose.
Principal #3: Consider (i.e., plan) the analysis
approach before collecting data
 Easy for experimental studies
 First explore experimental factors through ANOVA or linear regression
(continuous outcome) or logistic regression (binary outcome)
 Code multi-category responses into binary ones (e.g., combining very likely and somewhat likely)
 Conduct statistical tests
– Significance of main effects and interaction
 Presentation visualization
– Bar charts of %s (binary outcome)
– Box and whisker plots (continuous outcomes)
– For interactions, “cross-over plots” are VERY helpful
 Second, compare significance and form of interactions between key
groups
 Respondents interested in topic should be less influenced by incentive and duration than those who aren’t
 Third, explore whether adding additional predictors (e.g., R
characteristics change nature of interaction)
244/17/19 - FedCASIC 2019 - BLS

25. ICF proprietary and confidential. Do not copy, distribute, or disclose.
Analysis methods for our LST experiment
svy:logistic Q40_important_will_bin ///
int_dur_bin##inc_amt_bin
contrast int_dur_bin##inc_amt_bin
margins int_dur_bin#inc_amt_bin
marginsplot…
254/17/19 - FedCASIC 2019 - BLS
Incentive main effect
(F = 11.35, p = .0008)
Duration main effect
(F = 3.68, p = .0551)
Interaction btwn
Incentive and Duration
(F = 5.86, p = .0155)

26. ICF proprietary and confidential. Do not copy, distribute, or disclose.
Exploring the effect of R characteristics – Residence
location
svy:logistic Q40_important_will_bin ///
int_dur_bin##inc_amt_bin##town_large_city_bin
contrast int_dur_bin##inc_amt_bin@town_large_city_bin
margins int_dur_bin#inc_amt_bin#town_large_city_bin
marginsplot…
264/17/19 - FedCASIC 2019 - BLS
Incentive main effect
(F = 8.68, p = .0033)
Duration main effect
(F = 3.54, p = .0602
Incentive main effect
(F = 3.57, p = .0591)
Interaction btwn
incentive and duration
(F = 6.38, p = .0116)

27. ICF proprietary and confidential. Do not copy, distribute, or disclose.
Exploring the effect of R characteristics - Age
274/17/19 - FedCASIC 2019 - BLS

28. ICF proprietary and confidential. Do not copy, distribute, or disclose.
LST Design and Analysis Checklist
284/17/19 - FedCASIC 2019 - BLS

29. ICF proprietary and confidential. Do not copy, distribute, or disclose.
Checklists flow from the framework
 Principal #1: Start with problem/question/theory-based research
questions
 Define the problem statement or hypothesis you want to test
 Which design features do you want to manipulate? How many levels (and what values) will they have?
 Will you use a “captive audience“ and hypothetical response, or “field survey”
 What respondent (or sample member) sub-groups do you want to compare? What indicators of topic interest
do you have?
 Write specific research questions or hypotheses, including the direction of expected effect
294/17/19 - FedCASIC 2019 - BLS

30. ICF proprietary and confidential. Do not copy, distribute, or disclose.
Checklists flow from the framework
 Principal #2: Experimental factors flow from research questions
Define the factors you want to study (i.e., comparisons you want to make)
 Consider counter-factuals (i.e., comparison and control groups)
 If you want to make causal inferences, make them experimental manipulations via random assignment
 Confirm that the experimental factors will answer your research questions (or adjust research questions)
304/17/19 - FedCASIC 2019 - BLS

31. ICF proprietary and confidential. Do not copy, distribute, or disclose.
Checklists flow from the framework
 Principal #3: Consider (i.e., plan) the analysis approach before collecting
data
Choose modeling method (e.g., ANOVA, linear regression, logistic regression)
 Define how effects and interactions will be tested and reported (e.g., odds ratios v. coefficients)
 Determine visualizations that you will create before or during modeling and testing
314/17/19 - FedCASIC 2019 - BLS

32. ICF proprietary and confidential. Do not copy, distribute, or disclose. 32
Matt Jans: matt.jans@icf.com
4/17/19 - FedCASIC 2019 - BLS