Recruitment for Hard-to-Reach Populations: LGBTQ Youth (E. Fordyce)

1. At the Crossroads of Social Media and Clinical Trials: A Workshop
on the Future of Clinician, Patient and Community Engagement
June 8, 2018
Erin Fordyce, MS, MEd
*The findings and conclusions in this presentation are those of
the authors and do not necessarily represent the official position
of the Centers for Disease Control and Prevention.
Social Media Recruitment for Hard-to-Reach
Populations:
Surveying LGBTQ Youth

2. • Benefits and Disadvantages of Social Media Recruitment
• Know Your Audience
• Sampling and Data Quality
• Recruiting LGBTQ Youth
• Implications for Future Research
• Upcoming Work
2
Overview

3. 3
Benefits and Disadvantages
Benefits Declining response rates in other modes
Advancing Technology
Faster Data Collection
Less Expensive
Hard-to-Reach Populations (conflicting findings)
Decreased respondent burden
Disadvantages Potential Bias
Mode Effects
Data Quality
Site Popularity Changes

4. Footer Information Here 4
Know Your Audience -
Social Media Site

5. Footer Information Here 5
Know Your Audience -
Messaging Apps

6. • Nonprobability sampling
• Population not given equal chance to participate
• Little progress in attempts to show how data collected through the use of
social media sites can represent the general population (Murphy et al
2014).
• Coverage error
– Multiple accounts on the same platform,
– Shared accounts
– Accounts can represent companies or products instead of individuals
(Murphy et al 2014, Nexgate 2013).
• Active sampling
• Using social media sites and search engines to collect sample, collect
survey responses, or both.
6
Sampling and Data Quality

7. • Measuring data quality with paradata
• Completion time
• Breakoffs
• Nonsubstantive answers
• Rounding/heaping
• Mobile device
• Incentives
7
Sampling and Data Quality

8. 8
Recruiting LGBTQ Youth
• Sponsor: CDC Division of Adolescent and School
Health
• Purpose: Learn about knowledge, attitudes, and
behaviors related to HIV prevention. Develop effective
methods for recruiting hard-to-reach populations.
• Target Population: Males living in the U.S. ages 13-18 who are attracted to males and
transgender individuals ages 13-24. Oversampling Black/African American and Hispanic.
• OMB/IRB: Approved by OMB as formative research and NORC’s IRB
• Pre-Testing: Cognitive interviews (completed) and Pilot (completed)
• Current Step: National survey (finishing data collection)
• Next Steps: Tool development

9. Footer Information Here 9
Ad Development
• Video and static ads created for social media recruitment
• Ads designed to target one of three groups – msm, trans, general teen
population
• Ads designed to also target Black/African American and Hispanic teens
• Experimented with eligibility language
• Developed by NORC staff and Youth Community Advisory Board (YCAB)
• YCAB organized by Fenway Institute (Boston, MA)
• 13 YCAB members (ages 15-19), with diverse sexual orientations,
gender identities, and racial/ethnic identities
• Final ads presented to cognitive interview respondents

10. Footer Information Here 10
Ad Targeting – Facebook/Instagram
• AMSM
• U.S.
• 13-18 years old
• Males
• Interested in men or unspecified
• Transgender
• U.S.
• 13-24 years old
• All genders
• Additional interests: Adam
Bouska, Laverne Cox, National
Center for Transgender Equality,
RuPaul’s Drag Race, The Trevor
Project, transgenderism
• General Population
• U.S.
• 13-24 years old
• All genders

11. Footer Information Here 11
Ad Targeting - Snapchat
• U.S.
• 13-24 years old
• All genders
• Likes (any of the following):
• Zendaya
• Bryan Yambao
• Ellen DeGeneres
• Lady Gaga
• Laverne Cox
• Kylie Jenner
• Tyler Oakley
• Interests:
– adventure seekers, beauty mavens
music fans, fashion and style gurus, high
schoolers, film and TV fans, fitness enthusiasts

12. Footer Information Here 12
Ad Targeting - Google
U.S.
13-24 year olds
Keywords
 Gay, teens, trans, gay men,
gay boys, trans people, gender identity, msm, HIV, LGBT, act
up, sexual identity, pflag, transgender, ILGA, GLAAD, pride
parade, GLSEN, Human Rights Campaign, Being trans
 (Phase 2 only) Gender non-binary, gender queer, STI, STD,
AIDS, PrEP, nPEP, Truvada, condom, prevention, testing, youth,
adolescent, teen, bullying, pride, queer, gay pride, gay-straight,
non-binary, pansexual, bisexual, questioning, VD, rainbow flag

13. Footer Information Here 13
START Methodology
Phase 1
• Launched 5pm on Friday, January 19, 2018
• Closed 12pm on Saturday, January 20, 2018
• Ads open approximately 19 hours
• Ads posted on Facebook, Instagram, Snapchat, and Google AdWords
Phase 2
• Launched 9:30am on Friday, February 16, 2018
• Closed 1pm on Friday, February 16, 2018
• Ads open approximately 3 1/2 hours
• Ads posted on Facebook, Instagram, and Google AdWords
Phase 3
• Launched 2pm on Monday, March 19, 2018
• Closed 9am on Monday, March 26, 2018
• Ads open one week
• Ads posted on Facebook, Instagram, and Snapchat
Pilot

14. Footer Information Here 14
Challenges/Solutions

15. Footer Information Here 15
Challenges/Solutions

16. https://drive.google.com/file/d/1b90zD2fe_pze268yFPz_Gze5-yafGY9E/viewr 16
Most Completes
Most Completes
317
264
251

17. https://drive.google.com/drive/folders/0B0QGPMMWUcmGTC1lcElIRGZZNFE 17
Most Completes
Most Completes
241
137

18. Footer Information Here 18
Results
Total (n) Facebook % (n) Instagram % (n) Snapchat % (n)
Started Screener 3,952 100% (253) 100% (1,675) 100% (2,020)
Partial Screener 189 3% (7) 5% (85) 5% (97)
Ineligible* 1,532 16% (40) 17% (283) 60% (1,206)
Screener
complete
2,231 81% (206) 78% (1,307) 35% (717)
Nonresponse
Ineligible**
146 0% (0) 3% (55) 5% (91)
Partial survey
complete
536 7% (18) 17% (277) 12% (241)
Complete 1,504 74% (187) 57% (959) 18% (357)
Table 1. Completion Percentage within Platforms

19. 19
Results
6%
42%
51%
12%
64% 9%
0
500
1000
1500
2000
2500
3000
Facebook Instagram Snapchat
Percentage Started Survey vs. Percentage Completed
Across Platforms
Started Screener Completed

20. Footer Information Here 20
Results
934
(62%)
570
(38%)
0
100
200
300
400
500
600
700
800
900
1000
MSM Trans
Completes by Targeted Group

21. Footer Information Here 21
Results
12%
8% 3% 46%
68%
74%
75%
41%
20%
18%
22%
13%
0
100
200
300
400
500
600
Black/African
American
Latino/a 13-14 years old Born male trans
Completes by Groups of Interest
Snapchat
Instagram
Facebook

22. 22
Implications for Future Research
• Ads on Facebook, Instagram, and Snapchat proved to be effective for
recruiting young MSM and trans respondents
• Data collected quickly and cost was relatively low
• Recommend using a diverse ad set
• Google was not productive
• Data quality is still a concern
• Respondents clicking through for gift code
• Using fake email accounts to retrieve additional codes
• Completing on separate devices
• Coverage issues
• Ad sharing
• Facebook Data Breach
• News published after Phase 3 of data collection
• Possible slowed approval process for ad postings

23. • Study: Using Social Media for
Recruitment in Cancer Prevention
(SMFR)
• Sponsor: CDC, Division of Cancer
Prevention and Control
• Purpose: Assess the feasibility of
conducting survey-based cancer prevention and
control research using social media platforms for study recruitment.
• Target Population: Adult cancer survivors, those at high risk for cancer, and
general population living in the U.S.
• Methods: Ads to be posted to Facebook and Twitter
23
Upcoming Work

24. • Antoun, C., Zhang, C., Conrad, F.G. & Schober, M.F. (2013) Comparisons of Online
Recruitment Strategies: Craigslist, Google Ads and Amazon’s Mechanical Turk. Presented at
the Annual Conference of the American Association of Public Opinion Research. Boston,
MA.
• Bilgen, I., Fontes, A., Horton, R., and Ventura, I. 2015. Recruitment of Targeted Populations
via Social Media: Examination of Nonprobability Based Sampling Approaches. Presented at
2016 American Association for Public Opinion Research Conference, Austin.
• Duggan, M., & Brenner, J. 2013. The demographics of social media users—2012. Pew
Research Center. Retrieved from
http://www.pewinternet.org/files/oldmedia//Files/Reports/2013/PIP_SocialMediaUsers.pdf
• Duggan, M., & Smith, A. 2013. Social media update 2013. Pew Research Center. Retrieved
from http://pewinternet.org/Reports/2013/Social-Media-Update.aspx
• Greenwood, S., Perrin, A., and Duggan, M. 2016. Social Media Update 2016. Pew Research
Center. http://www.pewinternet.org/2016/11/11/social-media-update-2016/
• Lenhart, Amanda. "Teens, Social Media & Technology Overview 2015." Pew Research
Center Internet Science Tech RSS. Pew Research Center, 08 Apr. 2015. Web. 29 Feb. 2016.
• Morales, L. 2011. Google and Facebook users skew young, affluent, and educated. The
Gallup Organization. Retrieved from http://www.gallup.com/poll/146159/facebook-google-
users-skew-young-affluenteducated.aspx
24
References

25. • Murphy, J., Link, M., Childs, J., Tesfaye, C., Dean, E., Stern, M., & Schober, M. F.
(2014). Social media in public opinion research: Report of the AAPOR task force on
emerging technologies in public opinion research.[American Association for Public
Opinion Research].
• Nexgate. (2013). 2013 State of Social Media Spam. http://nexgate.com/wp-
content/uploads/2013/09/Nexgate-2013-State-of-Social-Media-Spam-Research-
Report.pdf
• Perrin, A. 2015. Social Media Site Usage: 2005- 2015. Pew Research Center.
http://www.pewinternet.org/2015/10/08/social-networking-usage-2005-2015/
• Randall Craig. 2013 How Many Social Media Sites Will Survive? Huffington Post.
Published May 21, 2013 http://www.huffingtonpost.ca/randall-craig/social-media-
sites_b_2925234.html
• Stern, Michael J., et al. Effective sampling from social media sites and search engines
for web surveys: Demographic and data quality differences in surveys of Google and
Facebook users. Social Science Computer Review (2016): 0894439316683344.
• Stern, Wolter, and Bilgen. 2013. Can We Effectively Sample from Social Media Sites
and Search Engines? Results from Two Sampling Experiments. Presented at American
Association for Public Opinion Research Annual Conference.
Footer Information Here 25
References Cont’d

26. Thank You!
Erin Fordyce: fordyce-erin@norc.org