1.
Digital Scholar Webinar:
Recruiting Research Participants
Online Using Reddit
Raymond Luong
PhD Student
McGill University

2.
Roadmap
1. Primer on online data collection
2. Intro to r/SampleSize and best practices
3. (Brief) intro to Amazon Mechanical Turk (MTurk)
4. r/SampleSize vs. MTurk
5. Current issues and limitations
2

3.
Resources
• All references and links in this webinar here:
tinyurl.com/DSWReddit
• Slide numbers for specific questions
3

4.
Online recruitment for research
• Better demographic diversity
and data quality than usual
undergrad samples
• More flexibility (e.g., global
pandemics)
• Paid and unpaid participants
4

5.
Motivation from undergrad
• Did not have funding or
access to undergraduate pool
• Was (am) a Redditor
• Proof-of-concept
5

6.
r/SampleSize (Reddit Community)
6

7.
r/SampleSize
• Reddit community of >171,000 volunteers
• Compensation is optional and flexible (e.g., gift cards)
• Study advertisement is moderated
• Ideal for short surveys, simple experiments, course projects
7

8.
r/SampleSize Procedure
1. Create a Reddit account
(free)
2. Make study post on
r/SampleSize
3. Repeat (repost) every 24
hours as needed
9

9.
Example: Signing up on Reddit
10

10.
Example: Survey on r/SampleSize
11

11.
Example: Survey on r/SampleSize
12

12.
Example: Survey on r/SampleSize
13
Tag Study Title
Intended
Demographic Study Link

13.
Example: Survey on r/SampleSize
14

14.
Best practices on r/SampleSize
• Follow the rules closely, particularly the 24-hour rule
• Daytime posting (EST) works well
15
Also discussed for Reddit in general by Shatz (2016)

15.
Best practices on r/SampleSize
• Post studies that are short and simple (<20 minutes)
• Consider readability indices (e.g., Flesch–Kincaid) for study
descriptions, consent forms, materials
• Avoid clumping long questionnaires
16
Also discussed for Reddit in general by Shatz (2016)

16.
Best practices on r/SampleSize
• Be responsive!
17
Also discussed for Reddit in general by Shatz (2016)

17.
FYI: Other Reddit communities
• r/SampleSize is dedicated to recruitment
• Other subreddits can be used for recruitment (with
permission from moderators)
• See Shatz (2016)
18

18.
FYI: Other Reddit communities
• r/SampleSize is dedicated to recruitment
• Other subreddits can be used for recruitment with
permission from moderators
• See Shatz (2016)
19

19.
Amazon Mechanical Turk (MTurk)
20

20.
21

21.
Amazon Mechanical Turk
• General crowdsourcing
marketplace
• Workers (Turkers) complete
“Human intelligence tasks” (HITs)
• Filter workers by Qualifications
for a fee
• Useful for a variety of behavioral
experiments and surveys
22

22.
23

23.
MTurk Procedure
24
1. Create Amazon.com account
2. Purchase prepaid HIT compensation
3. Create HIT with ID link to survey/experiment
4. Review and approve/decline worker responses
5. Repeat as necessary

24.
Example: Survey on MTurk
25

25.
FYI: CloudResearch
• Formerly known as TurkPrime
• Provides paid MTurk add-ons:
1. MTurk toolkit
2. Prime Panels
3. Managed Research
4. SENTRY
• Also hosts Innovations in Online Research Conference
• See cloudresearch.com
26

26.
FYI: Alternatives to MTurk
• Prolific Academic (Prolific),
panel specifically for
academic research
• Diverse demographics and
sample characteristics
• See Peer et al. (2017) and
prolific.co
27

27.
Viability of r/SampleSize
28

28.
Previous Research
• Few previous r/SampleSize studies
• Existing studies reported more diverse ages, education,
and equal gender representation compared to traditional
samples
• Previous psychology findings replicated
• Scale reliabilities similar to undergraduate samples
29
Brickman & Silva (2017), Jamnik & Lane (2017), Luong et al. (2019), Record et al. (2018)

29.
Is r/SampleSize viable for research?
• We compared r/SampleSize
(n = 277) to MTurk (n = 256):
1. Demographics
2. Data quality
3. Psychological
characteristics
• Sensitivity: Small to medium
effects as per Cohen (1988)
unless otherwise stated
30
Luong & Lomanowska (2021)

30.
r/SampleSize Demographics
• Balanced male-female gender
distribution
• Low non-binary gender
representation
• Predominantly White and from
the USA
• Diverse income ranges and
education
• Predominantly younger ages
31
Table 1 from Luong & Lomanowska (2021)

31.
r/SampleSize Demographics
• Balanced male-female gender
distribution
• Low non-binary gender
representation
• Predominantly White and from
the USA
• Diverse income ranges and
education
• Predominantly younger ages
32
Table 1 from Luong & Lomanowska (2021)

32.
r/SampleSize Demographics
• Balanced male-female gender
distribution
• Low non-binary gender
representation
• Predominantly White and from
the USA
• Diverse income ranges and
education
• Predominantly younger ages
33
Table 1 from Luong & Lomanowska (2021)

33.
r/SampleSize Demographics
• Balanced male-female gender
distribution
• Low non-binary gender
representation
• Predominantly White and from
the USA
• Diverse income ranges and
education
• Predominantly younger ages
34
Table 1 from Luong & Lomanowska (2021)

34.
r/SampleSize Data Quality
• Socially desirable
responding: Slightly higher
for r/SampleSize than MTurk
35
Luong & Lomanowska (2021)

35.
r/SampleSize Data Quality
• Demand characteristics: No
evidence of difference
between r/SampleSize and
MTurk
36
Luong & Lomanowska (2021)

36.
r/SampleSize Data Quality
• Naivety: Mixed results,
both have relatively
high levels of scale
familiarity
37
Table 4, Luong & Lomanowska (2021)

37.
r/SampleSize Data Quality
• Attention checks: Slightly fewer failed on average for
r/SampleSize than MTurk
• Survey completion time: Turkers finished survey ~5 minutes
faster (exploratory analysis!)
• Reliability of psychological scales: No evidence of
differences, or r/SampleSize was more reliable compared to
MTurk and lab samples
38
Luong & Lomanowska (2021)

38.
Psychological Characteristics
• More intrinsically
motivated to participate
than MTurk workers
• Lower ratings of social
and economic
conservatism
39
Luong & Lomanowska (2021)

39.
Psychological Characteristics
• More intrinsically
motivated to participate
than MTurk
• Lower ratings of social
and economic
conservatism
• No evidence of
difference in altruism
• Slightly higher need for
cognition
40
Luong & Lomanowska (2021)

40.
Some caveats
• Data collection is much slower (5 days vs. 27 days)
• Intended demographics cannot be enforced on
r/SampleSize
41
Luong & Lomanowska (2021)

41.
Current Issues and Limitations
42

42.
Current issues and limitations
1. Transient sample characteristics
2. Non-naïve participants
3. Bots and random/fraudulent responses
4. Questionnaire design
5. Ethical compensation
43

43.
Transient sample characteristics
• Sample characteristics can depend on when data is
collected (e.g., Casey et al., 2017)
• Research on MTurk and r/SampleSize is always ongoing
• Report demographics and data collection date/time
44

44.
Non-Naïve Participants
• Both r/SampleSize and MTurk have issues with non-naivety
• Recommend measures on hypothesis guessing
• CloudResearch and Prolific claim greater numbers of naïve
participants
45
See Meyers et al. (2020) for a review

45.
Bots and random/fraudulent responses
• Bots fraudulently
completing surveys for
compensation
• Participants might answer
randomly or multiple
times for compensation
• Note: No incentives to
use bots on r/SampleSize
but also no built-in
safeguards
46

46.
Bots and random/fraudulent responses
• Minimum
countermeasures:
Attention checks,
survey software (e.g.,
Qualtrics), qualitative
responses
• More sophisticated:
Honeypot items,
statistical detection
• CloudResearch and
Prolific screen for
bots
47
See Howell (2019) for a primer

47.
Questionnaire design: Batches
• MTurk charges 20% extra for HITs for 10+ workers
• Common workaround: Batches with 9 workers
• See Foster (2015)
• CloudResearch automates this process
• r/SampleSize has no such functionality
48

48.
Ethical compensation
• Controversial: Compensation is up to the researcher
• Moss (2019) recommends minimum $6.00/hr
• Prolific requires minimum $6.50/hr
• My recommended guideline: Federal minimum wage in US
or better (currently $7.25/hr)
49

49.
Ethical compensation
• Better compensation also leads to better response rates
50
Also reported elsewhere, e.g., Buhrmester et al. (2011)

50.
Ethical compensation
• Issue of fairness
• Our example: Recruited MTurk and r/SampleSize
participants, provided with equivalent compensation
• Only 50% of r/SampleSize opted into the raffle
51

51.
Thank you! Questions?
52
Raymond Luong
PhD Student
McGill University
@RayLuongQuant
raymond.luong@mail.mcgill.ca
References from webinar: tinyurl.com/DSWReddit