1. 1
The Effects of Crowd Worker Biases
in Fact-Checking Tasks
FAccT ‘22 – June 21-24, 2022 – Seoul, Republic of Korea
Tim Draws1, David La Barbera2, Michael Soprano2, Kevin Roitero2,
Davide Ceolin3, Alessandro Checco4, Stefano Mizzaro2
t.a.draws@tudelft.nl
@timdr4ws
https://timdraws.net
1Delft University of Technology, 2University of Udine,
3Centrum voor Wiskunde & Informatica (CWI), 4University of Rome La Sapienza

2. 2
Photo by Jorge Franganillo on Unsplash.

3. 3
Worker Biases in Fact-Checking
Exploratory study on public data set
– 180 Politician statements with ground truth and crowd annotations (false/in-between/true)
– Per crowd worker also extra variables from a questionnaire (e.g., party affiliation)
– Goal: identify potential biases of crowd workers in fact-checking

4. 4
Hypotheses
• General worker traits
– H1a: stronger trust in politics ~ less accurate fact-checking
– H1b: stronger belief in science ~ more accurate fact-checking
– H1c: better cognitive reasoning abilities ~ more accurate fact-checking
• Cognitive biases
– H2a: workers generally overestimate truthfulness
– H2b: workers’ sympathy for the statement claimant ~ likelihood of judging ‘true’
– H2c: workers’ sympathy for the goal behind the statement ~ likelihood of judging ‘true’
– H2d: workers’ confidence in their fact-checking abilities ~ less accurate fact-checking

5. 5
Worker Biases in Fact-Checking
Exploratory study on public data set
– 180 Politician statements with ground truth and crowd annotations (false/in-between/true)
– Per crowd worker also extra variables from a questionnaire (e.g., party affiliation)
– Goal: identify potential biases of crowd workers in fact-checking
New crowdsourcing study
– Same setup as earlier study + new variables
– 2200 judgments from 302 crowd workers
– Goal: test hypotheses identified from the exploratory study

6. 6
Hypotheses
• General worker traits
– H1a: stronger trust in politics ~ less accurate fact-checking
– H1b: stronger belief in science ~ more accurate fact-checking
– H1c: better cognitive reasoning abilities ~ more accurate fact-checking
• Cognitive biases
– H2a: workers generally overestimate truthfulness
– H2b: workers’ sympathy for the statement claimant ~ likelihood of judging ‘true’
– H2c: workers’ sympathy for the goal behind the statement ~ likelihood of judging ‘true’
– H2d: workers’ confidence in their fact-checking abilities ~ less accurate fact-checking

7. 7
Summary
• Findings
– Workers generally overestimate the truthfulness of politician statements
– Belief in science may be associated with less accurate fact-checking
– Cognitive biases of crowd workers such as the affect heuristic and
overconfidence can lead to less accurate judgments
• Repository: https://osf.io/8yu5z/
t.a.draws@tudelft.nl
@timdr4ws
https://timdraws.net