Successfully reported this slideshow.
Your SlideShare is downloading. ×

A Checklist to Combat Cognitive Biases in Crowdsourcing

Ad

1
WIS
Web
Information
Systems
A Checklist to Combat
Cognitive Biases in Crowdsourcing
HCOMP, Nov 14-18, 2021, Virtual
Tim ...

Ad

2
WIS
Web
Information
Systems
Cognitive Biases in Crowdsourcing…
• …can cause poor data quality
– Examples: anchoring effe...

Ad

3
WIS
Web
Information
Systems
Introducing a Checklist
• Starting point: bias checklist for business decisions
• Adaptation...

Ad

Ad

Ad

Ad

Ad

Ad

Check these out next

1 of 9 Ad
1 of 9 Ad
Advertisement

More Related Content

Advertisement

A Checklist to Combat Cognitive Biases in Crowdsourcing

  1. 1. 1 WIS Web Information Systems A Checklist to Combat Cognitive Biases in Crowdsourcing HCOMP, Nov 14-18, 2021, Virtual Tim Draws1, Alisa Rieger1, Oana Inel1, Ujwal Gadiraju1, and Nava Tintarev2 t.a.draws@tudelft.nl @tmdrws https://timdraws.net 1Delft University of Technology, 2Maastricht University
  2. 2. 2 WIS Web Information Systems Cognitive Biases in Crowdsourcing… • …can cause poor data quality – Examples: anchoring effect, confirmation bias • ...may occur often but are rarely considered – Retrospective analysis • …are difficult to deal with – Many cognitive biases exist – Unclear which bias may apply where References: Eickhoff (2018); Hube, Fetahu, & Gadiraju (2019); Tversky & Kahneman (1974)
  3. 3. 3 WIS Web Information Systems Introducing a Checklist • Starting point: bias checklist for business decisions • Adaptation to fit the crowdsourcing context • Result: 12-item checklist to combat cognitive biases in crowdsourcing (incl. running example) References: Kahneman, Lovallo, & Sibony (2011)
  4. 4. 4 WIS Web Information Systems Example (3) Groupthink or Bandwagon Effect. Does my task design give crowd workers some notion of other people’s evaluation of the items they annotate? For example, crowd workers may judge products as more likely to be relevant to “paella pan” when they see that a majority of other crowd workers have judged this product as being relevant or if it has received high ratings from consumers.
  5. 5. 5 WIS Web Information Systems 12 Checklist Items 1. Self-interest Bias 2. Affect Heuristic 3. Groupthink or Bandwagon Effect 4. Salience Bias 5. Confirmation Bias 6. Availability Bias 7. Anchoring Effect 8. Halo Effect 9. Sunk Cost Fallacy 10. Overconfidence 11. Disaster Neglect 12. Loss Aversion
  6. 6. 6 WIS Web Information Systems Using the Checklist 1.Measure / assess cognitive biases 2.Mitigate cognitive biases 3.Document cognitive biases
  7. 7. 7 WIS Web Information Systems Community Updates • Online version of checklist: https://osf.io/g5b82/ • Idea: community can suggest edits 1. Go to repository, download checklist document (.md) 2. Incorporate suggested edits 3. Send edited file + description to us (email on repository) 4. Committee decides on whether/how to update checklist
  8. 8. 8 WIS Web Information Systems Discussion & Conclusion • Contributions – Checklist to combat cognitive biases in crowdsourcing – Case study – Retrospective analysis • Future work – Understanding influence of cognitive biases across crowdsourcing scenarios – Mitigation strategies – Updates to checklist • Take home: use checklist to assess, mitigate, and document cognitive biases in crowdsourcing t.a.draws@tudelft.nl @tmdrws https://timdraws.net Paper: https://timdraws.net/files/papers/A_Checklist_to_Combat_Cognitive_Biases_in_Crowdsourcing.pdf Repository: https://osf.io/rbucj/
  9. 9. 9 WIS Web Information Systems References Carsten Eickhoff. 2018. Cognitive biases in crowdsourcing. WSDM 2018 - Proceedings of the 11th ACM International Conference on Web Search and Data Mining 2018-Febua (2018), 162–170. https://doi.org/10.1145/3159652.3159654 Tim Draws, Alisa Rieger, Oana Inel, Ujwal Gadiraju, and Nava Tintarev. 2021. A Checklist to Combat Cognitive Biases in Crowdsourcing. Proceedings on the Ninth AAAI Conference on Human Computation and Crowdsourcing (2021). https://timdraws.net/files/papers/A_Checklist_to_Combat_ Cognitive_Biases_in_Crowdsourcing.pdf Christoph Hube, Besnik Fetahu, and Ujwal Gadiraju. 2019. Understanding and mitigating worker biases in the crowdsourced collection of subjective judgments. Conference on Human Factors in Computing Systems - Proceedings (2019). https://doi.org/10.1145/3290605.3300637 Daniel Kahneman, Dan Lovallo, and Olivier Sibony. 2011. Before you make that big decision... Harvard business review 89, 6 (2011). Amos Tversky and Daniel Kahneman. 1974. Judgment under Uncertainty: Heuristics and Biases. Science 185 (Sept. 1974), 1124– 1131. https://doi.org/10.1126/science.185.4157.1124

×