SlideShare a Scribd company logo
1
WIS
Web
Information
Systems
Introducing the
Cognitive-Biases-in-Crowdsourcing
Checklist
CSCW2021Workshop–InvestigatingandMitigatingBiasesinCrowdsourcedData,October23,2021,Virtual
Tim Draws1, Alisa Rieger1, Oana Inel1, Ujwal Gadiraju1, and Nava Tintarev2
t.a.draws@tudelft.nl
@tmdrws
https://timdraws.net
1Delft University of Technology, 2Maastricht University
2
WIS
Web
Information
Systems
Cognitive Biases in Crowdsourcing
• Cognitive biases of crowdworkers can negatively
affect annotation quality
– Anchoring effect
– Confirmation bias
• Combating cognitive biases is tricky
– Many cognitive biases exist
– Unclear which bias may apply where
References: Eickhoff (2018); Hube, Fetahu, & Gadiraju (2019); Tversky & Kahneman (1974)
3
WIS
Web
Information
Systems
Introducing a Checklist
• Starting point: bias checklist for business decisions
• Adaptation to fit the crowdsourcing context
• Result: 12-item checklist to combat cognitive biases
in crowdsourcing (incl. running example)
References: Kahneman, Lovallo, & Sibony (2011)
4
WIS
Web
Information
Systems
Example
(3) Groupthink or Bandwagon Effect. Does my task design give crowd
workers some notion of other people’s evaluation of the items they annotate?
For example, crowd workers may judge products as more likely to be relevant to
“paella pan” when they see that a majority of other crowd workers have judged
this product as being relevant or if it has received high ratings from consumers.
5
WIS
Web
Information
Systems
Using the Checklist
1.Measure / assess cognitive biases
2.Mitigate cognitive biases
3.Document cognitive biases
6
WIS
Web
Information
Systems
Discussion & Conclusion
• Covering all different types of biases in crowdsourcing
• Updated version of checklist available on repository (link below)
• HCOMP paper: case study + retrospective analysis
t.a.draws@tudelft.nl
@tmdrws
https://timdraws.net
Paper: https://timdraws.net/files/papers/A_Checklist_to_Combat_Cognitive_Biases_in_Crowdsourcing.pdf
Preregistration and supplementary material: https://osf.io/rbucj/
7
WIS
Web
Information
Systems
References
Carsten Eickhoff. 2018. Cognitive biases in crowdsourcing. WSDM 2018 - Proceedings of the 11th ACM International Conference on
Web Search and Data Mining 2018-Febua (2018), 162–170. https://doi.org/10.1145/3159652.3159654
Tim Draws, Alisa Rieger, Oana Inel, Ujwal Gadiraju, and Nava Tintarev. 2021. A Checklist to Combat Cognitive Biases in
Crowdsourcing. Proceedings on the Ninth AAAI Conference on Human Computation and Crowdsourcing (2021).
https://timdraws.net/files/papers/A_Checklist_to_Combat_ Cognitive_Biases_in_Crowdsourcing.pdf
Christoph Hube, Besnik Fetahu, and Ujwal Gadiraju. 2019. Understanding and mitigating worker biases in the crowdsourced collection
of subjective judgments. Conference on Human Factors in Computing Systems - Proceedings (2019).
https://doi.org/10.1145/3290605.3300637
Daniel Kahneman, Dan Lovallo, and Olivier Sibony. 2011. Before you make that big decision... Harvard business review 89, 6 (2011).
Amos Tversky and Daniel Kahneman. 1974. Judgment under Uncertainty: Heuristics and Biases. Science 185 (Sept. 1974), 1124–
1131. https://doi.org/10.1126/science.185.4157.1124

More Related Content

What's hot

Code4 lib2012
Code4 lib2012Code4 lib2012
Code4 lib2012
William Gunn
 
The Analytics and Data Science Landscape
The Analytics and Data Science LandscapeThe Analytics and Data Science Landscape
The Analytics and Data Science Landscape
Philip Bourne
 
tools for communicating in the computational sciences
tools for communicating in the computational sciencestools for communicating in the computational sciences
tools for communicating in the computational sciences
Brian Bot
 
How can machine learning and AI in the cloud improve research?
How can machine learning and AI in the cloud improve research?How can machine learning and AI in the cloud improve research?
How can machine learning and AI in the cloud improve research?
Jisc
 
The Future of FAIR Data: An international social, legal and technological inf...
The Future of FAIR Data: An international social, legal and technological inf...The Future of FAIR Data: An international social, legal and technological inf...
The Future of FAIR Data: An international social, legal and technological inf...
Michel Dumontier
 
Keynote Talk - Gaining Powerful Insights into Social Media Listening
Keynote Talk - Gaining Powerful Insights into Social Media ListeningKeynote Talk - Gaining Powerful Insights into Social Media Listening
Keynote Talk - Gaining Powerful Insights into Social Media Listening
Dr Wasim Ahmed
 
LISA VII: The Scientific and Technical Foundation for Altmetrics in the Unite...
LISA VII: The Scientific and Technical Foundation for Altmetrics in the Unite...LISA VII: The Scientific and Technical Foundation for Altmetrics in the Unite...
LISA VII: The Scientific and Technical Foundation for Altmetrics in the Unite...
William Gunn
 
State of Florida Neo4J Graph Briefing - Keynote
State of Florida Neo4J Graph Briefing - KeynoteState of Florida Neo4J Graph Briefing - Keynote
State of Florida Neo4J Graph Briefing - Keynote
Neo4j
 
Big Data Brown Bag
Big Data Brown BagBig Data Brown Bag
Big Data Brown Bag
usmanqureshi
 
Towards an integrated research management digital ecosystem
Towards an integrated research management digital ecosystemTowards an integrated research management digital ecosystem
Towards an integrated research management digital ecosystem
Jisc
 
Getting (and giving) credit for all that we do
Getting (and giving) credit for all that we doGetting (and giving) credit for all that we do
Getting (and giving) credit for all that we do
mhaendel
 
infrastructure for communicating data-intensive science
infrastructure for communicating data-intensive scienceinfrastructure for communicating data-intensive science
infrastructure for communicating data-intensive science
Brian Bot
 
20160414 23 Research Data Things
20160414 23 Research Data Things20160414 23 Research Data Things
20160414 23 Research Data Things
Katina Toufexis
 
Big Data as a Catalyst for Collaboration & Innovation
Big Data as a Catalyst for Collaboration & InnovationBig Data as a Catalyst for Collaboration & Innovation
Big Data as a Catalyst for Collaboration & Innovation
Philip Bourne
 
Trust threads: Provenance for Data Reuse in Long Tail Science
Trust threads: Provenance for Data Reuse in Long Tail ScienceTrust threads: Provenance for Data Reuse in Long Tail Science
Trust threads: Provenance for Data Reuse in Long Tail Science
Beth Plale
 
Towards a Data Commons
Towards a Data CommonsTowards a Data Commons
Towards a Data Commons
Michael Becich
 
SGCI Science Gateways: Ushering in a New Era of Sustainability
SGCI Science Gateways: Ushering in a New Era of Sustainability SGCI Science Gateways: Ushering in a New Era of Sustainability
SGCI Science Gateways: Ushering in a New Era of Sustainability
Sandra Gesing
 
Noshir Contractor's view on the future of Linked Data
Noshir Contractor's view on the future of Linked DataNoshir Contractor's view on the future of Linked Data
Noshir Contractor's view on the future of Linked Data
Carlos Pedrinaci
 
20160301 23 Research Data Things
20160301 23 Research Data Things20160301 23 Research Data Things
20160301 23 Research Data Things
Katina Toufexis
 
VALA14_burrows
VALA14_burrowsVALA14_burrows
VALA14_burrows
Deb Verhoeven
 

What's hot (20)

Code4 lib2012
Code4 lib2012Code4 lib2012
Code4 lib2012
 
The Analytics and Data Science Landscape
The Analytics and Data Science LandscapeThe Analytics and Data Science Landscape
The Analytics and Data Science Landscape
 
tools for communicating in the computational sciences
tools for communicating in the computational sciencestools for communicating in the computational sciences
tools for communicating in the computational sciences
 
How can machine learning and AI in the cloud improve research?
How can machine learning and AI in the cloud improve research?How can machine learning and AI in the cloud improve research?
How can machine learning and AI in the cloud improve research?
 
The Future of FAIR Data: An international social, legal and technological inf...
The Future of FAIR Data: An international social, legal and technological inf...The Future of FAIR Data: An international social, legal and technological inf...
The Future of FAIR Data: An international social, legal and technological inf...
 
Keynote Talk - Gaining Powerful Insights into Social Media Listening
Keynote Talk - Gaining Powerful Insights into Social Media ListeningKeynote Talk - Gaining Powerful Insights into Social Media Listening
Keynote Talk - Gaining Powerful Insights into Social Media Listening
 
LISA VII: The Scientific and Technical Foundation for Altmetrics in the Unite...
LISA VII: The Scientific and Technical Foundation for Altmetrics in the Unite...LISA VII: The Scientific and Technical Foundation for Altmetrics in the Unite...
LISA VII: The Scientific and Technical Foundation for Altmetrics in the Unite...
 
State of Florida Neo4J Graph Briefing - Keynote
State of Florida Neo4J Graph Briefing - KeynoteState of Florida Neo4J Graph Briefing - Keynote
State of Florida Neo4J Graph Briefing - Keynote
 
Big Data Brown Bag
Big Data Brown BagBig Data Brown Bag
Big Data Brown Bag
 
Towards an integrated research management digital ecosystem
Towards an integrated research management digital ecosystemTowards an integrated research management digital ecosystem
Towards an integrated research management digital ecosystem
 
Getting (and giving) credit for all that we do
Getting (and giving) credit for all that we doGetting (and giving) credit for all that we do
Getting (and giving) credit for all that we do
 
infrastructure for communicating data-intensive science
infrastructure for communicating data-intensive scienceinfrastructure for communicating data-intensive science
infrastructure for communicating data-intensive science
 
20160414 23 Research Data Things
20160414 23 Research Data Things20160414 23 Research Data Things
20160414 23 Research Data Things
 
Big Data as a Catalyst for Collaboration & Innovation
Big Data as a Catalyst for Collaboration & InnovationBig Data as a Catalyst for Collaboration & Innovation
Big Data as a Catalyst for Collaboration & Innovation
 
Trust threads: Provenance for Data Reuse in Long Tail Science
Trust threads: Provenance for Data Reuse in Long Tail ScienceTrust threads: Provenance for Data Reuse in Long Tail Science
Trust threads: Provenance for Data Reuse in Long Tail Science
 
Towards a Data Commons
Towards a Data CommonsTowards a Data Commons
Towards a Data Commons
 
SGCI Science Gateways: Ushering in a New Era of Sustainability
SGCI Science Gateways: Ushering in a New Era of Sustainability SGCI Science Gateways: Ushering in a New Era of Sustainability
SGCI Science Gateways: Ushering in a New Era of Sustainability
 
Noshir Contractor's view on the future of Linked Data
Noshir Contractor's view on the future of Linked DataNoshir Contractor's view on the future of Linked Data
Noshir Contractor's view on the future of Linked Data
 
20160301 23 Research Data Things
20160301 23 Research Data Things20160301 23 Research Data Things
20160301 23 Research Data Things
 
VALA14_burrows
VALA14_burrowsVALA14_burrows
VALA14_burrows
 

Similar to Introducing the Cognitive-Biases-in-Crowdsourcing Checklist

The Internet of Things: What's next?
The Internet of Things: What's next? The Internet of Things: What's next?
The Internet of Things: What's next?
PayamBarnaghi
 
Opportunities and methodological challenges of Big Data for official statist...
Opportunities and methodological challenges of  Big Data for official statist...Opportunities and methodological challenges of  Big Data for official statist...
Opportunities and methodological challenges of Big Data for official statist...
Piet J.H. Daas
 
Service-oriented Cognitive Analytics in Smart Service Systems
Service-oriented Cognitive Analytics in Smart Service SystemsService-oriented Cognitive Analytics in Smart Service Systems
Service-oriented Cognitive Analytics in Smart Service Systems
Dr.-Ing. Robin Hirt
 
Distributed Trust Architecture: The New Foundation of Everything
Distributed Trust Architecture: The New Foundation of EverythingDistributed Trust Architecture: The New Foundation of Everything
Distributed Trust Architecture: The New Foundation of Everything
Liming Zhu
 
KIT-601-L-UNIT-1 (Revised) Introduction to Data Analytcs.pdf
KIT-601-L-UNIT-1 (Revised) Introduction to Data Analytcs.pdfKIT-601-L-UNIT-1 (Revised) Introduction to Data Analytcs.pdf
KIT-601-L-UNIT-1 (Revised) Introduction to Data Analytcs.pdf
Dr. Radhey Shyam
 
Introduction to Data Analytics and data analytics life cycle
Introduction to Data Analytics and data analytics life cycleIntroduction to Data Analytics and data analytics life cycle
Introduction to Data Analytics and data analytics life cycle
Dr. Radhey Shyam
 
wireless sensor network
wireless sensor networkwireless sensor network
wireless sensor network
parry prabhu
 
The Science of Data Science
The Science of Data Science The Science of Data Science
The Science of Data Science
James Hendler
 
Distributed Trust Architecture: The New Reality of ML-based Systems
Distributed Trust Architecture: The New Reality of ML-based SystemsDistributed Trust Architecture: The New Reality of ML-based Systems
Distributed Trust Architecture: The New Reality of ML-based Systems
Liming Zhu
 
Collective Intelligence and Online Deliberation Platforms for Citizen Engagem...
Collective Intelligence and Online Deliberation Platforms for Citizen Engagem...Collective Intelligence and Online Deliberation Platforms for Citizen Engagem...
Collective Intelligence and Online Deliberation Platforms for Citizen Engagem...
Anna De Liddo
 
Dynamic Data Analytics for the Internet of Things: Challenges and Opportunities
Dynamic Data Analytics for the Internet of Things: Challenges and OpportunitiesDynamic Data Analytics for the Internet of Things: Challenges and Opportunities
Dynamic Data Analytics for the Internet of Things: Challenges and Opportunities
PayamBarnaghi
 
Search, Discovery and Analysis of Sensory Data Streams
Search, Discovery and Analysis of Sensory Data StreamsSearch, Discovery and Analysis of Sensory Data Streams
Search, Discovery and Analysis of Sensory Data Streams
PayamBarnaghi
 
BIMCV, Banco de Imagen Medica de la Comunidad Valenciana. María de la Iglesia
BIMCV, Banco de Imagen Medica de la Comunidad Valenciana. María de la IglesiaBIMCV, Banco de Imagen Medica de la Comunidad Valenciana. María de la Iglesia
BIMCV, Banco de Imagen Medica de la Comunidad Valenciana. María de la Iglesia
Maria de la Iglesia
 
Responsible AI & Cybersecurity: A tale of two technology risks
Responsible AI & Cybersecurity: A tale of two technology risksResponsible AI & Cybersecurity: A tale of two technology risks
Responsible AI & Cybersecurity: A tale of two technology risks
Liming Zhu
 
Bigger and Better: Employing a Holistic Strategy for Big Data toward a Strong...
Bigger and Better: Employing a Holistic Strategy for Big Data toward a Strong...Bigger and Better: Employing a Holistic Strategy for Big Data toward a Strong...
Bigger and Better: Employing a Holistic Strategy for Big Data toward a Strong...
IT Network marcus evans
 
Information entanglement
Information entanglementInformation entanglement
Information entanglement
Willard Van De Bogart
 
CI_for_NA
CI_for_NACI_for_NA
CI_for_NA
webuploader
 
Big data: Challenges, Practices and Technologies
Big data: Challenges, Practices and TechnologiesBig data: Challenges, Practices and Technologies
Big data: Challenges, Practices and Technologies
Navneet Randhawa
 
Optimising Scientific Knowledge Transfer: How Collective Sensemaking Can Ena...
Optimising Scientific Knowledge Transfer: How Collective Sensemaking Can Ena...Optimising Scientific Knowledge Transfer: How Collective Sensemaking Can Ena...
Optimising Scientific Knowledge Transfer: How Collective Sensemaking Can Ena...
Anita de Waard
 
Beyond Meta-Data: Nano-Publications Recording Scientific Endeavour
Beyond Meta-Data: Nano-Publications Recording Scientific EndeavourBeyond Meta-Data: Nano-Publications Recording Scientific Endeavour
Beyond Meta-Data: Nano-Publications Recording Scientific Endeavour
KNOWeSCAPE2014
 

Similar to Introducing the Cognitive-Biases-in-Crowdsourcing Checklist (20)

The Internet of Things: What's next?
The Internet of Things: What's next? The Internet of Things: What's next?
The Internet of Things: What's next?
 
Opportunities and methodological challenges of Big Data for official statist...
Opportunities and methodological challenges of  Big Data for official statist...Opportunities and methodological challenges of  Big Data for official statist...
Opportunities and methodological challenges of Big Data for official statist...
 
Service-oriented Cognitive Analytics in Smart Service Systems
Service-oriented Cognitive Analytics in Smart Service SystemsService-oriented Cognitive Analytics in Smart Service Systems
Service-oriented Cognitive Analytics in Smart Service Systems
 
Distributed Trust Architecture: The New Foundation of Everything
Distributed Trust Architecture: The New Foundation of EverythingDistributed Trust Architecture: The New Foundation of Everything
Distributed Trust Architecture: The New Foundation of Everything
 
KIT-601-L-UNIT-1 (Revised) Introduction to Data Analytcs.pdf
KIT-601-L-UNIT-1 (Revised) Introduction to Data Analytcs.pdfKIT-601-L-UNIT-1 (Revised) Introduction to Data Analytcs.pdf
KIT-601-L-UNIT-1 (Revised) Introduction to Data Analytcs.pdf
 
Introduction to Data Analytics and data analytics life cycle
Introduction to Data Analytics and data analytics life cycleIntroduction to Data Analytics and data analytics life cycle
Introduction to Data Analytics and data analytics life cycle
 
wireless sensor network
wireless sensor networkwireless sensor network
wireless sensor network
 
The Science of Data Science
The Science of Data Science The Science of Data Science
The Science of Data Science
 
Distributed Trust Architecture: The New Reality of ML-based Systems
Distributed Trust Architecture: The New Reality of ML-based SystemsDistributed Trust Architecture: The New Reality of ML-based Systems
Distributed Trust Architecture: The New Reality of ML-based Systems
 
Collective Intelligence and Online Deliberation Platforms for Citizen Engagem...
Collective Intelligence and Online Deliberation Platforms for Citizen Engagem...Collective Intelligence and Online Deliberation Platforms for Citizen Engagem...
Collective Intelligence and Online Deliberation Platforms for Citizen Engagem...
 
Dynamic Data Analytics for the Internet of Things: Challenges and Opportunities
Dynamic Data Analytics for the Internet of Things: Challenges and OpportunitiesDynamic Data Analytics for the Internet of Things: Challenges and Opportunities
Dynamic Data Analytics for the Internet of Things: Challenges and Opportunities
 
Search, Discovery and Analysis of Sensory Data Streams
Search, Discovery and Analysis of Sensory Data StreamsSearch, Discovery and Analysis of Sensory Data Streams
Search, Discovery and Analysis of Sensory Data Streams
 
BIMCV, Banco de Imagen Medica de la Comunidad Valenciana. María de la Iglesia
BIMCV, Banco de Imagen Medica de la Comunidad Valenciana. María de la IglesiaBIMCV, Banco de Imagen Medica de la Comunidad Valenciana. María de la Iglesia
BIMCV, Banco de Imagen Medica de la Comunidad Valenciana. María de la Iglesia
 
Responsible AI & Cybersecurity: A tale of two technology risks
Responsible AI & Cybersecurity: A tale of two technology risksResponsible AI & Cybersecurity: A tale of two technology risks
Responsible AI & Cybersecurity: A tale of two technology risks
 
Bigger and Better: Employing a Holistic Strategy for Big Data toward a Strong...
Bigger and Better: Employing a Holistic Strategy for Big Data toward a Strong...Bigger and Better: Employing a Holistic Strategy for Big Data toward a Strong...
Bigger and Better: Employing a Holistic Strategy for Big Data toward a Strong...
 
Information entanglement
Information entanglementInformation entanglement
Information entanglement
 
CI_for_NA
CI_for_NACI_for_NA
CI_for_NA
 
Big data: Challenges, Practices and Technologies
Big data: Challenges, Practices and TechnologiesBig data: Challenges, Practices and Technologies
Big data: Challenges, Practices and Technologies
 
Optimising Scientific Knowledge Transfer: How Collective Sensemaking Can Ena...
Optimising Scientific Knowledge Transfer: How Collective Sensemaking Can Ena...Optimising Scientific Knowledge Transfer: How Collective Sensemaking Can Ena...
Optimising Scientific Knowledge Transfer: How Collective Sensemaking Can Ena...
 
Beyond Meta-Data: Nano-Publications Recording Scientific Endeavour
Beyond Meta-Data: Nano-Publications Recording Scientific EndeavourBeyond Meta-Data: Nano-Publications Recording Scientific Endeavour
Beyond Meta-Data: Nano-Publications Recording Scientific Endeavour
 

Recently uploaded

Rice Genome Project a complete saga .(1).pptx
Rice Genome  Project a complete saga .(1).pptxRice Genome  Project a complete saga .(1).pptx
Rice Genome Project a complete saga .(1).pptx
SoumyaDixit11
 
Surface properties of the seas of Titan as revealed by Cassini mission bistat...
Surface properties of the seas of Titan as revealed by Cassini mission bistat...Surface properties of the seas of Titan as revealed by Cassini mission bistat...
Surface properties of the seas of Titan as revealed by Cassini mission bistat...
Sérgio Sacani
 
Synopsis: Analysis of a Metallic Specimen
Synopsis: Analysis of a Metallic SpecimenSynopsis: Analysis of a Metallic Specimen
Synopsis: Analysis of a Metallic Specimen
Sérgio Sacani
 
17. 20240529_Ingrid Olesen_MariGreen summer school.pdf
17. 20240529_Ingrid Olesen_MariGreen summer school.pdf17. 20240529_Ingrid Olesen_MariGreen summer school.pdf
17. 20240529_Ingrid Olesen_MariGreen summer school.pdf
marigreenproject
 
Lake classification and Morphometry.pptx
Lake classification and Morphometry.pptxLake classification and Morphometry.pptx
Lake classification and Morphometry.pptx
boobalanbfsc
 
16. 20240529_Ailin Molosag_MARIGREEN_SS_Day3_Ailin.pdf
16. 20240529_Ailin Molosag_MARIGREEN_SS_Day3_Ailin.pdf16. 20240529_Ailin Molosag_MARIGREEN_SS_Day3_Ailin.pdf
16. 20240529_Ailin Molosag_MARIGREEN_SS_Day3_Ailin.pdf
marigreenproject
 
How Does Simulation-Based Testing for Self-Driving Cars Match Human Perception?
How Does Simulation-Based Testing for Self-Driving Cars Match Human Perception?How Does Simulation-Based Testing for Self-Driving Cars Match Human Perception?
How Does Simulation-Based Testing for Self-Driving Cars Match Human Perception?
Christian Birchler
 
Concept of Balanced Diet & Nutrients.pdf
Concept of Balanced Diet & Nutrients.pdfConcept of Balanced Diet & Nutrients.pdf
Concept of Balanced Diet & Nutrients.pdf
SELF-EXPLANATORY
 
Accessing Data to Support Pesticide Residue and Emerging Contaminant Analysis...
Accessing Data to Support Pesticide Residue and Emerging Contaminant Analysis...Accessing Data to Support Pesticide Residue and Emerging Contaminant Analysis...
Accessing Data to Support Pesticide Residue and Emerging Contaminant Analysis...
US Environmental Protection Agency (EPA), Center for Computational Toxicology and Exposure
 
All-domain Anomaly Resolution Office Supplement to Oak Ridge National Laborat...
All-domain Anomaly Resolution Office Supplement to Oak Ridge National Laborat...All-domain Anomaly Resolution Office Supplement to Oak Ridge National Laborat...
All-domain Anomaly Resolution Office Supplement to Oak Ridge National Laborat...
Sérgio Sacani
 
Complementary interstellar detections from the heliotail
Complementary interstellar detections from the heliotailComplementary interstellar detections from the heliotail
Complementary interstellar detections from the heliotail
Sérgio Sacani
 
Animal biotechnology Cattle breeds....pptx
Animal biotechnology Cattle breeds....pptxAnimal biotechnology Cattle breeds....pptx
Animal biotechnology Cattle breeds....pptx
Ritesh Chavan
 
Traditional, current and future use of fish and seaweed for fertilisation - ...
Traditional, current and future use of fish and seaweed for fertilisation -  ...Traditional, current and future use of fish and seaweed for fertilisation -  ...
Traditional, current and future use of fish and seaweed for fertilisation - ...
Faculty of Applied Chemistry and Materials Science
 
The CGIAR needs a revolution John McIntire a, Achim Dobermann b
The CGIAR needs a revolution John McIntire a, Achim Dobermann bThe CGIAR needs a revolution John McIntire a, Achim Dobermann b
The CGIAR needs a revolution John McIntire a, Achim Dobermann b
Abdellah HAMMA
 
MACRAMÉ ChIPs @Behoerdenklausur 2024 (Berlin)
MACRAMÉ ChIPs @Behoerdenklausur 2024 (Berlin)MACRAMÉ ChIPs @Behoerdenklausur 2024 (Berlin)
MACRAMÉ ChIPs @Behoerdenklausur 2024 (Berlin)
Steffi Friedrichs
 
Bacteria l classification(sperical).pptx
Bacteria l classification(sperical).pptxBacteria l classification(sperical).pptx
Bacteria l classification(sperical).pptx
shubhamve111yadav
 
Rapid pulse drying of marine biomasses - Sigurd Sannan
Rapid pulse drying of marine biomasses - Sigurd SannanRapid pulse drying of marine biomasses - Sigurd Sannan
Rapid pulse drying of marine biomasses - Sigurd Sannan
Faculty of Applied Chemistry and Materials Science
 
GENE THERAPY [Autosaved].pptx A brief explanation about Gene Therapy
GENE THERAPY [Autosaved].pptx A brief explanation about Gene TherapyGENE THERAPY [Autosaved].pptx A brief explanation about Gene Therapy
GENE THERAPY [Autosaved].pptx A brief explanation about Gene Therapy
shubhamve111yadav
 
A NICER VIEW OF THE NEAREST AND BRIGHTEST MILLISECOND PULSAR: PSR J0437−4715
A NICER VIEW OF THE NEAREST AND BRIGHTEST MILLISECOND PULSAR: PSR J0437−4715A NICER VIEW OF THE NEAREST AND BRIGHTEST MILLISECOND PULSAR: PSR J0437−4715
A NICER VIEW OF THE NEAREST AND BRIGHTEST MILLISECOND PULSAR: PSR J0437−4715
Sérgio Sacani
 
AlgaeBrew project - Unlocking the potential of microalgae for the valorisatio...
AlgaeBrew project - Unlocking the potential of microalgae for the valorisatio...AlgaeBrew project - Unlocking the potential of microalgae for the valorisatio...
AlgaeBrew project - Unlocking the potential of microalgae for the valorisatio...
Faculty of Applied Chemistry and Materials Science
 

Recently uploaded (20)

Rice Genome Project a complete saga .(1).pptx
Rice Genome  Project a complete saga .(1).pptxRice Genome  Project a complete saga .(1).pptx
Rice Genome Project a complete saga .(1).pptx
 
Surface properties of the seas of Titan as revealed by Cassini mission bistat...
Surface properties of the seas of Titan as revealed by Cassini mission bistat...Surface properties of the seas of Titan as revealed by Cassini mission bistat...
Surface properties of the seas of Titan as revealed by Cassini mission bistat...
 
Synopsis: Analysis of a Metallic Specimen
Synopsis: Analysis of a Metallic SpecimenSynopsis: Analysis of a Metallic Specimen
Synopsis: Analysis of a Metallic Specimen
 
17. 20240529_Ingrid Olesen_MariGreen summer school.pdf
17. 20240529_Ingrid Olesen_MariGreen summer school.pdf17. 20240529_Ingrid Olesen_MariGreen summer school.pdf
17. 20240529_Ingrid Olesen_MariGreen summer school.pdf
 
Lake classification and Morphometry.pptx
Lake classification and Morphometry.pptxLake classification and Morphometry.pptx
Lake classification and Morphometry.pptx
 
16. 20240529_Ailin Molosag_MARIGREEN_SS_Day3_Ailin.pdf
16. 20240529_Ailin Molosag_MARIGREEN_SS_Day3_Ailin.pdf16. 20240529_Ailin Molosag_MARIGREEN_SS_Day3_Ailin.pdf
16. 20240529_Ailin Molosag_MARIGREEN_SS_Day3_Ailin.pdf
 
How Does Simulation-Based Testing for Self-Driving Cars Match Human Perception?
How Does Simulation-Based Testing for Self-Driving Cars Match Human Perception?How Does Simulation-Based Testing for Self-Driving Cars Match Human Perception?
How Does Simulation-Based Testing for Self-Driving Cars Match Human Perception?
 
Concept of Balanced Diet & Nutrients.pdf
Concept of Balanced Diet & Nutrients.pdfConcept of Balanced Diet & Nutrients.pdf
Concept of Balanced Diet & Nutrients.pdf
 
Accessing Data to Support Pesticide Residue and Emerging Contaminant Analysis...
Accessing Data to Support Pesticide Residue and Emerging Contaminant Analysis...Accessing Data to Support Pesticide Residue and Emerging Contaminant Analysis...
Accessing Data to Support Pesticide Residue and Emerging Contaminant Analysis...
 
All-domain Anomaly Resolution Office Supplement to Oak Ridge National Laborat...
All-domain Anomaly Resolution Office Supplement to Oak Ridge National Laborat...All-domain Anomaly Resolution Office Supplement to Oak Ridge National Laborat...
All-domain Anomaly Resolution Office Supplement to Oak Ridge National Laborat...
 
Complementary interstellar detections from the heliotail
Complementary interstellar detections from the heliotailComplementary interstellar detections from the heliotail
Complementary interstellar detections from the heliotail
 
Animal biotechnology Cattle breeds....pptx
Animal biotechnology Cattle breeds....pptxAnimal biotechnology Cattle breeds....pptx
Animal biotechnology Cattle breeds....pptx
 
Traditional, current and future use of fish and seaweed for fertilisation - ...
Traditional, current and future use of fish and seaweed for fertilisation -  ...Traditional, current and future use of fish and seaweed for fertilisation -  ...
Traditional, current and future use of fish and seaweed for fertilisation - ...
 
The CGIAR needs a revolution John McIntire a, Achim Dobermann b
The CGIAR needs a revolution John McIntire a, Achim Dobermann bThe CGIAR needs a revolution John McIntire a, Achim Dobermann b
The CGIAR needs a revolution John McIntire a, Achim Dobermann b
 
MACRAMÉ ChIPs @Behoerdenklausur 2024 (Berlin)
MACRAMÉ ChIPs @Behoerdenklausur 2024 (Berlin)MACRAMÉ ChIPs @Behoerdenklausur 2024 (Berlin)
MACRAMÉ ChIPs @Behoerdenklausur 2024 (Berlin)
 
Bacteria l classification(sperical).pptx
Bacteria l classification(sperical).pptxBacteria l classification(sperical).pptx
Bacteria l classification(sperical).pptx
 
Rapid pulse drying of marine biomasses - Sigurd Sannan
Rapid pulse drying of marine biomasses - Sigurd SannanRapid pulse drying of marine biomasses - Sigurd Sannan
Rapid pulse drying of marine biomasses - Sigurd Sannan
 
GENE THERAPY [Autosaved].pptx A brief explanation about Gene Therapy
GENE THERAPY [Autosaved].pptx A brief explanation about Gene TherapyGENE THERAPY [Autosaved].pptx A brief explanation about Gene Therapy
GENE THERAPY [Autosaved].pptx A brief explanation about Gene Therapy
 
A NICER VIEW OF THE NEAREST AND BRIGHTEST MILLISECOND PULSAR: PSR J0437−4715
A NICER VIEW OF THE NEAREST AND BRIGHTEST MILLISECOND PULSAR: PSR J0437−4715A NICER VIEW OF THE NEAREST AND BRIGHTEST MILLISECOND PULSAR: PSR J0437−4715
A NICER VIEW OF THE NEAREST AND BRIGHTEST MILLISECOND PULSAR: PSR J0437−4715
 
AlgaeBrew project - Unlocking the potential of microalgae for the valorisatio...
AlgaeBrew project - Unlocking the potential of microalgae for the valorisatio...AlgaeBrew project - Unlocking the potential of microalgae for the valorisatio...
AlgaeBrew project - Unlocking the potential of microalgae for the valorisatio...
 

Introducing the Cognitive-Biases-in-Crowdsourcing Checklist

  • 1. 1 WIS Web Information Systems Introducing the Cognitive-Biases-in-Crowdsourcing Checklist CSCW2021Workshop–InvestigatingandMitigatingBiasesinCrowdsourcedData,October23,2021,Virtual Tim Draws1, Alisa Rieger1, Oana Inel1, Ujwal Gadiraju1, and Nava Tintarev2 t.a.draws@tudelft.nl @tmdrws https://timdraws.net 1Delft University of Technology, 2Maastricht University
  • 2. 2 WIS Web Information Systems Cognitive Biases in Crowdsourcing • Cognitive biases of crowdworkers can negatively affect annotation quality – Anchoring effect – Confirmation bias • Combating cognitive biases is tricky – Many cognitive biases exist – Unclear which bias may apply where References: Eickhoff (2018); Hube, Fetahu, & Gadiraju (2019); Tversky & Kahneman (1974)
  • 3. 3 WIS Web Information Systems Introducing a Checklist • Starting point: bias checklist for business decisions • Adaptation to fit the crowdsourcing context • Result: 12-item checklist to combat cognitive biases in crowdsourcing (incl. running example) References: Kahneman, Lovallo, & Sibony (2011)
  • 4. 4 WIS Web Information Systems Example (3) Groupthink or Bandwagon Effect. Does my task design give crowd workers some notion of other people’s evaluation of the items they annotate? For example, crowd workers may judge products as more likely to be relevant to “paella pan” when they see that a majority of other crowd workers have judged this product as being relevant or if it has received high ratings from consumers.
  • 5. 5 WIS Web Information Systems Using the Checklist 1.Measure / assess cognitive biases 2.Mitigate cognitive biases 3.Document cognitive biases
  • 6. 6 WIS Web Information Systems Discussion & Conclusion • Covering all different types of biases in crowdsourcing • Updated version of checklist available on repository (link below) • HCOMP paper: case study + retrospective analysis t.a.draws@tudelft.nl @tmdrws https://timdraws.net Paper: https://timdraws.net/files/papers/A_Checklist_to_Combat_Cognitive_Biases_in_Crowdsourcing.pdf Preregistration and supplementary material: https://osf.io/rbucj/
  • 7. 7 WIS Web Information Systems References Carsten Eickhoff. 2018. Cognitive biases in crowdsourcing. WSDM 2018 - Proceedings of the 11th ACM International Conference on Web Search and Data Mining 2018-Febua (2018), 162–170. https://doi.org/10.1145/3159652.3159654 Tim Draws, Alisa Rieger, Oana Inel, Ujwal Gadiraju, and Nava Tintarev. 2021. A Checklist to Combat Cognitive Biases in Crowdsourcing. Proceedings on the Ninth AAAI Conference on Human Computation and Crowdsourcing (2021). https://timdraws.net/files/papers/A_Checklist_to_Combat_ Cognitive_Biases_in_Crowdsourcing.pdf Christoph Hube, Besnik Fetahu, and Ujwal Gadiraju. 2019. Understanding and mitigating worker biases in the crowdsourced collection of subjective judgments. Conference on Human Factors in Computing Systems - Proceedings (2019). https://doi.org/10.1145/3290605.3300637 Daniel Kahneman, Dan Lovallo, and Olivier Sibony. 2011. Before you make that big decision... Harvard business review 89, 6 (2011). Amos Tversky and Daniel Kahneman. 1974. Judgment under Uncertainty: Heuristics and Biases. Science 185 (Sept. 1974), 1124– 1131. https://doi.org/10.1126/science.185.4157.1124

Editor's Notes

  1. 1. Cognitive biases of crowd workers are an impactful but often neglected type of systemic bias that can reduce the quality of crowdsourced data labels. 2. These cognitive biases are general human tendencies towards irrational decision-making that often occur subconsciously. 3. For example, crowd workers may fall prey to the *anchoring effect* when they are overly influenced by information they encounter first or the *confirmation bias* when judging in line with some (false) preexisting beliefs. 4. Previous work that a plurality of cognitive biases can occur in different types of crowdsourcing tasks. 5. So why do requesters rarely consider the influence of cognitive biases in the tasks they design? 6. One important reason here may be that combating cognitive biases is simply tricky. 7. A large number of different cognitive biases have been identified in psychological literature and may often be unclear which specific cognitive bias may occur in a given crowdsourcing task. To efficiently document, assess, and mitigate cognitive biases in crowdsourcing, requesters need a practical tool to help them navigate this space.
  2. 1. Proposing such a practical tool is what we aimed to do in this research. 2. As a starting point, we used a checklist proposed by Kahneman, Lovallo, and Sibony (2011) for the context of business psychology. 3. The big advantage of using a checklist is that it can reduce a complex space (e.g., cognitive biases) to a considerable degree while retaining useful information. 4. Specifically, the checklist proposed by Kahneman, Lovallo, and Sibony aims to help business decision-makers to avoid falling prey to cognitive biases using 12 items. 5. These 12 items cover the vast majority of potential rational mistakes by focusing on most commonly occurring ones and grouping biases together (give example). 6. To develop a similar checklist tool for our context, we adapted the checklist proposed by Kahneman, Lovallo, and Sibony to the context of cognitive biases in crowdsourcing; which we propose in an upcoming paper at HCOMP this year. 7. Our proposed checklist similarly contains 12 items that requesters can consider to identify potential cognitive biases elicited by their crowdsourcing task.
  3. 8. Each of the 12 items covers a specific cognitive bias or family of biases that may occur in the crowdsourcing context. 9. For example, the third item in our checklist concerns *groupthink* or the *bandwagon effect*. (read out loud) 10. Going through each of the 12 items in this format should help requesters efficiently combat cognitive biases in crowdsourcing.
  4. What can requesters do with the information they get from the checklist? 1. Having identified one or more cognitive biases that may affect crowd workers in the task at hand (ideally before collecting the data), the requester may use this information for three different purposes. 2. First, they may wish to *measure* the cognitive biases in question to assess whether crowd workers are actually affected by them. This may require adding additional items or metrics to the crowdsourcing task. 3. Second, requesters could *mitigate* the cognitive biases. Earlier work has already proposed a couple of solutions for this. 4. Third, requesters may *document* the cognitive biases so that the collected data is put in the right perspective. Pointing out such potential limitations can help others when interpreting results or re-using the data.
  5. Of course, checklist is limited That’s why we have a live version of it In our upcoming HCOMP paper, we illustrate the use of our checklist at the hand of a case study. We there also present a retrospective analysis of past HCOMP papers to show that cognitive biases may affect crowd workers in a majority of crowdsourcing tasks but are rarely dealt with. We hope that our proposed checklist can meaningfully contribute to general efforts towards more reliable human-labeled data.