SlideShare a Scribd company logo

A Checklist to Combat Cognitive Biases in Crowdsourcing

T
TimDraws

Paper presentation at HCOMP 2021. This paper won the Amazon Best Paper Award at HCOMP.

1 of 9
Download to read offline
1
WIS
Web
Information
Systems
A Checklist to Combat
Cognitive Biases in Crowdsourcing
HCOMP, Nov 14-18, 2021, Virtual
Tim Draws1, Alisa Rieger1, Oana Inel1, Ujwal Gadiraju1, and Nava Tintarev2
t.a.draws@tudelft.nl
@tmdrws
https://timdraws.net
1Delft University of Technology, 2Maastricht University
2
WIS
Web
Information
Systems
Cognitive Biases in Crowdsourcing…
• …can cause poor data quality
– Examples: anchoring effect, confirmation bias
• ...may occur often but are rarely considered
– Retrospective analysis
• …are difficult to deal with
– Many cognitive biases exist
– Unclear which bias may apply where
References: Eickhoff (2018); Hube, Fetahu, & Gadiraju (2019); Tversky & Kahneman (1974)
3
WIS
Web
Information
Systems
Introducing a Checklist
• Starting point: bias checklist for business decisions
• Adaptation to fit the crowdsourcing context
• Result: 12-item checklist to combat cognitive biases
in crowdsourcing (incl. running example)
References: Kahneman, Lovallo, & Sibony (2011)
4
WIS
Web
Information
Systems
Example
(3) Groupthink or Bandwagon Effect. Does my task design give crowd
workers some notion of other people’s evaluation of the items they annotate?
For example, crowd workers may judge products as more likely to be relevant to
“paella pan” when they see that a majority of other crowd workers have judged
this product as being relevant or if it has received high ratings from consumers.
5
WIS
Web
Information
Systems
12 Checklist Items
1. Self-interest Bias
2. Affect Heuristic
3. Groupthink or Bandwagon Effect
4. Salience Bias
5. Confirmation Bias
6. Availability Bias
7. Anchoring Effect
8. Halo Effect
9. Sunk Cost Fallacy
10. Overconfidence
11. Disaster Neglect
12. Loss Aversion
6
WIS
Web
Information
Systems
Using the Checklist
1.Measure / assess cognitive biases
2.Mitigate cognitive biases
3.Document cognitive biases

Recommended

Introducing the Cognitive-Biases-in-Crowdsourcing Checklist
Introducing the Cognitive-Biases-in-Crowdsourcing ChecklistIntroducing the Cognitive-Biases-in-Crowdsourcing Checklist
Introducing the Cognitive-Biases-in-Crowdsourcing ChecklistTimDraws
 
European Open Science Cloud
European Open Science CloudEuropean Open Science Cloud
European Open Science CloudJisc RDM
 
Digitalisation and the future of research environments
Digitalisation and the future of research environmentsDigitalisation and the future of research environments
Digitalisation and the future of research environmentsJisc
 
Inconsistencies in big data
Inconsistencies in big dataInconsistencies in big data
Inconsistencies in big dataminujoseph
 
Jisc's new shared data centre
Jisc's new shared data centreJisc's new shared data centre
Jisc's new shared data centreJisc
 
Data, Data Everywhere: What's A Publisher to Do?
Data, Data Everywhere: What's  A Publisher to Do?Data, Data Everywhere: What's  A Publisher to Do?
Data, Data Everywhere: What's A Publisher to Do?Anita de Waard
 
Boss, Broussard, Revear: lighntning talk, Challenges facing the preservation ...
Boss, Broussard, Revear: lighntning talk, Challenges facing the preservation ...Boss, Broussard, Revear: lighntning talk, Challenges facing the preservation ...
Boss, Broussard, Revear: lighntning talk, Challenges facing the preservation ...Reynolds Journalism Institute (RJI)
 
EOSC pilot STFC
EOSC pilot STFCEOSC pilot STFC
EOSC pilot STFCJisc RDM
 

More Related Content

What's hot

Big Data and the Future of Publishing
Big Data and the Future of PublishingBig Data and the Future of Publishing
Big Data and the Future of PublishingAnita de Waard
 
Thoughts on Big Data and more for the WA State Legislature
Thoughts on Big Data and more for the WA State LegislatureThoughts on Big Data and more for the WA State Legislature
Thoughts on Big Data and more for the WA State LegislatureUniversity of Washington
 
RDN Lightning talk - Open Research Leeds (@OpenResLeeds): networks, metrics a...
RDN Lightning talk - Open Research Leeds (@OpenResLeeds): networks, metrics a...RDN Lightning talk - Open Research Leeds (@OpenResLeeds): networks, metrics a...
RDN Lightning talk - Open Research Leeds (@OpenResLeeds): networks, metrics a...Nick Sheppard
 
Frances Burton on sensitive data
Frances Burton on sensitive dataFrances Burton on sensitive data
Frances Burton on sensitive dataJisc RDM
 
tools for communicating in the computational sciences
tools for communicating in the computational sciencestools for communicating in the computational sciences
tools for communicating in the computational sciencesBrian Bot
 
The Analytics and Data Science Landscape
The Analytics and Data Science LandscapeThe Analytics and Data Science Landscape
The Analytics and Data Science LandscapePhilip Bourne
 
How can machine learning and AI in the cloud improve research?
How can machine learning and AI in the cloud improve research?How can machine learning and AI in the cloud improve research?
How can machine learning and AI in the cloud improve research?Jisc
 
The Future of FAIR Data: An international social, legal and technological inf...
The Future of FAIR Data: An international social, legal and technological inf...The Future of FAIR Data: An international social, legal and technological inf...
The Future of FAIR Data: An international social, legal and technological inf...Michel Dumontier
 
Big Data Brown Bag
Big Data Brown BagBig Data Brown Bag
Big Data Brown Bagusmanqureshi
 
LISA VII: The Scientific and Technical Foundation for Altmetrics in the Unite...
LISA VII: The Scientific and Technical Foundation for Altmetrics in the Unite...LISA VII: The Scientific and Technical Foundation for Altmetrics in the Unite...
LISA VII: The Scientific and Technical Foundation for Altmetrics in the Unite...William Gunn
 
State of Florida Neo4J Graph Briefing - Keynote
State of Florida Neo4J Graph Briefing - KeynoteState of Florida Neo4J Graph Briefing - Keynote
State of Florida Neo4J Graph Briefing - KeynoteNeo4j
 
infrastructure for communicating data-intensive science
infrastructure for communicating data-intensive scienceinfrastructure for communicating data-intensive science
infrastructure for communicating data-intensive scienceBrian Bot
 
Towards an integrated research management digital ecosystem
Towards an integrated research management digital ecosystemTowards an integrated research management digital ecosystem
Towards an integrated research management digital ecosystemJisc
 
20160414 23 Research Data Things
20160414 23 Research Data Things20160414 23 Research Data Things
20160414 23 Research Data ThingsKatina Toufexis
 
Getting (and giving) credit for all that we do
Getting (and giving) credit for all that we doGetting (and giving) credit for all that we do
Getting (and giving) credit for all that we domhaendel
 
Big Data as a Catalyst for Collaboration & Innovation
Big Data as a Catalyst for Collaboration & InnovationBig Data as a Catalyst for Collaboration & Innovation
Big Data as a Catalyst for Collaboration & InnovationPhilip Bourne
 
Trust threads: Provenance for Data Reuse in Long Tail Science
Trust threads: Provenance for Data Reuse in Long Tail ScienceTrust threads: Provenance for Data Reuse in Long Tail Science
Trust threads: Provenance for Data Reuse in Long Tail ScienceBeth Plale
 
Towards a Data Commons
Towards a Data CommonsTowards a Data Commons
Towards a Data CommonsMichael Becich
 
SGCI Science Gateways: Ushering in a New Era of Sustainability
SGCI Science Gateways: Ushering in a New Era of Sustainability SGCI Science Gateways: Ushering in a New Era of Sustainability
SGCI Science Gateways: Ushering in a New Era of Sustainability Sandra Gesing
 

What's hot (20)

Big Data and the Future of Publishing
Big Data and the Future of PublishingBig Data and the Future of Publishing
Big Data and the Future of Publishing
 
Thoughts on Big Data and more for the WA State Legislature
Thoughts on Big Data and more for the WA State LegislatureThoughts on Big Data and more for the WA State Legislature
Thoughts on Big Data and more for the WA State Legislature
 
RDN Lightning talk - Open Research Leeds (@OpenResLeeds): networks, metrics a...
RDN Lightning talk - Open Research Leeds (@OpenResLeeds): networks, metrics a...RDN Lightning talk - Open Research Leeds (@OpenResLeeds): networks, metrics a...
RDN Lightning talk - Open Research Leeds (@OpenResLeeds): networks, metrics a...
 
Frances Burton on sensitive data
Frances Burton on sensitive dataFrances Burton on sensitive data
Frances Burton on sensitive data
 
Code4 lib2012
Code4 lib2012Code4 lib2012
Code4 lib2012
 
tools for communicating in the computational sciences
tools for communicating in the computational sciencestools for communicating in the computational sciences
tools for communicating in the computational sciences
 
The Analytics and Data Science Landscape
The Analytics and Data Science LandscapeThe Analytics and Data Science Landscape
The Analytics and Data Science Landscape
 
How can machine learning and AI in the cloud improve research?
How can machine learning and AI in the cloud improve research?How can machine learning and AI in the cloud improve research?
How can machine learning and AI in the cloud improve research?
 
The Future of FAIR Data: An international social, legal and technological inf...
The Future of FAIR Data: An international social, legal and technological inf...The Future of FAIR Data: An international social, legal and technological inf...
The Future of FAIR Data: An international social, legal and technological inf...
 
Big Data Brown Bag
Big Data Brown BagBig Data Brown Bag
Big Data Brown Bag
 
LISA VII: The Scientific and Technical Foundation for Altmetrics in the Unite...
LISA VII: The Scientific and Technical Foundation for Altmetrics in the Unite...LISA VII: The Scientific and Technical Foundation for Altmetrics in the Unite...
LISA VII: The Scientific and Technical Foundation for Altmetrics in the Unite...
 
State of Florida Neo4J Graph Briefing - Keynote
State of Florida Neo4J Graph Briefing - KeynoteState of Florida Neo4J Graph Briefing - Keynote
State of Florida Neo4J Graph Briefing - Keynote
 
infrastructure for communicating data-intensive science
infrastructure for communicating data-intensive scienceinfrastructure for communicating data-intensive science
infrastructure for communicating data-intensive science
 
Towards an integrated research management digital ecosystem
Towards an integrated research management digital ecosystemTowards an integrated research management digital ecosystem
Towards an integrated research management digital ecosystem
 
20160414 23 Research Data Things
20160414 23 Research Data Things20160414 23 Research Data Things
20160414 23 Research Data Things
 
Getting (and giving) credit for all that we do
Getting (and giving) credit for all that we doGetting (and giving) credit for all that we do
Getting (and giving) credit for all that we do
 
Big Data as a Catalyst for Collaboration & Innovation
Big Data as a Catalyst for Collaboration & InnovationBig Data as a Catalyst for Collaboration & Innovation
Big Data as a Catalyst for Collaboration & Innovation
 
Trust threads: Provenance for Data Reuse in Long Tail Science
Trust threads: Provenance for Data Reuse in Long Tail ScienceTrust threads: Provenance for Data Reuse in Long Tail Science
Trust threads: Provenance for Data Reuse in Long Tail Science
 
Towards a Data Commons
Towards a Data CommonsTowards a Data Commons
Towards a Data Commons
 
SGCI Science Gateways: Ushering in a New Era of Sustainability
SGCI Science Gateways: Ushering in a New Era of Sustainability SGCI Science Gateways: Ushering in a New Era of Sustainability
SGCI Science Gateways: Ushering in a New Era of Sustainability
 

Similar to A Checklist to Combat Cognitive Biases in Crowdsourcing

Opportunities and Challenges for International Cooperation Around Big Data
Opportunities and Challenges for International Cooperation Around Big DataOpportunities and Challenges for International Cooperation Around Big Data
Opportunities and Challenges for International Cooperation Around Big DataPhilip Bourne
 
Real-time applications of Data Science.pptx
Real-time applications  of Data Science.pptxReal-time applications  of Data Science.pptx
Real-time applications of Data Science.pptxshalini s
 
Tragedy of the (Data) Commons
Tragedy of the (Data) CommonsTragedy of the (Data) Commons
Tragedy of the (Data) CommonsJames Hendler
 
Meeting the Computational Challenges Associated with Human Health
Meeting the Computational Challenges Associated with Human HealthMeeting the Computational Challenges Associated with Human Health
Meeting the Computational Challenges Associated with Human HealthPhilip Bourne
 
What is eScience, and where does it go from here?
What is eScience, and where does it go from here?What is eScience, and where does it go from here?
What is eScience, and where does it go from here?Daniel S. Katz
 
Data Mining and Big Data Challenges and Research Opportunities
Data Mining and Big Data Challenges and Research OpportunitiesData Mining and Big Data Challenges and Research Opportunities
Data Mining and Big Data Challenges and Research OpportunitiesKathirvel Ayyaswamy
 
Introduction to Big Data and its Potential for Dementia Research
Introduction to Big Data and its Potential for Dementia ResearchIntroduction to Big Data and its Potential for Dementia Research
Introduction to Big Data and its Potential for Dementia ResearchDavid De Roure
 
KIT-601-L-UNIT-1 (Revised) Introduction to Data Analytcs.pdf
KIT-601-L-UNIT-1 (Revised) Introduction to Data Analytcs.pdfKIT-601-L-UNIT-1 (Revised) Introduction to Data Analytcs.pdf
KIT-601-L-UNIT-1 (Revised) Introduction to Data Analytcs.pdfDr. Radhey Shyam
 
Introduction to Data Analytics and data analytics life cycle
Introduction to Data Analytics and data analytics life cycleIntroduction to Data Analytics and data analytics life cycle
Introduction to Data Analytics and data analytics life cycleDr. Radhey Shyam
 
Data_Science_Applications_&_Use_Cases.pptx
Data_Science_Applications_&_Use_Cases.pptxData_Science_Applications_&_Use_Cases.pptx
Data_Science_Applications_&_Use_Cases.pptxssuser1a4f0f
 
Data_Science_Applications_&_Use_Cases.pptx
Data_Science_Applications_&_Use_Cases.pptxData_Science_Applications_&_Use_Cases.pptx
Data_Science_Applications_&_Use_Cases.pptxwahiba ben abdessalem
 
Data_Science_Applications_&_Use_Cases.pdf
Data_Science_Applications_&_Use_Cases.pdfData_Science_Applications_&_Use_Cases.pdf
Data_Science_Applications_&_Use_Cases.pdfvishal choudhary
 
Session 0.0 poster minutes madness
Session 0.0   poster minutes madnessSession 0.0   poster minutes madness
Session 0.0 poster minutes madnesssemanticsconference
 
Optimising Scientific Knowledge Transfer: How Collective Sensemaking Can Ena...
Optimising Scientific Knowledge Transfer: How Collective Sensemaking Can Ena...Optimising Scientific Knowledge Transfer: How Collective Sensemaking Can Ena...
Optimising Scientific Knowledge Transfer: How Collective Sensemaking Can Ena...Anita de Waard
 
Australia's Environmental Predictive Capability
Australia's Environmental Predictive CapabilityAustralia's Environmental Predictive Capability
Australia's Environmental Predictive CapabilityTERN Australia
 
#ALAAC15 Linked Data Love
#ALAAC15 Linked Data Love #ALAAC15 Linked Data Love
#ALAAC15 Linked Data Love Kristi Holmes
 
Lauren Michael: The Missing Millions Democratizing Computation and Data ...
Lauren Michael: The Missing Millions Democratizing Computation and Data      ...Lauren Michael: The Missing Millions Democratizing Computation and Data      ...
Lauren Michael: The Missing Millions Democratizing Computation and Data ...Larry Smarr
 
Towards the Intelligent Internet of Everything
Towards the Intelligent Internet of EverythingTowards the Intelligent Internet of Everything
Towards the Intelligent Internet of EverythingRECAP Project
 

Similar to A Checklist to Combat Cognitive Biases in Crowdsourcing (20)

Opportunities and Challenges for International Cooperation Around Big Data
Opportunities and Challenges for International Cooperation Around Big DataOpportunities and Challenges for International Cooperation Around Big Data
Opportunities and Challenges for International Cooperation Around Big Data
 
Real-time applications of Data Science.pptx
Real-time applications  of Data Science.pptxReal-time applications  of Data Science.pptx
Real-time applications of Data Science.pptx
 
Tragedy of the (Data) Commons
Tragedy of the (Data) CommonsTragedy of the (Data) Commons
Tragedy of the (Data) Commons
 
Meeting the Computational Challenges Associated with Human Health
Meeting the Computational Challenges Associated with Human HealthMeeting the Computational Challenges Associated with Human Health
Meeting the Computational Challenges Associated with Human Health
 
COVID-19 Evidence Network to support Decision-making (COVID-END): Resources t...
COVID-19 Evidence Network to support Decision-making (COVID-END): Resources t...COVID-19 Evidence Network to support Decision-making (COVID-END): Resources t...
COVID-19 Evidence Network to support Decision-making (COVID-END): Resources t...
 
What is eScience, and where does it go from here?
What is eScience, and where does it go from here?What is eScience, and where does it go from here?
What is eScience, and where does it go from here?
 
Data Mining and Big Data Challenges and Research Opportunities
Data Mining and Big Data Challenges and Research OpportunitiesData Mining and Big Data Challenges and Research Opportunities
Data Mining and Big Data Challenges and Research Opportunities
 
Introduction to Big Data and its Potential for Dementia Research
Introduction to Big Data and its Potential for Dementia ResearchIntroduction to Big Data and its Potential for Dementia Research
Introduction to Big Data and its Potential for Dementia Research
 
KIT-601-L-UNIT-1 (Revised) Introduction to Data Analytcs.pdf
KIT-601-L-UNIT-1 (Revised) Introduction to Data Analytcs.pdfKIT-601-L-UNIT-1 (Revised) Introduction to Data Analytcs.pdf
KIT-601-L-UNIT-1 (Revised) Introduction to Data Analytcs.pdf
 
Introduction to Data Analytics and data analytics life cycle
Introduction to Data Analytics and data analytics life cycleIntroduction to Data Analytics and data analytics life cycle
Introduction to Data Analytics and data analytics life cycle
 
Data_Science_Applications_&_Use_Cases.pptx
Data_Science_Applications_&_Use_Cases.pptxData_Science_Applications_&_Use_Cases.pptx
Data_Science_Applications_&_Use_Cases.pptx
 
Data_Science_Applications_&_Use_Cases.pptx
Data_Science_Applications_&_Use_Cases.pptxData_Science_Applications_&_Use_Cases.pptx
Data_Science_Applications_&_Use_Cases.pptx
 
Data_Science_Applications_&_Use_Cases.pdf
Data_Science_Applications_&_Use_Cases.pdfData_Science_Applications_&_Use_Cases.pdf
Data_Science_Applications_&_Use_Cases.pdf
 
Session 0.0 poster minutes madness
Session 0.0   poster minutes madnessSession 0.0   poster minutes madness
Session 0.0 poster minutes madness
 
Optimising Scientific Knowledge Transfer: How Collective Sensemaking Can Ena...
Optimising Scientific Knowledge Transfer: How Collective Sensemaking Can Ena...Optimising Scientific Knowledge Transfer: How Collective Sensemaking Can Ena...
Optimising Scientific Knowledge Transfer: How Collective Sensemaking Can Ena...
 
Australia's Environmental Predictive Capability
Australia's Environmental Predictive CapabilityAustralia's Environmental Predictive Capability
Australia's Environmental Predictive Capability
 
#ALAAC15 Linked Data Love
#ALAAC15 Linked Data Love #ALAAC15 Linked Data Love
#ALAAC15 Linked Data Love
 
CI_for_NA
CI_for_NACI_for_NA
CI_for_NA
 
Lauren Michael: The Missing Millions Democratizing Computation and Data ...
Lauren Michael: The Missing Millions Democratizing Computation and Data      ...Lauren Michael: The Missing Millions Democratizing Computation and Data      ...
Lauren Michael: The Missing Millions Democratizing Computation and Data ...
 
Towards the Intelligent Internet of Everything
Towards the Intelligent Internet of EverythingTowards the Intelligent Internet of Everything
Towards the Intelligent Internet of Everything
 

Recently uploaded

PSILOTUM : structure, morphology, anatomy, reproduction , life cycle etc.
PSILOTUM : structure, morphology, anatomy,  reproduction , life cycle etc.PSILOTUM : structure, morphology, anatomy,  reproduction , life cycle etc.
PSILOTUM : structure, morphology, anatomy, reproduction , life cycle etc.Silpa Selvaraj
 
Genetic Code. A comprehensive overview..pdf
Genetic Code. A comprehensive overview..pdfGenetic Code. A comprehensive overview..pdf
Genetic Code. A comprehensive overview..pdfmughalgumar440
 
1.0 - The Light Miscroscope.ppt microscopy
1.0 - The Light Miscroscope.ppt microscopy1.0 - The Light Miscroscope.ppt microscopy
1.0 - The Light Miscroscope.ppt microscopystephenopokuasante
 
Elbow joint - Anatomy of the Elbow joint
Elbow joint - Anatomy of the Elbow jointElbow joint - Anatomy of the Elbow joint
Elbow joint - Anatomy of the Elbow jointTELISHA2
 
Cytotoxic Activity of Linum usitatissimum L. Essential oil against Lung Adeno...
Cytotoxic Activity of Linum usitatissimum L. Essential oil against Lung Adeno...Cytotoxic Activity of Linum usitatissimum L. Essential oil against Lung Adeno...
Cytotoxic Activity of Linum usitatissimum L. Essential oil against Lung Adeno...AmalDhivaharS
 
2024 Insilicogen Company English Brochure
2024 Insilicogen Company English Brochure2024 Insilicogen Company English Brochure
2024 Insilicogen Company English BrochureInsilico Gen
 
Presentacion Mariana Arango- biología molecular
Presentacion Mariana Arango- biología molecularPresentacion Mariana Arango- biología molecular
Presentacion Mariana Arango- biología molecularmarianaarangop
 
Anti-Obesity Activity of Anthocyanins and Corresponding Introduction in Dieta...
Anti-Obesity Activity of Anthocyanins and Corresponding Introduction in Dieta...Anti-Obesity Activity of Anthocyanins and Corresponding Introduction in Dieta...
Anti-Obesity Activity of Anthocyanins and Corresponding Introduction in Dieta...AmalDhivaharS
 
Differences between syrups and elixirs .pptx
Differences between  syrups and elixirs .pptxDifferences between  syrups and elixirs .pptx
Differences between syrups and elixirs .pptxushakiranmai4
 
Open Access Publishing in Astrophysics and the Open Journal of Astrophysics
Open Access Publishing in Astrophysics and the Open Journal of AstrophysicsOpen Access Publishing in Astrophysics and the Open Journal of Astrophysics
Open Access Publishing in Astrophysics and the Open Journal of AstrophysicsPeter Coles
 
Seminario biología molecular Lina Charris
Seminario biología molecular Lina CharrisSeminario biología molecular Lina Charris
Seminario biología molecular Lina CharrisLinaMarcelaCharrisRa
 
Salesforce Starter Package Presentation.
Salesforce Starter Package Presentation.Salesforce Starter Package Presentation.
Salesforce Starter Package Presentation.Naresh Gupta
 
Seminario biología molecular Lina Charris
Seminario biología molecular Lina CharrisSeminario biología molecular Lina Charris
Seminario biología molecular Lina CharrisLinaMarcelaCharrisRa
 
Microbial Fermentation(Strain Improvement)
Microbial  Fermentation(Strain Improvement)Microbial  Fermentation(Strain Improvement)
Microbial Fermentation(Strain Improvement)Rachana Choudhary
 
Safety_and_Health_Programs_v-03-01-17.pptx
Safety_and_Health_Programs_v-03-01-17.pptxSafety_and_Health_Programs_v-03-01-17.pptx
Safety_and_Health_Programs_v-03-01-17.pptxmohamedsami2233
 
Study of X - Ray Spectra and its types
Study  of X  - Ray Spectra and its typesStudy  of X  - Ray Spectra and its types
Study of X - Ray Spectra and its typestanishashukla147
 
electrophoresis: types, advantages, disadvantages and applications.
electrophoresis: types, advantages, disadvantages and applications.electrophoresis: types, advantages, disadvantages and applications.
electrophoresis: types, advantages, disadvantages and applications.Silpa Selvaraj
 
Open Access Publishing in Astrophysics and the Open Journal of Astrophysics
Open Access Publishing in Astrophysics and the Open Journal of AstrophysicsOpen Access Publishing in Astrophysics and the Open Journal of Astrophysics
Open Access Publishing in Astrophysics and the Open Journal of AstrophysicsPeter Coles
 
Earth and Planetary Science | Volume 01 | Issue 01 | April 2022
Earth and Planetary Science | Volume 01 | Issue 01 | April 2022Earth and Planetary Science | Volume 01 | Issue 01 | April 2022
Earth and Planetary Science | Volume 01 | Issue 01 | April 2022Nan Yang Academy of Sciences
 

Recently uploaded (20)

PSILOTUM : structure, morphology, anatomy, reproduction , life cycle etc.
PSILOTUM : structure, morphology, anatomy,  reproduction , life cycle etc.PSILOTUM : structure, morphology, anatomy,  reproduction , life cycle etc.
PSILOTUM : structure, morphology, anatomy, reproduction , life cycle etc.
 
Genetic Code. A comprehensive overview..pdf
Genetic Code. A comprehensive overview..pdfGenetic Code. A comprehensive overview..pdf
Genetic Code. A comprehensive overview..pdf
 
1.0 - The Light Miscroscope.ppt microscopy
1.0 - The Light Miscroscope.ppt microscopy1.0 - The Light Miscroscope.ppt microscopy
1.0 - The Light Miscroscope.ppt microscopy
 
Elbow joint - Anatomy of the Elbow joint
Elbow joint - Anatomy of the Elbow jointElbow joint - Anatomy of the Elbow joint
Elbow joint - Anatomy of the Elbow joint
 
Cytotoxic Activity of Linum usitatissimum L. Essential oil against Lung Adeno...
Cytotoxic Activity of Linum usitatissimum L. Essential oil against Lung Adeno...Cytotoxic Activity of Linum usitatissimum L. Essential oil against Lung Adeno...
Cytotoxic Activity of Linum usitatissimum L. Essential oil against Lung Adeno...
 
2024 Insilicogen Company English Brochure
2024 Insilicogen Company English Brochure2024 Insilicogen Company English Brochure
2024 Insilicogen Company English Brochure
 
Presentacion Mariana Arango- biología molecular
Presentacion Mariana Arango- biología molecularPresentacion Mariana Arango- biología molecular
Presentacion Mariana Arango- biología molecular
 
Anti-Obesity Activity of Anthocyanins and Corresponding Introduction in Dieta...
Anti-Obesity Activity of Anthocyanins and Corresponding Introduction in Dieta...Anti-Obesity Activity of Anthocyanins and Corresponding Introduction in Dieta...
Anti-Obesity Activity of Anthocyanins and Corresponding Introduction in Dieta...
 
Differences between syrups and elixirs .pptx
Differences between  syrups and elixirs .pptxDifferences between  syrups and elixirs .pptx
Differences between syrups and elixirs .pptx
 
Open Access Publishing in Astrophysics and the Open Journal of Astrophysics
Open Access Publishing in Astrophysics and the Open Journal of AstrophysicsOpen Access Publishing in Astrophysics and the Open Journal of Astrophysics
Open Access Publishing in Astrophysics and the Open Journal of Astrophysics
 
Seminario biología molecular Lina Charris
Seminario biología molecular Lina CharrisSeminario biología molecular Lina Charris
Seminario biología molecular Lina Charris
 
Salesforce Starter Package Presentation.
Salesforce Starter Package Presentation.Salesforce Starter Package Presentation.
Salesforce Starter Package Presentation.
 
REGULATION OF METABOLISM IN PLANTS AND THE DIFFERENT MECHANISMS
REGULATION OF METABOLISM IN PLANTS  AND THE DIFFERENT MECHANISMSREGULATION OF METABOLISM IN PLANTS  AND THE DIFFERENT MECHANISMS
REGULATION OF METABOLISM IN PLANTS AND THE DIFFERENT MECHANISMS
 
Seminario biología molecular Lina Charris
Seminario biología molecular Lina CharrisSeminario biología molecular Lina Charris
Seminario biología molecular Lina Charris
 
Microbial Fermentation(Strain Improvement)
Microbial  Fermentation(Strain Improvement)Microbial  Fermentation(Strain Improvement)
Microbial Fermentation(Strain Improvement)
 
Safety_and_Health_Programs_v-03-01-17.pptx
Safety_and_Health_Programs_v-03-01-17.pptxSafety_and_Health_Programs_v-03-01-17.pptx
Safety_and_Health_Programs_v-03-01-17.pptx
 
Study of X - Ray Spectra and its types
Study  of X  - Ray Spectra and its typesStudy  of X  - Ray Spectra and its types
Study of X - Ray Spectra and its types
 
electrophoresis: types, advantages, disadvantages and applications.
electrophoresis: types, advantages, disadvantages and applications.electrophoresis: types, advantages, disadvantages and applications.
electrophoresis: types, advantages, disadvantages and applications.
 
Open Access Publishing in Astrophysics and the Open Journal of Astrophysics
Open Access Publishing in Astrophysics and the Open Journal of AstrophysicsOpen Access Publishing in Astrophysics and the Open Journal of Astrophysics
Open Access Publishing in Astrophysics and the Open Journal of Astrophysics
 
Earth and Planetary Science | Volume 01 | Issue 01 | April 2022
Earth and Planetary Science | Volume 01 | Issue 01 | April 2022Earth and Planetary Science | Volume 01 | Issue 01 | April 2022
Earth and Planetary Science | Volume 01 | Issue 01 | April 2022
 

A Checklist to Combat Cognitive Biases in Crowdsourcing

  • 1. 1 WIS Web Information Systems A Checklist to Combat Cognitive Biases in Crowdsourcing HCOMP, Nov 14-18, 2021, Virtual Tim Draws1, Alisa Rieger1, Oana Inel1, Ujwal Gadiraju1, and Nava Tintarev2 t.a.draws@tudelft.nl @tmdrws https://timdraws.net 1Delft University of Technology, 2Maastricht University
  • 2. 2 WIS Web Information Systems Cognitive Biases in Crowdsourcing… • …can cause poor data quality – Examples: anchoring effect, confirmation bias • ...may occur often but are rarely considered – Retrospective analysis • …are difficult to deal with – Many cognitive biases exist – Unclear which bias may apply where References: Eickhoff (2018); Hube, Fetahu, & Gadiraju (2019); Tversky & Kahneman (1974)
  • 3. 3 WIS Web Information Systems Introducing a Checklist • Starting point: bias checklist for business decisions • Adaptation to fit the crowdsourcing context • Result: 12-item checklist to combat cognitive biases in crowdsourcing (incl. running example) References: Kahneman, Lovallo, & Sibony (2011)
  • 4. 4 WIS Web Information Systems Example (3) Groupthink or Bandwagon Effect. Does my task design give crowd workers some notion of other people’s evaluation of the items they annotate? For example, crowd workers may judge products as more likely to be relevant to “paella pan” when they see that a majority of other crowd workers have judged this product as being relevant or if it has received high ratings from consumers.
  • 5. 5 WIS Web Information Systems 12 Checklist Items 1. Self-interest Bias 2. Affect Heuristic 3. Groupthink or Bandwagon Effect 4. Salience Bias 5. Confirmation Bias 6. Availability Bias 7. Anchoring Effect 8. Halo Effect 9. Sunk Cost Fallacy 10. Overconfidence 11. Disaster Neglect 12. Loss Aversion
  • 6. 6 WIS Web Information Systems Using the Checklist 1.Measure / assess cognitive biases 2.Mitigate cognitive biases 3.Document cognitive biases
  • 7. 7 WIS Web Information Systems Community Updates • Online version of checklist: https://osf.io/g5b82/ • Idea: community can suggest edits 1. Go to repository, download checklist document (.md) 2. Incorporate suggested edits 3. Send edited file + description to us (email on repository) 4. Committee decides on whether/how to update checklist
  • 8. 8 WIS Web Information Systems Discussion & Conclusion • Contributions – Checklist to combat cognitive biases in crowdsourcing – Case study – Retrospective analysis • Future work – Understanding influence of cognitive biases across crowdsourcing scenarios – Mitigation strategies – Updates to checklist • Take home: use checklist to assess, mitigate, and document cognitive biases in crowdsourcing t.a.draws@tudelft.nl @tmdrws https://timdraws.net Paper: https://timdraws.net/files/papers/A_Checklist_to_Combat_Cognitive_Biases_in_Crowdsourcing.pdf Repository: https://osf.io/rbucj/
  • 9. 9 WIS Web Information Systems References Carsten Eickhoff. 2018. Cognitive biases in crowdsourcing. WSDM 2018 - Proceedings of the 11th ACM International Conference on Web Search and Data Mining 2018-Febua (2018), 162–170. https://doi.org/10.1145/3159652.3159654 Tim Draws, Alisa Rieger, Oana Inel, Ujwal Gadiraju, and Nava Tintarev. 2021. A Checklist to Combat Cognitive Biases in Crowdsourcing. Proceedings on the Ninth AAAI Conference on Human Computation and Crowdsourcing (2021). https://timdraws.net/files/papers/A_Checklist_to_Combat_ Cognitive_Biases_in_Crowdsourcing.pdf Christoph Hube, Besnik Fetahu, and Ujwal Gadiraju. 2019. Understanding and mitigating worker biases in the crowdsourced collection of subjective judgments. Conference on Human Factors in Computing Systems - Proceedings (2019). https://doi.org/10.1145/3290605.3300637 Daniel Kahneman, Dan Lovallo, and Olivier Sibony. 2011. Before you make that big decision... Harvard business review 89, 6 (2011). Amos Tversky and Daniel Kahneman. 1974. Judgment under Uncertainty: Heuristics and Biases. Science 185 (Sept. 1974), 1124– 1131. https://doi.org/10.1126/science.185.4157.1124