SlideShare a Scribd company logo
Cognitive biases can lead to poor
reproducibility and replicability of science
Dorothy V. M. Bishop
Professor of Developmental Neuropsychology
University of Oxford
@deevybee
Two cognitive biases that make it
hard to do science well
• Schemata:
• see meaning in random noise
• communicate using simplified narratives
• Confirmation bias: selective attention/memory
• Failure to understand probability
• Misunderstand p-values; many false positives
• Misunderstand sampling error: many false negatives
• Asymmetric moral judgements
• Cherry-picking may not be deliberate
• We find it much easier to process and remember information
that agrees with our viewpoint
Confirmation bias: particular problem in citations
3
4
How common is this in practice?
Even when there are trial registries, we get publication bias
de Vries, Y. A., et al (2018). The cumulative effect of reporting and citation biases on
the apparent efficacy of treatments: the case of depression. Psychological Medicine
48, 2453-2455. doi:10.1017/S0033291718001873
5
How common is this in practice?
Even when there are trial registries, we get publication bias
de Vries, Y. A., et al (2018). The cumulative effect of reporting and citation biases on
the apparent efficacy of treatments: the case of depression. Psychological Medicine
48, 2453-2455. doi:10.1017/S0033291718001873
We only
know
these
studies
exist
because
they are
in a
registry!
6
Publication bias and reporting bias conspire to mislead us
de Vries, Y. A., et al (2018). The cumulative effect of reporting and citation biases on
the apparent efficacy of treatments: the case of depression. Psychological Medicine
48, 2453-2455. doi:10.1017/S0033291718001873
Inheritance of bias
• When we read a peer-reviewed paper, we tend
to trust the citations that back up a point
• When we come to write our own paper, we cite
the same materials
• If prior papers only cite materials agreeing with a
viewpoint, that viewpoint gets entrenched
• You won’t know – unless you explicitly search –
that there are other papers that give a different
picture
7
8
ADVICE TO THOSE STARTING OUT IN
RESEARCH
Errors of statistical reasoning
Failure to understand p-values leads to p-hacking
9
10
I test 16 compounds to see if they improve memory in rats
Should I be excited? Error bars show SE.
http://deevybee.blogspot.com/2016/01/the-amazing-significo-why-researchers.html
P-hacking
Simple explainer using poker
Probability from
unbiased deck of
cards = 1 in 50
• If magician tells you he’ll deal you ‘3 of a kind’,
and he does so, you should be impressed
3 of a kind
• If magician deals 50 hands, and one of them is ‘3 of a
kind’, you should not be impressed
‘Surprisingness’ of a result only interpretable in context of full dataset
http://deevybee.blogspot.com/2016/01/the-amazing-significo-why-researchers.html
P-hacking
Simple explainer using poker
Probability from
unbiased deck of
cards = 1 in 50
• If magician tells you he’ll deal you ‘3 of a kind’,
and he does so, you should be impressed
3 of a kind
• If magician deals 50 hands, and one of them is ‘3 of a
kind’, you should not be impressed
‘Surprisingness’ of a result only interpretable in context of full dataset
http://deevybee.blogspot.com/2016/01/the-amazing-significo-why-researchers.html
P-hacking
Simple explainer using poker
Probability from
unbiased deck of
cards = 1 in 50
• If magician tells you he’ll deal you ‘3 of a kind’,
and he does so, you should be impressed
3 of a kind
• If magician deals 50 hands, and one of them is ‘3 of a
kind’, you should not be impressed
‘Surprisingness’ of a result only interpretable in context of full dataset
14
Key error is to treat a p-value as an indicator of
importance of a finding, regardless of context
I’m not testing whether J is ‘significant’ here – I’m testing whether any of 16 compounds
is ‘significant’
Relevant probability is 1– (probability that all compounds are ‘nonsignificant’)
= 1 - .95^16 = 0.60
Simulated data from random normal distribution with mean = 0
15
To understand p-values:
SIMULATED DATA
“Just as lab scientists are not allowed to handle
dangerous substances without safety training,
researchers should not be allowed anywhere near a P
value or similar measure of statistical probability until
they have demonstrated understanding of what it
means.”
Bishop DVM. Nature 583,
World view: How scientists can stop fooling themselves. July 23 2020
16
Professor Dorothy Bishop, FRS, FMedSci, FBA,
Department of Experimental Psychology,
Anna Watts Building,
Woodstock Road,
Oxford,
OX2 6GG.
@deevybee
http://deevybee.blogspot.com/2012/11/bishopblog-catalogue-
updated-24th-nov.html
https://www.slideshare.net/deevybishop
https://orcid.org/0000-0002-2448-4033

More Related Content

What's hot

Aron chpt 5 ed
Aron chpt 5 edAron chpt 5 ed
Aron chpt 5 ed
Sandra Nicks
 
Crisis of confidence, p-hacking and the future of psychology
Crisis of confidence, p-hacking and the future of psychologyCrisis of confidence, p-hacking and the future of psychology
Crisis of confidence, p-hacking and the future of psychology
Matti Heino
 
Controversy Over the Significance Test Controversy
Controversy Over the Significance Test ControversyControversy Over the Significance Test Controversy
Controversy Over the Significance Test Controversy
jemille6
 
Mayo O&M slides (4-28-13)
Mayo O&M slides (4-28-13)Mayo O&M slides (4-28-13)
Mayo O&M slides (4-28-13)
jemille6
 
Exploratory Research is More Reliable Than Confirmatory Research
Exploratory Research is More Reliable Than Confirmatory ResearchExploratory Research is More Reliable Than Confirmatory Research
Exploratory Research is More Reliable Than Confirmatory Research
jemille6
 
Senn repligate
Senn repligateSenn repligate
Senn repligate
jemille6
 
Probing with Severity: Beyond Bayesian Probabilism and Frequentist Performance
Probing with Severity: Beyond Bayesian Probabilism and Frequentist PerformanceProbing with Severity: Beyond Bayesian Probabilism and Frequentist Performance
Probing with Severity: Beyond Bayesian Probabilism and Frequentist Performance
jemille6
 
Research method
Research methodResearch method
Research method
Ch Irfan
 
Behaviour change and intervention research
Behaviour change and intervention researchBehaviour change and intervention research
Behaviour change and intervention research
Matti Heino
 
"The Statistical Replication Crisis: Paradoxes and Scapegoats”
"The Statistical Replication Crisis: Paradoxes and Scapegoats”"The Statistical Replication Crisis: Paradoxes and Scapegoats”
"The Statistical Replication Crisis: Paradoxes and Scapegoats”
jemille6
 
Statistical skepticism: How to use significance tests effectively
Statistical skepticism: How to use significance tests effectively Statistical skepticism: How to use significance tests effectively
Statistical skepticism: How to use significance tests effectively
jemille6
 
Es estadísticas duro
Es estadísticas duroEs estadísticas duro
Es estadísticas duro
leonardo19940511
 
Final mayo's aps_talk
Final mayo's aps_talkFinal mayo's aps_talk
Final mayo's aps_talk
jemille6
 
CourseProjectReviewPaper.jktamanaPDF
CourseProjectReviewPaper.jktamanaPDFCourseProjectReviewPaper.jktamanaPDF
CourseProjectReviewPaper.jktamanaPDF
Jasmine Tamanaha
 
Mayo & parker spsp 2016 june 16
Mayo & parker   spsp 2016 june 16Mayo & parker   spsp 2016 june 16
Mayo & parker spsp 2016 june 16
jemille6
 
Surrogate Science: How Fisher, Neyman-Pearson, and Bayes Were Transformed int...
Surrogate Science: How Fisher, Neyman-Pearson, and Bayes Were Transformed int...Surrogate Science: How Fisher, Neyman-Pearson, and Bayes Were Transformed int...
Surrogate Science: How Fisher, Neyman-Pearson, and Bayes Were Transformed int...
jemille6
 
AAPOR 2016 Debriefing_v4_final public
AAPOR 2016 Debriefing_v4_final publicAAPOR 2016 Debriefing_v4_final public
AAPOR 2016 Debriefing_v4_final public
Suzanne Kaasa
 
Bayesian Bias Correction: Critically evaluating sets of studies in the presen...
Bayesian Bias Correction: Critically evaluating sets of studies in the presen...Bayesian Bias Correction: Critically evaluating sets of studies in the presen...
Bayesian Bias Correction: Critically evaluating sets of studies in the presen...
Alexander Etz
 
Yoav Benjamini, "In the world beyond p<.05: When & How to use P<.0499..."
Yoav Benjamini, "In the world beyond p<.05: When & How to use P<.0499..."Yoav Benjamini, "In the world beyond p<.05: When & How to use P<.0499..."
Yoav Benjamini, "In the world beyond p<.05: When & How to use P<.0499..."
jemille6
 
P-Value "Reforms": Fixing Science or Threat to Replication and Falsification
P-Value "Reforms": Fixing Science or Threat to Replication and FalsificationP-Value "Reforms": Fixing Science or Threat to Replication and Falsification
P-Value "Reforms": Fixing Science or Threat to Replication and Falsification
jemille6
 

What's hot (20)

Aron chpt 5 ed
Aron chpt 5 edAron chpt 5 ed
Aron chpt 5 ed
 
Crisis of confidence, p-hacking and the future of psychology
Crisis of confidence, p-hacking and the future of psychologyCrisis of confidence, p-hacking and the future of psychology
Crisis of confidence, p-hacking and the future of psychology
 
Controversy Over the Significance Test Controversy
Controversy Over the Significance Test ControversyControversy Over the Significance Test Controversy
Controversy Over the Significance Test Controversy
 
Mayo O&M slides (4-28-13)
Mayo O&M slides (4-28-13)Mayo O&M slides (4-28-13)
Mayo O&M slides (4-28-13)
 
Exploratory Research is More Reliable Than Confirmatory Research
Exploratory Research is More Reliable Than Confirmatory ResearchExploratory Research is More Reliable Than Confirmatory Research
Exploratory Research is More Reliable Than Confirmatory Research
 
Senn repligate
Senn repligateSenn repligate
Senn repligate
 
Probing with Severity: Beyond Bayesian Probabilism and Frequentist Performance
Probing with Severity: Beyond Bayesian Probabilism and Frequentist PerformanceProbing with Severity: Beyond Bayesian Probabilism and Frequentist Performance
Probing with Severity: Beyond Bayesian Probabilism and Frequentist Performance
 
Research method
Research methodResearch method
Research method
 
Behaviour change and intervention research
Behaviour change and intervention researchBehaviour change and intervention research
Behaviour change and intervention research
 
"The Statistical Replication Crisis: Paradoxes and Scapegoats”
"The Statistical Replication Crisis: Paradoxes and Scapegoats”"The Statistical Replication Crisis: Paradoxes and Scapegoats”
"The Statistical Replication Crisis: Paradoxes and Scapegoats”
 
Statistical skepticism: How to use significance tests effectively
Statistical skepticism: How to use significance tests effectively Statistical skepticism: How to use significance tests effectively
Statistical skepticism: How to use significance tests effectively
 
Es estadísticas duro
Es estadísticas duroEs estadísticas duro
Es estadísticas duro
 
Final mayo's aps_talk
Final mayo's aps_talkFinal mayo's aps_talk
Final mayo's aps_talk
 
CourseProjectReviewPaper.jktamanaPDF
CourseProjectReviewPaper.jktamanaPDFCourseProjectReviewPaper.jktamanaPDF
CourseProjectReviewPaper.jktamanaPDF
 
Mayo &amp; parker spsp 2016 june 16
Mayo &amp; parker   spsp 2016 june 16Mayo &amp; parker   spsp 2016 june 16
Mayo &amp; parker spsp 2016 june 16
 
Surrogate Science: How Fisher, Neyman-Pearson, and Bayes Were Transformed int...
Surrogate Science: How Fisher, Neyman-Pearson, and Bayes Were Transformed int...Surrogate Science: How Fisher, Neyman-Pearson, and Bayes Were Transformed int...
Surrogate Science: How Fisher, Neyman-Pearson, and Bayes Were Transformed int...
 
AAPOR 2016 Debriefing_v4_final public
AAPOR 2016 Debriefing_v4_final publicAAPOR 2016 Debriefing_v4_final public
AAPOR 2016 Debriefing_v4_final public
 
Bayesian Bias Correction: Critically evaluating sets of studies in the presen...
Bayesian Bias Correction: Critically evaluating sets of studies in the presen...Bayesian Bias Correction: Critically evaluating sets of studies in the presen...
Bayesian Bias Correction: Critically evaluating sets of studies in the presen...
 
Yoav Benjamini, "In the world beyond p<.05: When & How to use P<.0499..."
Yoav Benjamini, "In the world beyond p<.05: When & How to use P<.0499..."Yoav Benjamini, "In the world beyond p<.05: When & How to use P<.0499..."
Yoav Benjamini, "In the world beyond p<.05: When & How to use P<.0499..."
 
P-Value "Reforms": Fixing Science or Threat to Replication and Falsification
P-Value "Reforms": Fixing Science or Threat to Replication and FalsificationP-Value "Reforms": Fixing Science or Threat to Replication and Falsification
P-Value "Reforms": Fixing Science or Threat to Replication and Falsification
 

Similar to Short talk on 2 cognitive biases and reproducibility

Chapter 1 - AP Psychology
Chapter 1 - AP PsychologyChapter 1 - AP Psychology
Chapter 1 - AP Psychology
Dalat International School
 
Chapter1
Chapter1Chapter1
Chapter1
jenniferdavis22
 
AP Psychology - Research Methods
AP Psychology - Research MethodsAP Psychology - Research Methods
AP Psychology - Research Methods
Taylor Stevens
 
PSY101 Week 1 Critical Thinking
PSY101 Week 1 Critical ThinkingPSY101 Week 1 Critical Thinking
PSY101 Week 1 Critical Thinking
Dr. Russell Rodrigo
 
Myers 9e ch1 - Thinking Critically with Psychological Science
Myers 9e ch1 - Thinking Critically with Psychological ScienceMyers 9e ch1 - Thinking Critically with Psychological Science
Myers 9e ch1 - Thinking Critically with Psychological Science
Julia Isabel Rivera
 
Chapter 1, Myers Psychology 9e
Chapter 1, Myers Psychology 9eChapter 1, Myers Psychology 9e
Chapter 1, Myers Psychology 9e
Charleen Gribben
 
9e ch 01
9e ch 019e ch 01
Chapter 1 (thinking critically)
Chapter 1 (thinking critically)Chapter 1 (thinking critically)
Chapter 1 (thinking critically)
dcrocke1
 
Bergman Psych- ch 01
Bergman Psych- ch 01Bergman Psych- ch 01
Bergman Psych- ch 01
Squalicum High School
 
Chi square test
Chi square testChi square test
Chi square test
callr
 
009906275.pdf
009906275.pdf009906275.pdf
009906275.pdf
EidTahir
 
Sti2018 jws
Sti2018 jwsSti2018 jws
Sti2018 jws
Jesper Schneider
 
Research methods Ch. 2
Research methods Ch. 2Research methods Ch. 2
Research methods Ch. 2
kbolinsky
 
Cognitive biases - Logic vs. The brain
Cognitive biases - Logic vs. The brainCognitive biases - Logic vs. The brain
Cognitive biases - Logic vs. The brain
Aditya Shukla
 
Topic Learning TeamNumber of Pages 2 (Double Spaced)Num.docx
Topic Learning TeamNumber of Pages 2 (Double Spaced)Num.docxTopic Learning TeamNumber of Pages 2 (Double Spaced)Num.docx
Topic Learning TeamNumber of Pages 2 (Double Spaced)Num.docx
AASTHA76
 
Chapter 1 Ap Psych- Research Methods
Chapter 1 Ap Psych- Research MethodsChapter 1 Ap Psych- Research Methods
Chapter 1 Ap Psych- Research Methods
Dr. J's AP Psych Class
 
Page 266LEARNING OBJECTIVES· Explain how researchers use inf.docx
Page 266LEARNING OBJECTIVES· Explain how researchers use inf.docxPage 266LEARNING OBJECTIVES· Explain how researchers use inf.docx
Page 266LEARNING OBJECTIVES· Explain how researchers use inf.docx
karlhennesey
 
Module 2 research strategies how psychologists ask and answer questions
Module 2 research strategies  how psychologists ask and answer questionsModule 2 research strategies  how psychologists ask and answer questions
Module 2 research strategies how psychologists ask and answer questions
Tina Medley
 
Statistics in clinical and translational research common pitfalls
Statistics in clinical and translational research  common pitfallsStatistics in clinical and translational research  common pitfalls
Statistics in clinical and translational research common pitfalls
Pavlos Msaouel, MD, PhD
 
3. hypothesis
3. hypothesis3. hypothesis
3. hypothesis
Chanda Jabeen
 

Similar to Short talk on 2 cognitive biases and reproducibility (20)

Chapter 1 - AP Psychology
Chapter 1 - AP PsychologyChapter 1 - AP Psychology
Chapter 1 - AP Psychology
 
Chapter1
Chapter1Chapter1
Chapter1
 
AP Psychology - Research Methods
AP Psychology - Research MethodsAP Psychology - Research Methods
AP Psychology - Research Methods
 
PSY101 Week 1 Critical Thinking
PSY101 Week 1 Critical ThinkingPSY101 Week 1 Critical Thinking
PSY101 Week 1 Critical Thinking
 
Myers 9e ch1 - Thinking Critically with Psychological Science
Myers 9e ch1 - Thinking Critically with Psychological ScienceMyers 9e ch1 - Thinking Critically with Psychological Science
Myers 9e ch1 - Thinking Critically with Psychological Science
 
Chapter 1, Myers Psychology 9e
Chapter 1, Myers Psychology 9eChapter 1, Myers Psychology 9e
Chapter 1, Myers Psychology 9e
 
9e ch 01
9e ch 019e ch 01
9e ch 01
 
Chapter 1 (thinking critically)
Chapter 1 (thinking critically)Chapter 1 (thinking critically)
Chapter 1 (thinking critically)
 
Bergman Psych- ch 01
Bergman Psych- ch 01Bergman Psych- ch 01
Bergman Psych- ch 01
 
Chi square test
Chi square testChi square test
Chi square test
 
009906275.pdf
009906275.pdf009906275.pdf
009906275.pdf
 
Sti2018 jws
Sti2018 jwsSti2018 jws
Sti2018 jws
 
Research methods Ch. 2
Research methods Ch. 2Research methods Ch. 2
Research methods Ch. 2
 
Cognitive biases - Logic vs. The brain
Cognitive biases - Logic vs. The brainCognitive biases - Logic vs. The brain
Cognitive biases - Logic vs. The brain
 
Topic Learning TeamNumber of Pages 2 (Double Spaced)Num.docx
Topic Learning TeamNumber of Pages 2 (Double Spaced)Num.docxTopic Learning TeamNumber of Pages 2 (Double Spaced)Num.docx
Topic Learning TeamNumber of Pages 2 (Double Spaced)Num.docx
 
Chapter 1 Ap Psych- Research Methods
Chapter 1 Ap Psych- Research MethodsChapter 1 Ap Psych- Research Methods
Chapter 1 Ap Psych- Research Methods
 
Page 266LEARNING OBJECTIVES· Explain how researchers use inf.docx
Page 266LEARNING OBJECTIVES· Explain how researchers use inf.docxPage 266LEARNING OBJECTIVES· Explain how researchers use inf.docx
Page 266LEARNING OBJECTIVES· Explain how researchers use inf.docx
 
Module 2 research strategies how psychologists ask and answer questions
Module 2 research strategies  how psychologists ask and answer questionsModule 2 research strategies  how psychologists ask and answer questions
Module 2 research strategies how psychologists ask and answer questions
 
Statistics in clinical and translational research common pitfalls
Statistics in clinical and translational research  common pitfallsStatistics in clinical and translational research  common pitfalls
Statistics in clinical and translational research common pitfalls
 
3. hypothesis
3. hypothesis3. hypothesis
3. hypothesis
 

More from Dorothy Bishop

Exercise/fish oil intervention for dyslexia
Exercise/fish oil intervention for dyslexiaExercise/fish oil intervention for dyslexia
Exercise/fish oil intervention for dyslexia
Dorothy Bishop
 
Open Research Practices in the Age of a Papermill Pandemic
Open Research Practices in the Age of a Papermill PandemicOpen Research Practices in the Age of a Papermill Pandemic
Open Research Practices in the Age of a Papermill Pandemic
Dorothy Bishop
 
Language-impaired preschoolers: A follow-up into adolescence.
Language-impaired preschoolers: A follow-up into adolescence.Language-impaired preschoolers: A follow-up into adolescence.
Language-impaired preschoolers: A follow-up into adolescence.
Dorothy Bishop
 
Journal club summary: Open Science save lives
Journal club summary: Open Science save livesJournal club summary: Open Science save lives
Journal club summary: Open Science save lives
Dorothy Bishop
 
Otitis media with effusion: an illustration of ascertainment bias
Otitis media with effusion: an illustration of ascertainment biasOtitis media with effusion: an illustration of ascertainment bias
Otitis media with effusion: an illustration of ascertainment bias
Dorothy Bishop
 
What are metrics good for? Reflections on REF and TEF
What are metrics good for? Reflections on REF and TEFWhat are metrics good for? Reflections on REF and TEF
What are metrics good for? Reflections on REF and TEF
Dorothy Bishop
 
Biomarkers for psychological phenotypes?
Biomarkers for psychological phenotypes?Biomarkers for psychological phenotypes?
Biomarkers for psychological phenotypes?
Dorothy Bishop
 
Data simulation basics
Data simulation basicsData simulation basics
Data simulation basics
Dorothy Bishop
 
Simulating data to gain insights into power and p-hacking
Simulating data to gain insights intopower and p-hackingSimulating data to gain insights intopower and p-hacking
Simulating data to gain insights into power and p-hacking
Dorothy Bishop
 
Talk on reproducibility in EEG research
Talk on reproducibility in EEG researchTalk on reproducibility in EEG research
Talk on reproducibility in EEG research
Dorothy Bishop
 
What is Developmental Language Disorder
What is Developmental Language DisorderWhat is Developmental Language Disorder
What is Developmental Language Disorder
Dorothy Bishop
 
Developmental language disorder and auditory processing disorder: 
Same or di...
Developmental language disorder and auditory processing disorder: 
Same or di...Developmental language disorder and auditory processing disorder: 
Same or di...
Developmental language disorder and auditory processing disorder: 
Same or di...
Dorothy Bishop
 
Fallibility in science: Responsible ways to handle mistakes
Fallibility in science: Responsible ways to handle mistakesFallibility in science: Responsible ways to handle mistakes
Fallibility in science: Responsible ways to handle mistakes
Dorothy Bishop
 
Improve your study with pre-registration
Improve your study with pre-registrationImprove your study with pre-registration
Improve your study with pre-registration
Dorothy Bishop
 
Introduction to simulating data to improve your research
Introduction to simulating data to improve your researchIntroduction to simulating data to improve your research
Introduction to simulating data to improve your research
Dorothy Bishop
 
Southampton: lecture on TEF
Southampton: lecture on TEFSouthampton: lecture on TEF
Southampton: lecture on TEF
Dorothy Bishop
 
Reading list: What’s wrong with our universities
Reading list: What’s wrong with our universitiesReading list: What’s wrong with our universities
Reading list: What’s wrong with our universities
Dorothy Bishop
 
IJLCD Winter Lecture 2016-7 : References
IJLCD Winter Lecture 2016-7 : ReferencesIJLCD Winter Lecture 2016-7 : References
IJLCD Winter Lecture 2016-7 : References
Dorothy Bishop
 
What's wrong with our Universities, and will the Teaching Excellence Framewor...
What's wrong with our Universities, and will the Teaching Excellence Framewor...What's wrong with our Universities, and will the Teaching Excellence Framewor...
What's wrong with our Universities, and will the Teaching Excellence Framewor...
Dorothy Bishop
 
Bishop reproducibility references nov2016
Bishop reproducibility references nov2016Bishop reproducibility references nov2016
Bishop reproducibility references nov2016
Dorothy Bishop
 

More from Dorothy Bishop (20)

Exercise/fish oil intervention for dyslexia
Exercise/fish oil intervention for dyslexiaExercise/fish oil intervention for dyslexia
Exercise/fish oil intervention for dyslexia
 
Open Research Practices in the Age of a Papermill Pandemic
Open Research Practices in the Age of a Papermill PandemicOpen Research Practices in the Age of a Papermill Pandemic
Open Research Practices in the Age of a Papermill Pandemic
 
Language-impaired preschoolers: A follow-up into adolescence.
Language-impaired preschoolers: A follow-up into adolescence.Language-impaired preschoolers: A follow-up into adolescence.
Language-impaired preschoolers: A follow-up into adolescence.
 
Journal club summary: Open Science save lives
Journal club summary: Open Science save livesJournal club summary: Open Science save lives
Journal club summary: Open Science save lives
 
Otitis media with effusion: an illustration of ascertainment bias
Otitis media with effusion: an illustration of ascertainment biasOtitis media with effusion: an illustration of ascertainment bias
Otitis media with effusion: an illustration of ascertainment bias
 
What are metrics good for? Reflections on REF and TEF
What are metrics good for? Reflections on REF and TEFWhat are metrics good for? Reflections on REF and TEF
What are metrics good for? Reflections on REF and TEF
 
Biomarkers for psychological phenotypes?
Biomarkers for psychological phenotypes?Biomarkers for psychological phenotypes?
Biomarkers for psychological phenotypes?
 
Data simulation basics
Data simulation basicsData simulation basics
Data simulation basics
 
Simulating data to gain insights into power and p-hacking
Simulating data to gain insights intopower and p-hackingSimulating data to gain insights intopower and p-hacking
Simulating data to gain insights into power and p-hacking
 
Talk on reproducibility in EEG research
Talk on reproducibility in EEG researchTalk on reproducibility in EEG research
Talk on reproducibility in EEG research
 
What is Developmental Language Disorder
What is Developmental Language DisorderWhat is Developmental Language Disorder
What is Developmental Language Disorder
 
Developmental language disorder and auditory processing disorder: 
Same or di...
Developmental language disorder and auditory processing disorder: 
Same or di...Developmental language disorder and auditory processing disorder: 
Same or di...
Developmental language disorder and auditory processing disorder: 
Same or di...
 
Fallibility in science: Responsible ways to handle mistakes
Fallibility in science: Responsible ways to handle mistakesFallibility in science: Responsible ways to handle mistakes
Fallibility in science: Responsible ways to handle mistakes
 
Improve your study with pre-registration
Improve your study with pre-registrationImprove your study with pre-registration
Improve your study with pre-registration
 
Introduction to simulating data to improve your research
Introduction to simulating data to improve your researchIntroduction to simulating data to improve your research
Introduction to simulating data to improve your research
 
Southampton: lecture on TEF
Southampton: lecture on TEFSouthampton: lecture on TEF
Southampton: lecture on TEF
 
Reading list: What’s wrong with our universities
Reading list: What’s wrong with our universitiesReading list: What’s wrong with our universities
Reading list: What’s wrong with our universities
 
IJLCD Winter Lecture 2016-7 : References
IJLCD Winter Lecture 2016-7 : ReferencesIJLCD Winter Lecture 2016-7 : References
IJLCD Winter Lecture 2016-7 : References
 
What's wrong with our Universities, and will the Teaching Excellence Framewor...
What's wrong with our Universities, and will the Teaching Excellence Framewor...What's wrong with our Universities, and will the Teaching Excellence Framewor...
What's wrong with our Universities, and will the Teaching Excellence Framewor...
 
Bishop reproducibility references nov2016
Bishop reproducibility references nov2016Bishop reproducibility references nov2016
Bishop reproducibility references nov2016
 

Recently uploaded

Evidence of Jet Activity from the Secondary Black Hole in the OJ 287 Binary S...
Evidence of Jet Activity from the Secondary Black Hole in the OJ 287 Binary S...Evidence of Jet Activity from the Secondary Black Hole in the OJ 287 Binary S...
Evidence of Jet Activity from the Secondary Black Hole in the OJ 287 Binary S...
Sérgio Sacani
 
SDSS1335+0728: The awakening of a ∼ 106M⊙ black hole⋆
SDSS1335+0728: The awakening of a ∼ 106M⊙ black hole⋆SDSS1335+0728: The awakening of a ∼ 106M⊙ black hole⋆
SDSS1335+0728: The awakening of a ∼ 106M⊙ black hole⋆
Sérgio Sacani
 
LEARNING TO LIVE WITH LAWS OF MOTION .pptx
LEARNING TO LIVE WITH LAWS OF MOTION .pptxLEARNING TO LIVE WITH LAWS OF MOTION .pptx
LEARNING TO LIVE WITH LAWS OF MOTION .pptx
yourprojectpartner05
 
Methods of grain storage Structures in India.pdf
Methods of grain storage Structures in India.pdfMethods of grain storage Structures in India.pdf
Methods of grain storage Structures in India.pdf
PirithiRaju
 
cathode ray oscilloscope and its applications
cathode ray oscilloscope and its applicationscathode ray oscilloscope and its applications
cathode ray oscilloscope and its applications
sandertein
 
Candidate young stellar objects in the S-cluster: Kinematic analysis of a sub...
Candidate young stellar objects in the S-cluster: Kinematic analysis of a sub...Candidate young stellar objects in the S-cluster: Kinematic analysis of a sub...
Candidate young stellar objects in the S-cluster: Kinematic analysis of a sub...
Sérgio Sacani
 
CLASS 12th CHEMISTRY SOLID STATE ppt (Animated)
CLASS 12th CHEMISTRY SOLID STATE ppt (Animated)CLASS 12th CHEMISTRY SOLID STATE ppt (Animated)
CLASS 12th CHEMISTRY SOLID STATE ppt (Animated)
eitps1506
 
Authoring a personal GPT for your research and practice: How we created the Q...
Authoring a personal GPT for your research and practice: How we created the Q...Authoring a personal GPT for your research and practice: How we created the Q...
Authoring a personal GPT for your research and practice: How we created the Q...
Leonel Morgado
 
Introduction_Ch_01_Biotech Biotechnology course .pptx
Introduction_Ch_01_Biotech Biotechnology course .pptxIntroduction_Ch_01_Biotech Biotechnology course .pptx
Introduction_Ch_01_Biotech Biotechnology course .pptx
QusayMaghayerh
 
HUMAN EYE By-R.M Class 10 phy best digital notes.pdf
HUMAN EYE By-R.M Class 10 phy best digital notes.pdfHUMAN EYE By-R.M Class 10 phy best digital notes.pdf
HUMAN EYE By-R.M Class 10 phy best digital notes.pdf
Ritik83251
 
Discovery of An Apparent Red, High-Velocity Type Ia Supernova at 𝐳 = 2.9 wi...
Discovery of An Apparent Red, High-Velocity Type Ia Supernova at  𝐳 = 2.9  wi...Discovery of An Apparent Red, High-Velocity Type Ia Supernova at  𝐳 = 2.9  wi...
Discovery of An Apparent Red, High-Velocity Type Ia Supernova at 𝐳 = 2.9 wi...
Sérgio Sacani
 
Mending Clothing to Support Sustainable Fashion_CIMaR 2024.pdf
Mending Clothing to Support Sustainable Fashion_CIMaR 2024.pdfMending Clothing to Support Sustainable Fashion_CIMaR 2024.pdf
Mending Clothing to Support Sustainable Fashion_CIMaR 2024.pdf
Selcen Ozturkcan
 
(June 12, 2024) Webinar: Development of PET theranostics targeting the molecu...
(June 12, 2024) Webinar: Development of PET theranostics targeting the molecu...(June 12, 2024) Webinar: Development of PET theranostics targeting the molecu...
(June 12, 2024) Webinar: Development of PET theranostics targeting the molecu...
Scintica Instrumentation
 
BIRDS DIVERSITY OF SOOTEA BISWANATH ASSAM.ppt.pptx
BIRDS  DIVERSITY OF SOOTEA BISWANATH ASSAM.ppt.pptxBIRDS  DIVERSITY OF SOOTEA BISWANATH ASSAM.ppt.pptx
BIRDS DIVERSITY OF SOOTEA BISWANATH ASSAM.ppt.pptx
goluk9330
 
Sustainable Land Management - Climate Smart Agriculture
Sustainable Land Management - Climate Smart AgricultureSustainable Land Management - Climate Smart Agriculture
Sustainable Land Management - Climate Smart Agriculture
International Food Policy Research Institute- South Asia Office
 
Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...
Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...
Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...
PsychoTech Services
 
Randomised Optimisation Algorithms in DAPHNE
Randomised Optimisation Algorithms in DAPHNERandomised Optimisation Algorithms in DAPHNE
Randomised Optimisation Algorithms in DAPHNE
University of Maribor
 
Physiology of Nervous System presentation.pptx
Physiology of Nervous System presentation.pptxPhysiology of Nervous System presentation.pptx
Physiology of Nervous System presentation.pptx
fatima132662
 
Signatures of wave erosion in Titan’s coasts
Signatures of wave erosion in Titan’s coastsSignatures of wave erosion in Titan’s coasts
Signatures of wave erosion in Titan’s coasts
Sérgio Sacani
 
Clinical periodontology and implant dentistry 2003.pdf
Clinical periodontology and implant dentistry 2003.pdfClinical periodontology and implant dentistry 2003.pdf
Clinical periodontology and implant dentistry 2003.pdf
RAYMUNDONAVARROCORON
 

Recently uploaded (20)

Evidence of Jet Activity from the Secondary Black Hole in the OJ 287 Binary S...
Evidence of Jet Activity from the Secondary Black Hole in the OJ 287 Binary S...Evidence of Jet Activity from the Secondary Black Hole in the OJ 287 Binary S...
Evidence of Jet Activity from the Secondary Black Hole in the OJ 287 Binary S...
 
SDSS1335+0728: The awakening of a ∼ 106M⊙ black hole⋆
SDSS1335+0728: The awakening of a ∼ 106M⊙ black hole⋆SDSS1335+0728: The awakening of a ∼ 106M⊙ black hole⋆
SDSS1335+0728: The awakening of a ∼ 106M⊙ black hole⋆
 
LEARNING TO LIVE WITH LAWS OF MOTION .pptx
LEARNING TO LIVE WITH LAWS OF MOTION .pptxLEARNING TO LIVE WITH LAWS OF MOTION .pptx
LEARNING TO LIVE WITH LAWS OF MOTION .pptx
 
Methods of grain storage Structures in India.pdf
Methods of grain storage Structures in India.pdfMethods of grain storage Structures in India.pdf
Methods of grain storage Structures in India.pdf
 
cathode ray oscilloscope and its applications
cathode ray oscilloscope and its applicationscathode ray oscilloscope and its applications
cathode ray oscilloscope and its applications
 
Candidate young stellar objects in the S-cluster: Kinematic analysis of a sub...
Candidate young stellar objects in the S-cluster: Kinematic analysis of a sub...Candidate young stellar objects in the S-cluster: Kinematic analysis of a sub...
Candidate young stellar objects in the S-cluster: Kinematic analysis of a sub...
 
CLASS 12th CHEMISTRY SOLID STATE ppt (Animated)
CLASS 12th CHEMISTRY SOLID STATE ppt (Animated)CLASS 12th CHEMISTRY SOLID STATE ppt (Animated)
CLASS 12th CHEMISTRY SOLID STATE ppt (Animated)
 
Authoring a personal GPT for your research and practice: How we created the Q...
Authoring a personal GPT for your research and practice: How we created the Q...Authoring a personal GPT for your research and practice: How we created the Q...
Authoring a personal GPT for your research and practice: How we created the Q...
 
Introduction_Ch_01_Biotech Biotechnology course .pptx
Introduction_Ch_01_Biotech Biotechnology course .pptxIntroduction_Ch_01_Biotech Biotechnology course .pptx
Introduction_Ch_01_Biotech Biotechnology course .pptx
 
HUMAN EYE By-R.M Class 10 phy best digital notes.pdf
HUMAN EYE By-R.M Class 10 phy best digital notes.pdfHUMAN EYE By-R.M Class 10 phy best digital notes.pdf
HUMAN EYE By-R.M Class 10 phy best digital notes.pdf
 
Discovery of An Apparent Red, High-Velocity Type Ia Supernova at 𝐳 = 2.9 wi...
Discovery of An Apparent Red, High-Velocity Type Ia Supernova at  𝐳 = 2.9  wi...Discovery of An Apparent Red, High-Velocity Type Ia Supernova at  𝐳 = 2.9  wi...
Discovery of An Apparent Red, High-Velocity Type Ia Supernova at 𝐳 = 2.9 wi...
 
Mending Clothing to Support Sustainable Fashion_CIMaR 2024.pdf
Mending Clothing to Support Sustainable Fashion_CIMaR 2024.pdfMending Clothing to Support Sustainable Fashion_CIMaR 2024.pdf
Mending Clothing to Support Sustainable Fashion_CIMaR 2024.pdf
 
(June 12, 2024) Webinar: Development of PET theranostics targeting the molecu...
(June 12, 2024) Webinar: Development of PET theranostics targeting the molecu...(June 12, 2024) Webinar: Development of PET theranostics targeting the molecu...
(June 12, 2024) Webinar: Development of PET theranostics targeting the molecu...
 
BIRDS DIVERSITY OF SOOTEA BISWANATH ASSAM.ppt.pptx
BIRDS  DIVERSITY OF SOOTEA BISWANATH ASSAM.ppt.pptxBIRDS  DIVERSITY OF SOOTEA BISWANATH ASSAM.ppt.pptx
BIRDS DIVERSITY OF SOOTEA BISWANATH ASSAM.ppt.pptx
 
Sustainable Land Management - Climate Smart Agriculture
Sustainable Land Management - Climate Smart AgricultureSustainable Land Management - Climate Smart Agriculture
Sustainable Land Management - Climate Smart Agriculture
 
Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...
Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...
Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...
 
Randomised Optimisation Algorithms in DAPHNE
Randomised Optimisation Algorithms in DAPHNERandomised Optimisation Algorithms in DAPHNE
Randomised Optimisation Algorithms in DAPHNE
 
Physiology of Nervous System presentation.pptx
Physiology of Nervous System presentation.pptxPhysiology of Nervous System presentation.pptx
Physiology of Nervous System presentation.pptx
 
Signatures of wave erosion in Titan’s coasts
Signatures of wave erosion in Titan’s coastsSignatures of wave erosion in Titan’s coasts
Signatures of wave erosion in Titan’s coasts
 
Clinical periodontology and implant dentistry 2003.pdf
Clinical periodontology and implant dentistry 2003.pdfClinical periodontology and implant dentistry 2003.pdf
Clinical periodontology and implant dentistry 2003.pdf
 

Short talk on 2 cognitive biases and reproducibility

  • 1. Cognitive biases can lead to poor reproducibility and replicability of science Dorothy V. M. Bishop Professor of Developmental Neuropsychology University of Oxford @deevybee
  • 2. Two cognitive biases that make it hard to do science well • Schemata: • see meaning in random noise • communicate using simplified narratives • Confirmation bias: selective attention/memory • Failure to understand probability • Misunderstand p-values; many false positives • Misunderstand sampling error: many false negatives • Asymmetric moral judgements
  • 3. • Cherry-picking may not be deliberate • We find it much easier to process and remember information that agrees with our viewpoint Confirmation bias: particular problem in citations 3
  • 4. 4 How common is this in practice? Even when there are trial registries, we get publication bias de Vries, Y. A., et al (2018). The cumulative effect of reporting and citation biases on the apparent efficacy of treatments: the case of depression. Psychological Medicine 48, 2453-2455. doi:10.1017/S0033291718001873
  • 5. 5 How common is this in practice? Even when there are trial registries, we get publication bias de Vries, Y. A., et al (2018). The cumulative effect of reporting and citation biases on the apparent efficacy of treatments: the case of depression. Psychological Medicine 48, 2453-2455. doi:10.1017/S0033291718001873 We only know these studies exist because they are in a registry!
  • 6. 6 Publication bias and reporting bias conspire to mislead us de Vries, Y. A., et al (2018). The cumulative effect of reporting and citation biases on the apparent efficacy of treatments: the case of depression. Psychological Medicine 48, 2453-2455. doi:10.1017/S0033291718001873
  • 7. Inheritance of bias • When we read a peer-reviewed paper, we tend to trust the citations that back up a point • When we come to write our own paper, we cite the same materials • If prior papers only cite materials agreeing with a viewpoint, that viewpoint gets entrenched • You won’t know – unless you explicitly search – that there are other papers that give a different picture 7
  • 8. 8 ADVICE TO THOSE STARTING OUT IN RESEARCH
  • 9. Errors of statistical reasoning Failure to understand p-values leads to p-hacking 9
  • 10. 10 I test 16 compounds to see if they improve memory in rats Should I be excited? Error bars show SE.
  • 11. http://deevybee.blogspot.com/2016/01/the-amazing-significo-why-researchers.html P-hacking Simple explainer using poker Probability from unbiased deck of cards = 1 in 50 • If magician tells you he’ll deal you ‘3 of a kind’, and he does so, you should be impressed 3 of a kind • If magician deals 50 hands, and one of them is ‘3 of a kind’, you should not be impressed ‘Surprisingness’ of a result only interpretable in context of full dataset
  • 12. http://deevybee.blogspot.com/2016/01/the-amazing-significo-why-researchers.html P-hacking Simple explainer using poker Probability from unbiased deck of cards = 1 in 50 • If magician tells you he’ll deal you ‘3 of a kind’, and he does so, you should be impressed 3 of a kind • If magician deals 50 hands, and one of them is ‘3 of a kind’, you should not be impressed ‘Surprisingness’ of a result only interpretable in context of full dataset
  • 13. http://deevybee.blogspot.com/2016/01/the-amazing-significo-why-researchers.html P-hacking Simple explainer using poker Probability from unbiased deck of cards = 1 in 50 • If magician tells you he’ll deal you ‘3 of a kind’, and he does so, you should be impressed 3 of a kind • If magician deals 50 hands, and one of them is ‘3 of a kind’, you should not be impressed ‘Surprisingness’ of a result only interpretable in context of full dataset
  • 14. 14 Key error is to treat a p-value as an indicator of importance of a finding, regardless of context I’m not testing whether J is ‘significant’ here – I’m testing whether any of 16 compounds is ‘significant’ Relevant probability is 1– (probability that all compounds are ‘nonsignificant’) = 1 - .95^16 = 0.60 Simulated data from random normal distribution with mean = 0
  • 15. 15 To understand p-values: SIMULATED DATA “Just as lab scientists are not allowed to handle dangerous substances without safety training, researchers should not be allowed anywhere near a P value or similar measure of statistical probability until they have demonstrated understanding of what it means.” Bishop DVM. Nature 583, World view: How scientists can stop fooling themselves. July 23 2020
  • 16. 16 Professor Dorothy Bishop, FRS, FMedSci, FBA, Department of Experimental Psychology, Anna Watts Building, Woodstock Road, Oxford, OX2 6GG. @deevybee http://deevybee.blogspot.com/2012/11/bishopblog-catalogue- updated-24th-nov.html https://www.slideshare.net/deevybishop https://orcid.org/0000-0002-2448-4033