SlideShare a Scribd company logo
Macquarie University 4 Dec 2015
A new era of large-scale
replication and collaboration?
Alex.Holcombe@sydney.edu.au
@ceptional
http://www.slideshare.net/holcombea/
• Open access
• P-hacking
• Preregistration, Registered Reports
• Open data, open methods
• Registered Replication Reports
Macquarie University 4 Dec 2015 2
Macquarie University 4 Dec 2015 3
Open access - Research funders will mandate it.Will
we scholars shape the system of the future, or passively let
publishers do it in their own interest?
• Green or Gold
• Who pays?
• Institution, granting agency, library (from subscription
savings)
• Everything technically adequate should be published
• “contribution to the field” is subjective
• Competition for Nature on down wasteful, distorting
• Frontiers? PeerJ? Ubiquity Press?
Macquarie University 4 Dec 2015 4
Our average base Article Processing Charge (APC) is only £300 (€375, $500) and this is the only thing
we charge.
5
Slide courtesy Chris Chambers (Cardiff)
Science publishing has an incentive problem
Macquarie University 4 Dec 2015
"This one's Psych Science. 1998. Put
your money on ‘will replicate’?”
"What, you think I was born
yesterday?”
"Fine," muttered the bookie.
#AcademicNoir
Macquarie University 4 Dec 2015
P-hacking Simmons, J. P., Nelson, L. D., &
Simonsohn, U. (2011). False-
positive psychology undisclosed
flexibility in data collection and
analysis allows presenting anything
as significant. Psychological
Science
Macquarie University 4 Dec 2015
False positive zones
- researcher degrees of freedom high (p-hacking)
- victory declared whenever null hypothesis is rejected,
where size of effect, plausibility considered unimportant.
Where science is not self-correcting
The newer the domain, the more cutting-edge the
technology, the less-defined the expected analyses, and the
greater the opportunities for p-hacking
Macquarie University 4 Dec 2015
courtesy Chris Chambers (Cardiff)
Macquarie University 4 Dec 2015
Well, Mr. Hacker, your luck is about to run out.
Down at the journals, they’re running a new game.
They call it ‘preregistration’.
CC-BY Alex Holcombe
https://twitter.com/ceptional/status/663850868396044288
Macquarie University 4 Dec 2015
11
courtesy Chris Chambers (Cardiff)
Macquarie University 4 Dec 2015
12
courtesy Chris Chambers (Cardiff)
Macquarie University 4 Dec 2015
13
https://osf.io/tvyxz/wiki/home/
Macquarie University 4 Dec 2015
14
https://osf.io/tvyxz/wiki/home/
Macquarie University 4 Dec 2015
15
https://osf.io/tvyxz/wiki/home/
100% P-hacking Free
Here, check our
numbers.
Here’s how you
can replicate our
result.
The badges that are fixing science
Macquarie University 4 Dec 2015
"This one's Psych Science. 1998. Put
your money on ‘will replicate’?”
"What, you think I was born
yesterday?”
"Fine," muttered the bookie.
Macquarie University 4 Dec 2015 17
Failed to replicate my result?
Have you
considered
the unknown
moderators?
Macquarie University 4 Dec 2015 18
Any one of us alone, he’ll say we messed up.
But if we all run the replication together, we
can show the big guy we’re right.
Macquarie University 4 Dec 2015
Meta-analysis forest plot
Macquarie University 4 Dec 2015
Meta-analysis forest plot
Failed to replicate my
result?
Have you considered
the unknown
moderators?
You messed
up!
Macquarie University 4 Dec 2015
Macquarie University 4 Dec 2015
Macquarie University 4 Dec 2015
Macquarie University 4 Dec 2015
Registered studies
Large-scale replication
Open data
Open protocol and code
A new era of
large-scale replication
and collaboration?
Macquarie University 4 Dec 2015

More Related Content

What's hot

Innovating with Open Knowledge
Innovating with Open KnowledgeInnovating with Open Knowledge
Innovating with Open Knowledge
Lorna Campbell
 
Access to Research Data - Westminster Briefing
Access to Research Data - Westminster BriefingAccess to Research Data - Westminster Briefing
Access to Research Data - Westminster Briefing
Danny Kingsley
 
IFMSA EuRegMe Workshop 2015
IFMSA EuRegMe Workshop 2015IFMSA EuRegMe Workshop 2015
IFMSA EuRegMe Workshop 2015
Right to Research
 
So, what's it all about then? Why we share research data
So, what's it all about then? Why we share research dataSo, what's it all about then? Why we share research data
So, what's it all about then? Why we share research data
Danny Kingsley
 
UKSG Conference 2017 Breakout - Shifting sands – changing academic library sk...
UKSG Conference 2017 Breakout - Shifting sands – changing academic library sk...UKSG Conference 2017 Breakout - Shifting sands – changing academic library sk...
UKSG Conference 2017 Breakout - Shifting sands – changing academic library sk...
UKSG: connecting the knowledge community
 
Getting an Octopus into a String Bag - The complexity of communicating with t...
Getting an Octopus into a String Bag - The complexity of communicating with t...Getting an Octopus into a String Bag - The complexity of communicating with t...
Getting an Octopus into a String Bag - The complexity of communicating with t...
Danny Kingsley
 
Wikimedia historic perspective
Wikimedia historic perspectiveWikimedia historic perspective
Wikimedia historic perspective
Gerard Meijssen
 
Engaging participation from the chemistry community
Engaging participation from the chemistry communityEngaging participation from the chemistry community
What is ‘research impact’ in an interconnected world?
What is ‘research impact’ in an interconnected world?What is ‘research impact’ in an interconnected world?
What is ‘research impact’ in an interconnected world?
Danny Kingsley
 
1st Network-of-BioThings Hackathon
1st Network-of-BioThings Hackathon1st Network-of-BioThings Hackathon
1st Network-of-BioThings Hackathon
Michel Dumontier
 
OSFair2017 | Barriers to Open Science for junior researchers
OSFair2017 | Barriers to Open Science for junior researchersOSFair2017 | Barriers to Open Science for junior researchers
OSFair2017 | Barriers to Open Science for junior researchers
Open Science Fair
 
The OSC at Cambridge - a lightning tour
The OSC at Cambridge - a lightning tourThe OSC at Cambridge - a lightning tour
The OSC at Cambridge - a lightning tour
Danny Kingsley
 
Watch out, it's behind you: publishers' tactics and the challenge they pose f...
Watch out, it's behind you: publishers' tactics and the challenge they pose f...Watch out, it's behind you: publishers' tactics and the challenge they pose f...
Watch out, it's behind you: publishers' tactics and the challenge they pose f...
Danny Kingsley
 

What's hot (13)

Innovating with Open Knowledge
Innovating with Open KnowledgeInnovating with Open Knowledge
Innovating with Open Knowledge
 
Access to Research Data - Westminster Briefing
Access to Research Data - Westminster BriefingAccess to Research Data - Westminster Briefing
Access to Research Data - Westminster Briefing
 
IFMSA EuRegMe Workshop 2015
IFMSA EuRegMe Workshop 2015IFMSA EuRegMe Workshop 2015
IFMSA EuRegMe Workshop 2015
 
So, what's it all about then? Why we share research data
So, what's it all about then? Why we share research dataSo, what's it all about then? Why we share research data
So, what's it all about then? Why we share research data
 
UKSG Conference 2017 Breakout - Shifting sands – changing academic library sk...
UKSG Conference 2017 Breakout - Shifting sands – changing academic library sk...UKSG Conference 2017 Breakout - Shifting sands – changing academic library sk...
UKSG Conference 2017 Breakout - Shifting sands – changing academic library sk...
 
Getting an Octopus into a String Bag - The complexity of communicating with t...
Getting an Octopus into a String Bag - The complexity of communicating with t...Getting an Octopus into a String Bag - The complexity of communicating with t...
Getting an Octopus into a String Bag - The complexity of communicating with t...
 
Wikimedia historic perspective
Wikimedia historic perspectiveWikimedia historic perspective
Wikimedia historic perspective
 
Engaging participation from the chemistry community
Engaging participation from the chemistry communityEngaging participation from the chemistry community
Engaging participation from the chemistry community
 
What is ‘research impact’ in an interconnected world?
What is ‘research impact’ in an interconnected world?What is ‘research impact’ in an interconnected world?
What is ‘research impact’ in an interconnected world?
 
1st Network-of-BioThings Hackathon
1st Network-of-BioThings Hackathon1st Network-of-BioThings Hackathon
1st Network-of-BioThings Hackathon
 
OSFair2017 | Barriers to Open Science for junior researchers
OSFair2017 | Barriers to Open Science for junior researchersOSFair2017 | Barriers to Open Science for junior researchers
OSFair2017 | Barriers to Open Science for junior researchers
 
The OSC at Cambridge - a lightning tour
The OSC at Cambridge - a lightning tourThe OSC at Cambridge - a lightning tour
The OSC at Cambridge - a lightning tour
 
Watch out, it's behind you: publishers' tactics and the challenge they pose f...
Watch out, it's behind you: publishers' tactics and the challenge they pose f...Watch out, it's behind you: publishers' tactics and the challenge they pose f...
Watch out, it's behind you: publishers' tactics and the challenge they pose f...
 

Viewers also liked

DUBAI INFRAESTRUCTURA
DUBAI INFRAESTRUCTURADUBAI INFRAESTRUCTURA
DUBAI INFRAESTRUCTURA
equipocompu
 
Rayan Shipping Agencies
Rayan Shipping AgenciesRayan Shipping Agencies
Rayan Shipping Agencies
Saeed G Baterian
 
Biologia estructura
Biologia estructuraBiologia estructura
Biologia estructura
Karla Paguay
 
Metals
MetalsMetals
C.V.
C.V.C.V.
Leo mobile vs iphone
Leo mobile vs iphoneLeo mobile vs iphone
Leo mobile vs iphone
Zaimi gee
 
The gavel gamble_ Litigation emerges as an asset class
The gavel gamble_ Litigation emerges as an asset classThe gavel gamble_ Litigation emerges as an asset class
The gavel gamble_ Litigation emerges as an asset class
David Sedgwick
 
Facebook
FacebookFacebook
Facebook
姍熺 謝
 
Local crash lawyer emphasizes consequences of distracted driving
Local crash lawyer emphasizes consequences of distracted drivingLocal crash lawyer emphasizes consequences of distracted driving
Local crash lawyer emphasizes consequences of distracted driving
Naida Mink
 

Viewers also liked (9)

DUBAI INFRAESTRUCTURA
DUBAI INFRAESTRUCTURADUBAI INFRAESTRUCTURA
DUBAI INFRAESTRUCTURA
 
Rayan Shipping Agencies
Rayan Shipping AgenciesRayan Shipping Agencies
Rayan Shipping Agencies
 
Biologia estructura
Biologia estructuraBiologia estructura
Biologia estructura
 
Metals
MetalsMetals
Metals
 
C.V.
C.V.C.V.
C.V.
 
Leo mobile vs iphone
Leo mobile vs iphoneLeo mobile vs iphone
Leo mobile vs iphone
 
The gavel gamble_ Litigation emerges as an asset class
The gavel gamble_ Litigation emerges as an asset classThe gavel gamble_ Litigation emerges as an asset class
The gavel gamble_ Litigation emerges as an asset class
 
Facebook
FacebookFacebook
Facebook
 
Local crash lawyer emphasizes consequences of distracted driving
Local crash lawyer emphasizes consequences of distracted drivingLocal crash lawyer emphasizes consequences of distracted driving
Local crash lawyer emphasizes consequences of distracted driving
 

Similar to open access, p-hacking, registered reports, registered replication reports

Open Learning Analytics LSAC2018
Open Learning Analytics LSAC2018Open Learning Analytics LSAC2018
Open Learning Analytics LSAC2018
Ian Dolphin
 
Union Suisse Summer :: Science and Technology for a better society
Union Suisse  Summer :: Science and Technology for a better societyUnion Suisse  Summer :: Science and Technology for a better society
Union Suisse Summer :: Science and Technology for a better society
Catalyx
 
Uts talk openaccess_week_24_october
Uts talk openaccess_week_24_octoberUts talk openaccess_week_24_october
Uts talk openaccess_week_24_october
Alex Holcombe
 
Open Access & Research Data Management for Early Career Researchers at Imperi...
Open Access & Research Data Management for Early Career Researchers at Imperi...Open Access & Research Data Management for Early Career Researchers at Imperi...
Open Access & Research Data Management for Early Career Researchers at Imperi...
Imperial College London Library
 
HKU Data Curation MLIM7350 Class 7
HKU Data Curation MLIM7350 Class 7HKU Data Curation MLIM7350 Class 7
HKU Data Curation MLIM7350 Class 7
Scott Edmunds
 
Iniciativa Internacional Cambridge-CONACYT
Iniciativa Internacional Cambridge-CONACYTIniciativa Internacional Cambridge-CONACYT
Iniciativa Internacional Cambridge-CONACYT
Mexico Innova
 
How to Master Science Crowdfunding
How to Master Science CrowdfundingHow to Master Science Crowdfunding
How to Master Science Crowdfunding
Katja Reuter, PhD
 
Being open (source) in the traditionally secretive field of quant finance.
Being open (source) in the traditionally secretive field of quant finance.Being open (source) in the traditionally secretive field of quant finance.
Being open (source) in the traditionally secretive field of quant finance.
Quantopian
 
Data Science Meets Biomedicine, Does Anything Change
Data Science Meets Biomedicine, Does Anything ChangeData Science Meets Biomedicine, Does Anything Change
Data Science Meets Biomedicine, Does Anything Change
Philip Bourne
 
Managing and sharing data
Managing and sharing dataManaging and sharing data
Managing and sharing data
Sarah Jones
 
Adac intro
Adac introAdac intro
Adac intro
gpfigueredo
 
Is ‘Open Science’ a solution or a threat?
Is ‘Open Science’ a solution or a threat?Is ‘Open Science’ a solution or a threat?
Is ‘Open Science’ a solution or a threat?
Danny Kingsley
 
Pre equipment sharingandoa_v1_20160407
Pre equipment sharingandoa_v1_20160407Pre equipment sharingandoa_v1_20160407
Pre equipment sharingandoa_v1_20160407
Marta Teperek
 
Sadler niso-apr13
Sadler niso-apr13Sadler niso-apr13
UKSG Conference 2017 Breakout - Advancing the Research Paper of the Future: c...
UKSG Conference 2017 Breakout - Advancing the Research Paper of the Future: c...UKSG Conference 2017 Breakout - Advancing the Research Paper of the Future: c...
UKSG Conference 2017 Breakout - Advancing the Research Paper of the Future: c...
UKSG: connecting the knowledge community
 
Open Access at the Coal Face - Attitudes and Practical Responses (DARTS4)
Open Access at the Coal Face - Attitudes and Practical Responses (DARTS4)Open Access at the Coal Face - Attitudes and Practical Responses (DARTS4)
Open Access at the Coal Face - Attitudes and Practical Responses (DARTS4)
Yvonne Budden
 
Data Science and AI in Biomedicine: The World has Changed
Data Science and AI in Biomedicine: The World has ChangedData Science and AI in Biomedicine: The World has Changed
Data Science and AI in Biomedicine: The World has Changed
Philip Bourne
 
"Open Access at the Coal Face: attitudes and practical responses" Yvonne Budd...
"Open Access at the Coal Face: attitudes and practical responses" Yvonne Budd..."Open Access at the Coal Face: attitudes and practical responses" Yvonne Budd...
"Open Access at the Coal Face: attitudes and practical responses" Yvonne Budd...
ARLGSW
 
Soren J Jorgensen - Innovation Center Denmark - Stanford Engineering - Jan 4 ...
Soren J Jorgensen - Innovation Center Denmark - Stanford Engineering - Jan 4 ...Soren J Jorgensen - Innovation Center Denmark - Stanford Engineering - Jan 4 ...
Soren J Jorgensen - Innovation Center Denmark - Stanford Engineering - Jan 4 ...
Burton Lee
 
Digitization: Why YOU Should Be Leading The Charge
Digitization: Why YOU Should Be Leading The ChargeDigitization: Why YOU Should Be Leading The Charge
Digitization: Why YOU Should Be Leading The Charge
Mēgan Oliver, MLIS
 

Similar to open access, p-hacking, registered reports, registered replication reports (20)

Open Learning Analytics LSAC2018
Open Learning Analytics LSAC2018Open Learning Analytics LSAC2018
Open Learning Analytics LSAC2018
 
Union Suisse Summer :: Science and Technology for a better society
Union Suisse  Summer :: Science and Technology for a better societyUnion Suisse  Summer :: Science and Technology for a better society
Union Suisse Summer :: Science and Technology for a better society
 
Uts talk openaccess_week_24_october
Uts talk openaccess_week_24_octoberUts talk openaccess_week_24_october
Uts talk openaccess_week_24_october
 
Open Access & Research Data Management for Early Career Researchers at Imperi...
Open Access & Research Data Management for Early Career Researchers at Imperi...Open Access & Research Data Management for Early Career Researchers at Imperi...
Open Access & Research Data Management for Early Career Researchers at Imperi...
 
HKU Data Curation MLIM7350 Class 7
HKU Data Curation MLIM7350 Class 7HKU Data Curation MLIM7350 Class 7
HKU Data Curation MLIM7350 Class 7
 
Iniciativa Internacional Cambridge-CONACYT
Iniciativa Internacional Cambridge-CONACYTIniciativa Internacional Cambridge-CONACYT
Iniciativa Internacional Cambridge-CONACYT
 
How to Master Science Crowdfunding
How to Master Science CrowdfundingHow to Master Science Crowdfunding
How to Master Science Crowdfunding
 
Being open (source) in the traditionally secretive field of quant finance.
Being open (source) in the traditionally secretive field of quant finance.Being open (source) in the traditionally secretive field of quant finance.
Being open (source) in the traditionally secretive field of quant finance.
 
Data Science Meets Biomedicine, Does Anything Change
Data Science Meets Biomedicine, Does Anything ChangeData Science Meets Biomedicine, Does Anything Change
Data Science Meets Biomedicine, Does Anything Change
 
Managing and sharing data
Managing and sharing dataManaging and sharing data
Managing and sharing data
 
Adac intro
Adac introAdac intro
Adac intro
 
Is ‘Open Science’ a solution or a threat?
Is ‘Open Science’ a solution or a threat?Is ‘Open Science’ a solution or a threat?
Is ‘Open Science’ a solution or a threat?
 
Pre equipment sharingandoa_v1_20160407
Pre equipment sharingandoa_v1_20160407Pre equipment sharingandoa_v1_20160407
Pre equipment sharingandoa_v1_20160407
 
Sadler niso-apr13
Sadler niso-apr13Sadler niso-apr13
Sadler niso-apr13
 
UKSG Conference 2017 Breakout - Advancing the Research Paper of the Future: c...
UKSG Conference 2017 Breakout - Advancing the Research Paper of the Future: c...UKSG Conference 2017 Breakout - Advancing the Research Paper of the Future: c...
UKSG Conference 2017 Breakout - Advancing the Research Paper of the Future: c...
 
Open Access at the Coal Face - Attitudes and Practical Responses (DARTS4)
Open Access at the Coal Face - Attitudes and Practical Responses (DARTS4)Open Access at the Coal Face - Attitudes and Practical Responses (DARTS4)
Open Access at the Coal Face - Attitudes and Practical Responses (DARTS4)
 
Data Science and AI in Biomedicine: The World has Changed
Data Science and AI in Biomedicine: The World has ChangedData Science and AI in Biomedicine: The World has Changed
Data Science and AI in Biomedicine: The World has Changed
 
"Open Access at the Coal Face: attitudes and practical responses" Yvonne Budd...
"Open Access at the Coal Face: attitudes and practical responses" Yvonne Budd..."Open Access at the Coal Face: attitudes and practical responses" Yvonne Budd...
"Open Access at the Coal Face: attitudes and practical responses" Yvonne Budd...
 
Soren J Jorgensen - Innovation Center Denmark - Stanford Engineering - Jan 4 ...
Soren J Jorgensen - Innovation Center Denmark - Stanford Engineering - Jan 4 ...Soren J Jorgensen - Innovation Center Denmark - Stanford Engineering - Jan 4 ...
Soren J Jorgensen - Innovation Center Denmark - Stanford Engineering - Jan 4 ...
 
Digitization: Why YOU Should Be Leading The Charge
Digitization: Why YOU Should Be Leading The ChargeDigitization: Why YOU Should Be Leading The Charge
Digitization: Why YOU Should Be Leading The Charge
 

Recently uploaded

The cost of acquiring information by natural selection
The cost of acquiring information by natural selectionThe cost of acquiring information by natural selection
The cost of acquiring information by natural selection
Carl Bergstrom
 
8.Isolation of pure cultures and preservation of cultures.pdf
8.Isolation of pure cultures and preservation of cultures.pdf8.Isolation of pure cultures and preservation of cultures.pdf
8.Isolation of pure cultures and preservation of cultures.pdf
by6843629
 
Juaristi, Jon. - El canon espanol. El legado de la cultura española a la civi...
Juaristi, Jon. - El canon espanol. El legado de la cultura española a la civi...Juaristi, Jon. - El canon espanol. El legado de la cultura española a la civi...
Juaristi, Jon. - El canon espanol. El legado de la cultura española a la civi...
frank0071
 
Candidate young stellar objects in the S-cluster: Kinematic analysis of a sub...
Candidate young stellar objects in the S-cluster: Kinematic analysis of a sub...Candidate young stellar objects in the S-cluster: Kinematic analysis of a sub...
Candidate young stellar objects in the S-cluster: Kinematic analysis of a sub...
Sérgio Sacani
 
Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...
Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...
Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...
PsychoTech Services
 
Microbiology of Central Nervous System INFECTIONS.pdf
Microbiology of Central Nervous System INFECTIONS.pdfMicrobiology of Central Nervous System INFECTIONS.pdf
Microbiology of Central Nervous System INFECTIONS.pdf
sammy700571
 
Discovery of An Apparent Red, High-Velocity Type Ia Supernova at 𝐳 = 2.9 wi...
Discovery of An Apparent Red, High-Velocity Type Ia Supernova at  𝐳 = 2.9  wi...Discovery of An Apparent Red, High-Velocity Type Ia Supernova at  𝐳 = 2.9  wi...
Discovery of An Apparent Red, High-Velocity Type Ia Supernova at 𝐳 = 2.9 wi...
Sérgio Sacani
 
MICROBIAL INTERACTION PPT/ MICROBIAL INTERACTION AND THEIR TYPES // PLANT MIC...
MICROBIAL INTERACTION PPT/ MICROBIAL INTERACTION AND THEIR TYPES // PLANT MIC...MICROBIAL INTERACTION PPT/ MICROBIAL INTERACTION AND THEIR TYPES // PLANT MIC...
MICROBIAL INTERACTION PPT/ MICROBIAL INTERACTION AND THEIR TYPES // PLANT MIC...
ABHISHEK SONI NIMT INSTITUTE OF MEDICAL AND PARAMEDCIAL SCIENCES , GOVT PG COLLEGE NOIDA
 
HOW DO ORGANISMS REPRODUCE?reproduction part 1
HOW DO ORGANISMS REPRODUCE?reproduction part 1HOW DO ORGANISMS REPRODUCE?reproduction part 1
HOW DO ORGANISMS REPRODUCE?reproduction part 1
Shashank Shekhar Pandey
 
Summary Of transcription and Translation.pdf
Summary Of transcription and Translation.pdfSummary Of transcription and Translation.pdf
Summary Of transcription and Translation.pdf
vadgavevedant86
 
Evidence of Jet Activity from the Secondary Black Hole in the OJ 287 Binary S...
Evidence of Jet Activity from the Secondary Black Hole in the OJ 287 Binary S...Evidence of Jet Activity from the Secondary Black Hole in the OJ 287 Binary S...
Evidence of Jet Activity from the Secondary Black Hole in the OJ 287 Binary S...
Sérgio Sacani
 
Tissue fluids_etiology_volume regulation_pressure.pptx
Tissue fluids_etiology_volume regulation_pressure.pptxTissue fluids_etiology_volume regulation_pressure.pptx
Tissue fluids_etiology_volume regulation_pressure.pptx
muralinath2
 
Methods of grain storage Structures in India.pdf
Methods of grain storage Structures in India.pdfMethods of grain storage Structures in India.pdf
Methods of grain storage Structures in India.pdf
PirithiRaju
 
Travis Hills of MN is Making Clean Water Accessible to All Through High Flux ...
Travis Hills of MN is Making Clean Water Accessible to All Through High Flux ...Travis Hills of MN is Making Clean Water Accessible to All Through High Flux ...
Travis Hills of MN is Making Clean Water Accessible to All Through High Flux ...
Travis Hills MN
 
EWOCS-I: The catalog of X-ray sources in Westerlund 1 from the Extended Weste...
EWOCS-I: The catalog of X-ray sources in Westerlund 1 from the Extended Weste...EWOCS-I: The catalog of X-ray sources in Westerlund 1 from the Extended Weste...
EWOCS-I: The catalog of X-ray sources in Westerlund 1 from the Extended Weste...
Sérgio Sacani
 
Pests of Storage_Identification_Dr.UPR.pdf
Pests of Storage_Identification_Dr.UPR.pdfPests of Storage_Identification_Dr.UPR.pdf
Pests of Storage_Identification_Dr.UPR.pdf
PirithiRaju
 
Describing and Interpreting an Immersive Learning Case with the Immersion Cub...
Describing and Interpreting an Immersive Learning Case with the Immersion Cub...Describing and Interpreting an Immersive Learning Case with the Immersion Cub...
Describing and Interpreting an Immersive Learning Case with the Immersion Cub...
Leonel Morgado
 
JAMES WEBB STUDY THE MASSIVE BLACK HOLE SEEDS
JAMES WEBB STUDY THE MASSIVE BLACK HOLE SEEDSJAMES WEBB STUDY THE MASSIVE BLACK HOLE SEEDS
JAMES WEBB STUDY THE MASSIVE BLACK HOLE SEEDS
Sérgio Sacani
 
11.1 Role of physical biological in deterioration of grains.pdf
11.1 Role of physical biological in deterioration of grains.pdf11.1 Role of physical biological in deterioration of grains.pdf
11.1 Role of physical biological in deterioration of grains.pdf
PirithiRaju
 
Gadgets for management of stored product pests_Dr.UPR.pdf
Gadgets for management of stored product pests_Dr.UPR.pdfGadgets for management of stored product pests_Dr.UPR.pdf
Gadgets for management of stored product pests_Dr.UPR.pdf
PirithiRaju
 

Recently uploaded (20)

The cost of acquiring information by natural selection
The cost of acquiring information by natural selectionThe cost of acquiring information by natural selection
The cost of acquiring information by natural selection
 
8.Isolation of pure cultures and preservation of cultures.pdf
8.Isolation of pure cultures and preservation of cultures.pdf8.Isolation of pure cultures and preservation of cultures.pdf
8.Isolation of pure cultures and preservation of cultures.pdf
 
Juaristi, Jon. - El canon espanol. El legado de la cultura española a la civi...
Juaristi, Jon. - El canon espanol. El legado de la cultura española a la civi...Juaristi, Jon. - El canon espanol. El legado de la cultura española a la civi...
Juaristi, Jon. - El canon espanol. El legado de la cultura española a la civi...
 
Candidate young stellar objects in the S-cluster: Kinematic analysis of a sub...
Candidate young stellar objects in the S-cluster: Kinematic analysis of a sub...Candidate young stellar objects in the S-cluster: Kinematic analysis of a sub...
Candidate young stellar objects in the S-cluster: Kinematic analysis of a sub...
 
Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...
Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...
Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...
 
Microbiology of Central Nervous System INFECTIONS.pdf
Microbiology of Central Nervous System INFECTIONS.pdfMicrobiology of Central Nervous System INFECTIONS.pdf
Microbiology of Central Nervous System INFECTIONS.pdf
 
Discovery of An Apparent Red, High-Velocity Type Ia Supernova at 𝐳 = 2.9 wi...
Discovery of An Apparent Red, High-Velocity Type Ia Supernova at  𝐳 = 2.9  wi...Discovery of An Apparent Red, High-Velocity Type Ia Supernova at  𝐳 = 2.9  wi...
Discovery of An Apparent Red, High-Velocity Type Ia Supernova at 𝐳 = 2.9 wi...
 
MICROBIAL INTERACTION PPT/ MICROBIAL INTERACTION AND THEIR TYPES // PLANT MIC...
MICROBIAL INTERACTION PPT/ MICROBIAL INTERACTION AND THEIR TYPES // PLANT MIC...MICROBIAL INTERACTION PPT/ MICROBIAL INTERACTION AND THEIR TYPES // PLANT MIC...
MICROBIAL INTERACTION PPT/ MICROBIAL INTERACTION AND THEIR TYPES // PLANT MIC...
 
HOW DO ORGANISMS REPRODUCE?reproduction part 1
HOW DO ORGANISMS REPRODUCE?reproduction part 1HOW DO ORGANISMS REPRODUCE?reproduction part 1
HOW DO ORGANISMS REPRODUCE?reproduction part 1
 
Summary Of transcription and Translation.pdf
Summary Of transcription and Translation.pdfSummary Of transcription and Translation.pdf
Summary Of transcription and Translation.pdf
 
Evidence of Jet Activity from the Secondary Black Hole in the OJ 287 Binary S...
Evidence of Jet Activity from the Secondary Black Hole in the OJ 287 Binary S...Evidence of Jet Activity from the Secondary Black Hole in the OJ 287 Binary S...
Evidence of Jet Activity from the Secondary Black Hole in the OJ 287 Binary S...
 
Tissue fluids_etiology_volume regulation_pressure.pptx
Tissue fluids_etiology_volume regulation_pressure.pptxTissue fluids_etiology_volume regulation_pressure.pptx
Tissue fluids_etiology_volume regulation_pressure.pptx
 
Methods of grain storage Structures in India.pdf
Methods of grain storage Structures in India.pdfMethods of grain storage Structures in India.pdf
Methods of grain storage Structures in India.pdf
 
Travis Hills of MN is Making Clean Water Accessible to All Through High Flux ...
Travis Hills of MN is Making Clean Water Accessible to All Through High Flux ...Travis Hills of MN is Making Clean Water Accessible to All Through High Flux ...
Travis Hills of MN is Making Clean Water Accessible to All Through High Flux ...
 
EWOCS-I: The catalog of X-ray sources in Westerlund 1 from the Extended Weste...
EWOCS-I: The catalog of X-ray sources in Westerlund 1 from the Extended Weste...EWOCS-I: The catalog of X-ray sources in Westerlund 1 from the Extended Weste...
EWOCS-I: The catalog of X-ray sources in Westerlund 1 from the Extended Weste...
 
Pests of Storage_Identification_Dr.UPR.pdf
Pests of Storage_Identification_Dr.UPR.pdfPests of Storage_Identification_Dr.UPR.pdf
Pests of Storage_Identification_Dr.UPR.pdf
 
Describing and Interpreting an Immersive Learning Case with the Immersion Cub...
Describing and Interpreting an Immersive Learning Case with the Immersion Cub...Describing and Interpreting an Immersive Learning Case with the Immersion Cub...
Describing and Interpreting an Immersive Learning Case with the Immersion Cub...
 
JAMES WEBB STUDY THE MASSIVE BLACK HOLE SEEDS
JAMES WEBB STUDY THE MASSIVE BLACK HOLE SEEDSJAMES WEBB STUDY THE MASSIVE BLACK HOLE SEEDS
JAMES WEBB STUDY THE MASSIVE BLACK HOLE SEEDS
 
11.1 Role of physical biological in deterioration of grains.pdf
11.1 Role of physical biological in deterioration of grains.pdf11.1 Role of physical biological in deterioration of grains.pdf
11.1 Role of physical biological in deterioration of grains.pdf
 
Gadgets for management of stored product pests_Dr.UPR.pdf
Gadgets for management of stored product pests_Dr.UPR.pdfGadgets for management of stored product pests_Dr.UPR.pdf
Gadgets for management of stored product pests_Dr.UPR.pdf
 

open access, p-hacking, registered reports, registered replication reports

  • 1. Macquarie University 4 Dec 2015 A new era of large-scale replication and collaboration? Alex.Holcombe@sydney.edu.au @ceptional http://www.slideshare.net/holcombea/ • Open access • P-hacking • Preregistration, Registered Reports • Open data, open methods • Registered Replication Reports
  • 3. Macquarie University 4 Dec 2015 3 Open access - Research funders will mandate it.Will we scholars shape the system of the future, or passively let publishers do it in their own interest? • Green or Gold • Who pays? • Institution, granting agency, library (from subscription savings) • Everything technically adequate should be published • “contribution to the field” is subjective • Competition for Nature on down wasteful, distorting • Frontiers? PeerJ? Ubiquity Press?
  • 4. Macquarie University 4 Dec 2015 4 Our average base Article Processing Charge (APC) is only £300 (€375, $500) and this is the only thing we charge.
  • 5. 5 Slide courtesy Chris Chambers (Cardiff) Science publishing has an incentive problem
  • 6. Macquarie University 4 Dec 2015 "This one's Psych Science. 1998. Put your money on ‘will replicate’?” "What, you think I was born yesterday?” "Fine," muttered the bookie. #AcademicNoir
  • 7. Macquarie University 4 Dec 2015 P-hacking Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False- positive psychology undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science
  • 8. Macquarie University 4 Dec 2015 False positive zones - researcher degrees of freedom high (p-hacking) - victory declared whenever null hypothesis is rejected, where size of effect, plausibility considered unimportant. Where science is not self-correcting The newer the domain, the more cutting-edge the technology, the less-defined the expected analyses, and the greater the opportunities for p-hacking
  • 9. Macquarie University 4 Dec 2015 courtesy Chris Chambers (Cardiff)
  • 10. Macquarie University 4 Dec 2015 Well, Mr. Hacker, your luck is about to run out. Down at the journals, they’re running a new game. They call it ‘preregistration’. CC-BY Alex Holcombe https://twitter.com/ceptional/status/663850868396044288
  • 11. Macquarie University 4 Dec 2015 11 courtesy Chris Chambers (Cardiff)
  • 12. Macquarie University 4 Dec 2015 12 courtesy Chris Chambers (Cardiff)
  • 13. Macquarie University 4 Dec 2015 13 https://osf.io/tvyxz/wiki/home/
  • 14. Macquarie University 4 Dec 2015 14 https://osf.io/tvyxz/wiki/home/
  • 15. Macquarie University 4 Dec 2015 15 https://osf.io/tvyxz/wiki/home/ 100% P-hacking Free Here, check our numbers. Here’s how you can replicate our result. The badges that are fixing science
  • 16. Macquarie University 4 Dec 2015 "This one's Psych Science. 1998. Put your money on ‘will replicate’?” "What, you think I was born yesterday?” "Fine," muttered the bookie.
  • 17. Macquarie University 4 Dec 2015 17 Failed to replicate my result? Have you considered the unknown moderators?
  • 18. Macquarie University 4 Dec 2015 18 Any one of us alone, he’ll say we messed up. But if we all run the replication together, we can show the big guy we’re right.
  • 19. Macquarie University 4 Dec 2015 Meta-analysis forest plot
  • 20. Macquarie University 4 Dec 2015 Meta-analysis forest plot Failed to replicate my result? Have you considered the unknown moderators? You messed up!
  • 24. Macquarie University 4 Dec 2015 Registered studies Large-scale replication Open data Open protocol and code A new era of large-scale replication and collaboration?