Downloaded 25 times
More Related Content
Similar to Fixing Science: The Replicability Crisis
Fixing Science: The Replicability Crisis
- 1. Alex.Holcombe@sydney.edu.au School of Psychology @ceptional FixingScience: The Replicability Crisis http://www.slideshare.net/holcombea/ 1Wednesday, 1 May 13
- 2. In the courseof describing the scientific method, a popular second- grade textbook instructs students that "Experiments should be done more than once". But in the professional scientific literature, the proportion of published studies that report replications are very small. While individual research articles typically report statistics designed to insure that the false-positive rate is below a nominal value (e.g, .05), common forms of bias in scientific practice can inflate the false- positive rate to much higher levels. For this and other reasons, the second-graders are right: without replications, it is difficult to know whether to believe individual results. Science now has the means to solve the problem, by taking advantage of new technology (the "Internet"). We are in a new era of cheap, open access, unlimited publishing. Nevertheless, academia continues to actively suppress replication studies. As this gradually changes, we will see not only a continued increase in the demand for statisticians, but also the emergence of a new class of data re- analysts. From my perspective as a psychological scientist, I will describe new initiatives that are coaxing replication studies, and raw data, out of the closet and onto the internet. Abstract 2Wednesday, 1 May 13
- 3. The Replicability Crisis Ruleamong early-stage venture capital firms that “at least 50% of published studies, even those in top- tier academic journals, can't be repeated with the same conclusions by an industrial lab” - Prinz, Schlange, & Asadullah. Nature Rev. Drug Discov. 10, 712 (2011) Bayer HealthCare :only about 25% of published preclinical studies could be validated to the point at which projects could continue Amgen Fifty-three papers were deemed ‘landmark’ studies (see ‘Reproducibility of research findings’)... scientific findings were confirmed in only 6 (11%) cases 3Wednesday, 1 May 13
- 4. Reasons for TheReplicability Crisis • Errors • Fraud • Publication bias by researchers • Publication bias by journals • Researchers p-hacking Statistical Flukes } 4Wednesday, 1 May 13
- 5. http://www.nytimes.com/2013/04/19/opinion/krugman-the-excel-depression.html?_r=0 • Omitted somedata • Used unusual, questionable statistics • Made an Excel error ERRORS 5Wednesday, 1 May 13
- 6. Central to thisinitiative is a checklist intended to prompt authors to disclose technical and statistical information in their submissions and to encourage referees to consider aspects important for research reproducibility. 6Wednesday, 1 May 13
- 7. Why science isself-correcting There's no point in scientific misconduct; it is always found. Published on August 10, 2010 by Art Markman, Ph.D. in Ulterior Motives Because scientists are always repeating each other's experiments, it is hard for a fictitious result to hang on for very long. “three unidentified young researchers as the whistleblowers for the case, and implies that these whistleblowers spent months making observations of Stapel and his work before they concluded that something actually was wrong” http:// en.wikipedia.org/wiki/Diederik_Stapel FRAUD 7Wednesday, 1 May 13
- 8. p-values Compare two groups(of people/rats/molecules given different treatments). Null hypothesis: That the treatment had no effect on the measure of interest (e.g., cancer rate) Calculate the probability of the observed difference between the groups, assuming the treatment had no effect. Is the probability less than .05? Then it’s statistically significant Statistical Flukes 8Wednesday, 1 May 13
- 9. Publication Bias bias introducedinto the scientific literature by selective publication — chiefly by a tendency to publish positive results but not to publish negative or nonconfirmatory results. 9Wednesday, 1 May 13
- 10.
- 11. Publication bias • Byjournals • “Unfortunately, despite the tens of thousands of available journals, places to send negative results are exceedingly scarce.” • even for those that do, uphill battle to get accepted • By researchers • No glory (promotions, grants, prizes) for a negative result http://expertedge.journalexperts.com/ 2013/04/26/negative-results-the-dark- matter-of-research/ 11Wednesday, 1 May 13
- 12. Corollary 4: Thegreater the flexibility in designs, definitions, outcomes, and analytical modes in a scientific field, the less likely the research findings are to be true. Flexibility increases the potential for transforming what would be “negative” results into “positive” results. Corollary 6: The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true. Publication Bias “In summary, while we agree with Ioannidis that most research findings are false...” 12Wednesday, 1 May 13
- 13. Reasons for theReplicability Crisis • Errors • Fraud • Publication bias by researchers • Publication bias by journals • Researchers p-hacking Statistical Flukes } Partial solution: Conduct and publish replication studies 13Wednesday, 1 May 13
- 14. Publication bias: Replicationstudies •Difficult to publish non- replications and replications •Most journals only publish papers that “make a novel contribution” •Reviewers/editors tend to hold non-replicating manuscript to higher standard than original. •Bem •Little career incentive to publish a non-replication or a replication http://www.flickr.com/photos/nickperez/2569423078 t. magnum unpublished results files 14Wednesday, 1 May 13
- 15. “Which of thefollowing do you consider to be scientific fraud? 1) A scientist collects 100 observations in an experiment and discards the 80 that run counter to his desired outcome 2) A scientist runs ten experiments then selectively writes up the two that produced statistically significant effects 3) A journal reviews ten papers on the same topic and selectively publishes the two that reported statistically significant effects” Dr. Chris Chambers http://www.scilogs.com/sifting_the_evidence/tackling-the-f-word/ What’s fraud and what’s publication bias? 15Wednesday, 1 May 13
- 16. Reasons for theReplicability Crisis • Errors • Fraud • Publication bias by researchers • Publication bias by journals • Researchers p-hacking Statistical Flukes } Partial solution: Conduct and publish replication studies 16Wednesday, 1 May 13
- 17. Barriers to publishingreplications and failed- replications • No glory in publishing a replication • Few journals publish replications • usually uphill battle even with those that do • The wrath of the original researcher 17Wednesday, 1 May 13
- 18.
- 19.
- 20.
- 21.
- 22. File-drawer fixes • Journalsthat don’t reject replications as being uninteresting or unimportant • Pre-registration of study designs and analysis methods • Brief reporting of replications ✔•◦ ✔ ◦ ◦ ◦◦ ✔ 22Wednesday, 1 May 13
- 23. • Collect data,check for statistical significance, collect more data • Analyse multiple measures individually 23Wednesday, 1 May 13
- 24. • Pre-registration ofstudy designs and analysis methods 24Wednesday, 1 May 13
- 25. File-drawer fixes • Journalsthat don’t reject replications as being uninteresting or unimportant • Pre-registration of study designs and analysis methods • Brief reporting of replications ✔•◦ ✔ ◦ ◦ ◦◦ ✔ 25Wednesday, 1 May 13
- 26. Registered Replication Reports 1.Authors plan a replication study 2. They submit an introduction and methods section 3. Sent to reviewers, including author of to-be-replicated article 4. Editor decides whether to accept/reject, based on: 1. Reviewer comments regarding the proposed protocol 2. Importance of original study, judged by argument in the introduction, number of citations of original, reviewer comments 5. The Intro, Method and analysis plan, and reviewer comments are posted on the journal website 6. After the results come in, the authors submit a conventional results and discussion section and that together with the raw data are posted, yielding the complete publication ✔ Dan Simons ✔✔ http://www.psychologicalscience.org/index.php/replication 26Wednesday, 1 May 13
- 27. • Original authorsigned off on it, so can’t complain / hate the replication authors as much. • Good way to start for a new PhD student, anyone planning to build on some already-published results • Will post the raw data • Will facilitate, publish meta-analyses when replications accrue • Reduce the incentive to publish flashy, headline-grabbing but unreliable studies? ✔✔✔ Registered Replication Reports 27Wednesday, 1 May 13
- 28. Reasons for theReplicability Crisis • Errors • Fraud • Publication bias by researchers • Publication bias by journals • Researchers p-hacking Statistical Flukes } Partial solution: Conduct and publish replication studies 28Wednesday, 1 May 13
- 29. Open Research • Postall experiment software online • As data comes in, put on web • Electronic lab notebook • Post all data analyses and calculations • Papers written via open collaborative documents on the web The Tasman Declaration on Open Research https://sites.google.com/site/nzauopenresearch/sign-up-to-the-tasman-declaration 29Wednesday, 1 May 13
- 30. Open Data NHMRC: Thenext steps will be improving public and other researchers’ access to publicly funded data. https://theconversation.edu.au/all-research-funded-by-nhmrc-to-be-accessible-free-of-charge-5486 30Wednesday, 1 May 13
- 31. • Errors • Fraud •Publication bias by researchers • Publication bias by journals • Researchers p-hacking Statistical Flukes } @ceptionalhttp://www.slideshare.net/holcombea/ The Replicability Crisis 31Wednesday, 1 May 13