Recommended
More Related Content
Similar to Ruminations on replication
Similar to Ruminations on replication (20)
Recently uploaded
Recently uploaded (20)
Ruminations on replication
- 1. REPRODUCIBILITY: FOUNDATION OF SCIENCE - OR - INVITATION TO TEST TRIVIAL HYPOTHESES? http://dirnaglfens
- 5. Reproducibility is one of the cornerstones of science. Robert Boyle (1627-1691) New Experiments Physico-Mechanical, Touching The Spring of the Air, and its Effects; Made, for the most part, in a New Pneumatical Engine (1660)
- 6. ‚ .. non-reproducible single occurrences are of no significance to science …‘ The Logic of Scientific Discovery (1934) Sir Karl Popper (1902-1994) ‘We do not take even our own observations quite seriously, or accept them as scientific observations, until we have repeated and tested them. Only by such repetitions can we convince ourselves that we are not dealing with a mere isolated ‘coincidence’, but with events which, on account of their regularity and reproducibility, are in principle inter- subjectively testable.’
- 7. Reproduction is in the 'DNA' of the scientific process 'Differentielle Reproduktion von Experimentalsystemen' ‘An experimental system owes its temporal coherence to its reproduction, and its development depends on whether one manages to produce differences without destroying its reproductive coherence.’ Hans-Jörg Rheinberger (*1946) Experiment-Differenz-Schrift. Zur Geschichte epistemischer Dinge), Marburg an der Lahn: Basilisken-Presse, 1992.
- 8. The lexicon of reproducibility Methods reproducibility: Same data, same tools, same results? Adds no additional evidence! Results reproducibility (aka „replication“): Technically competent repetition, i.e. a new study. Could be strict: identical conditions: or conceptual: altered conditions (does causal claim extend to previously unsampled settings?) Inferential reproducibility: Same conclusions from study replication or re-analysis? Not all scientists come to the same conclusions from same results, or may make different analytic choices. What is concluded or recommended from a study is often the only thing that matters! Adapted from Goodman et al. Sci Transl Med. 2016;8:341ps12.
- 9. What do we mean by 'reproducible'? Significance and P values: Evaluating replication effect against null hypothesis of no effect Evaluating replication effect against original effect size: Is the original effect size within the 95% CI of the effect size estimate from the replication. Alternatively: Comparing original and replication effect sizes Meta-analysis combining original and replication effects: Combining original and replication effect sizes for cumulative evidence Subjective assessment of “Did it replicate?” From the Open Science Collaboration, Psychology Replication, Science. 2015 ;349(6251):aac4716
- 10. A false dichotomy Replication Non-Replication
- 11. The emptiness of failed replication Mitchell J (2014) On the evidentiary evidence of failed replication http://jasonmitchell.fas.harvard.edu/Papers/Mitchell_failed_science_2014.pdf
- 12. The emptiness of failed replication Does a failure to replicate mean that the original result was a false positive? Or was the failed replication a false negative? Does successful replication mean that the original result was correct? Or are both results false positives?
- 13. Hidden moderators - Contextual sensitivity – Tacit knowledge ‚We analyzed 100 replication attempts in psychology and found that the extent to which the research topic was likely to be contextually sensitive (varying in time, culture, or location) was associated with replication success. This relationship remained a significant predictor of replication success even after adjusting for characteristics of the original and replication studies that previously had been associated with replication success (e.g., effect size, statistical power).‘ Proc Natl Acad Sci. 2016;113:6454-9.
- 14. "Standardization fallacy": Low external validity, poor reproducibility Nat Methods. 2009;6:257-61.Trends Pharmacol Sci. 2016;37:509-10
- 15. The stigma of nonreplication - The stigma of the replicator Dirnagl U (2018) Can (Non)-Replication be a Sin? https://dirnagl.com/2018/05/16/can-non-replication-be-a-sin/
- 16. p = 0.049 (p< α = 0.05) Assume that the experimental result is correct, i.e. measured difference equals (unknown) treatment effect. Repeat experiment under identical conditions (i.e. 'strict replication'). What is the probability to reproduce the significant findings? 50 %! How likely is strict replication ?
- 17. Nonreproducibility as an indicator of cutting edge research? Dirnagl (2017) How likely are your hypotheses, really? https://dirnagl.com/2017/04/13/how-original-are-your-scientific-hypotheses-really/
- 18. Confirmation – weeding out the false positives of exploration Jonathan Kimmelman PLoS Biol. (2014) 12:e1001863.
- 19. Reward the replicators and the replicated – fund replications
- 20. Resolving the tension: Discovery & Replication Suggested reading: Wagenmakers EJ, Dutilh G, Sarafoglou A. Perspect Psychol Sci. 2018 Jul;13(4):418-427 Chang and Eng Bunker circa 1865. Foto Hulton/Getty
- 21. No scientific progress without nonreproducibility To boldly go where no man… Exploration at low base rate Innovation ‚Paradigm shift‘ Incompetence Bad designs Tacit knowledge (bad reporting) Low validity (bias) Misconduct The Good The Bad Essential non-reproducibility (Kuhn) Detrimental nonreproducibility (Popper)
Editor's Notes
- The first to stress the importance of reproducibility in science was the Irish chemist Robert Boyle, in England in the 17th century. Boyle's air pump was designed to generate and study vacuum, which at the time was a very controversial concept. Indeed, distinguished philosophers such as René Descartes and Thomas Hobbes denied the very possibility of vacuum existence. Historians of science e.g. Steven Shapin and Simon Schaffer, in their 1985 book Leviathan and the Air-Pump, describe the debate between Boyle and Hobbes, ostensibly over the nature of vacuum, as fundamentally an argument about how useful knowledge should be gained. Boyle, a pioneer of the experimental method, maintained that the foundations of knowledge should be constituted by experimentally produced facts, which can be made believable to a scientific community by their reproducibility. By repeating the same experiment over and over again, Boyle argued, the certainty of fact will emerge. https://scientistseessquirrel.wordpress.com/2015/02/27/reproducibility-your-methods-section-and-400-years-of-angst/ The famous physicist Robert Boyle grappled with this question in the middle of the 1600s, and his answer had three elements (Shapin 1984). First, Boyle gave exhaustive detail of equipment, material, and procedures, so that readers could (at least in principle) reproduce his experiments. Second, he argued for “communal witnessing”: if results were to have authority, experiments should be witnessed – so Boyle conducted many of his key experiments in public, and published the names and qualifications of witnessing scientists along with his results. Third, Boyle described in exhaustive detail not just his methods, but his experiments’ circumstances and settings, his false starts and failures, and much else. For example, to accompany his reports of experiments using his famous vacuum pump, he provided an illustration (above) of the pump. Not, importantly, of a vacuum pump, but of the vacuum pump he used, complete with irregularities, dents, and dings. The point of all this description was to make readers feel as if they had been there – to recruit readers as “virtual witnesses”
- Kant was perhaps the first to realize that the objectivity of scientific statements is closely connected with the construction of theories — with the use of hypotheses and universal statements. Only when certain events recur in accordance with rules or regularities, as is the case with repeatable experiments, can our observations be tested — in principle — by anyone. We do not take even our own observations quite seriously, or accept them as scientific observations, until we have repeated and tested them. Only by such repetitions can we convince ourselves that we are not dealing with a mere isolated ‘coincidence’, but with events which, on account of their regularity and reproducibility, are in principle inter-subjectively testable. Every experimental physicist knows those surprising and inexplicable apparent ‘effects’ which in his laboratory can perhaps even be reproduced for some time, but which finally disappear without trace. Of course, no physicist would say in such a case that he had made a scientific discovery (though he might try to rearrange his experiments so as to make the effect reproducible). Indeed the scientifically significant physical effect may be defined as that which can be regularly reproduced by anyone who carries out the appropriate experiment in the way prescribed. No serious physicist would offer for publication, as a scientific discovery, any such ‘occult effect,’ as I propose to call it — one for whose reproduction he could give no instructions. The ‘discovery’ would be only too soon rejected as chimerical, simply because attempts to test it would lead to negative results. (It follows that any controversy over the question whether events which are in principle unrepeatable and unique ever do occur cannot be decided by science: it would be a metaphysical controversy.) – Karl Popper (1959/2002), The Logic of Scientific Discovery, pp. 23-24.
- An experimental system owes its temporal coherence to its reproduction, and its development depends on whether one manages to produce differences without destroying its reproductive coherence. Together, these two factors make up its differential reproduction. The construction process is dominated by a kind of probing movement which with regard to the scientific object can be described as a “jeu des possibles”24 or a “‘game’ of difference.”25 I would like to suggest that it is precisely the way in which it is “falling prey to its own work” that makes the scientific enterprise similar in a certain sense to what Derrida called “the enterprise of deconstruction.”26 To play this game productively requires “Erfahrenheit”27 on the part of the experimenter, something that can perhaps best be paraphrased using the paradoxical expression ‘acquired intuition’.28 We can conclude from what has just been said that one never knows exactly where an experimental system will lead. As soon as one knows exactly what it produces it is no longer a research system. An experimental system in which a scientific object gradually takes on contours in the sense that certain signals can be handled in a reproducible way, has to simultaneously open windows in which new signals are visible. Once it is stabilised in one respect, it can and must be destabilised in another in order to arrive at new ‘results’.29 Stabilisation and destabilisation are interdependent. In order to remain productive, an experimental set-up has to be sufficiently open to produce unforeseeable signals and to let new technologies, instruments, and model substances seep in.
- • Recent hand-wringing over failed replications in social psychology is largely pointless, because unsuccessful experiments have no meaningful scientific value. • Because experiments can be undermined by a vast number of practical mistakes, the likeliest explanation for any failed replication will always be that the replicator bungled something along the way. Unless direct replications are conducted by flawless experimenters, nothing interesting can be learned from them. • Three standard rejoinders to this critique are considered and rejected. Despite claims to the contrary, failed replications do not provide meaningful information if they closely follow original methodology; they do not necessarily identify effects that may be too small or flimsy to be worth studying; and they cannot contribute to a cumulative understanding of scientific phenomena. • Replication efforts appear to reflect strong prior expectations that published findings are not reliable, and as such, do not constitute scientific output. • The field of social psychology can be improved, but not by the publication of negative findings. Experimenters should be encouraged to restrict their “degrees of freedom,” for example, by specifying designs in advance. • Whether they mean to or not, authors and editors of failed replications are publicly impugning the scientific integrity of their colleagues. Targets of failed replications are justifiably upset, particularly given the inadequate basis for replicators’ extraordinary claims.
- to be in the pillory - am pranger stehen
- power irrelevant, as experiment reproduced under identical conditions
- The Amazing American Story of the Original Siamese Twins Few newcomers to the U.S. have crossed more daunting barriers than Chang and Eng Bunker