5. Reproducibility is one of the cornerstones
of science.
Robert Boyle (1627-1691)
New Experiments Physico-Mechanical,
Touching The Spring of the Air, and its
Effects; Made, for the most part, in a New
Pneumatical Engine (1660)
6. ‚ .. non-reproducible single occurrences
are of no significance to science …‘
The Logic of Scientific Discovery (1934)
Sir Karl Popper
(1902-1994)
‘We do not take even our own
observations quite seriously, or accept
them as scientific observations, until we
have repeated and tested them. Only by
such repetitions can we convince ourselves
that we are not dealing with a mere
isolated ‘coincidence’, but with events
which, on account of their regularity and
reproducibility, are in principle inter-
subjectively testable.’
7. Reproduction is in the 'DNA' of the
scientific process
'Differentielle Reproduktion von
Experimentalsystemen'
‘An experimental system owes its
temporal coherence to its
reproduction, and its development
depends on whether one manages to
produce differences without
destroying its reproductive coherence.’
Hans-Jörg Rheinberger
(*1946)
Experiment-Differenz-Schrift. Zur
Geschichte epistemischer Dinge), Marburg
an der Lahn: Basilisken-Presse, 1992.
8. The lexicon of reproducibility
Methods reproducibility: Same data, same tools, same
results? Adds no additional evidence!
Results reproducibility (aka „replication“): Technically
competent repetition, i.e. a new study. Could be strict:
identical conditions: or conceptual: altered conditions (does
causal claim extend to previously unsampled settings?)
Inferential reproducibility: Same conclusions from study
replication or re-analysis? Not all scientists come to the
same conclusions from same results, or may make different
analytic choices. What is concluded or recommended from
a study is often the only thing that matters!
Adapted from Goodman et al. Sci Transl Med. 2016;8:341ps12.
9. What do we mean by 'reproducible'?
Significance and P values: Evaluating replication effect against null
hypothesis of no effect
Evaluating replication effect against original effect size: Is the
original effect size within the 95% CI of the effect size estimate
from the replication. Alternatively: Comparing original and
replication effect sizes
Meta-analysis combining original and replication effects:
Combining original and replication effect sizes for cumulative
evidence
Subjective assessment of “Did it replicate?”
From the Open Science Collaboration, Psychology Replication, Science. 2015 ;349(6251):aac4716
11. The emptiness of failed replication
Mitchell J (2014) On the evidentiary evidence of failed replication
http://jasonmitchell.fas.harvard.edu/Papers/Mitchell_failed_science_2014.pdf
12. The emptiness of failed replication
Does a failure to replicate mean that the original
result was a false positive? Or was the failed
replication a false negative?
Does successful replication mean that the original
result was correct? Or are both results false positives?
13. Hidden moderators - Contextual
sensitivity – Tacit knowledge
‚We analyzed 100 replication attempts in psychology and found that the
extent to which the research topic was likely to be contextually sensitive
(varying in time, culture, or location) was associated with replication
success. This relationship remained a significant predictor of replication
success even after adjusting for characteristics of the original and
replication studies that previously had been associated with replication
success (e.g., effect size, statistical power).‘
Proc Natl Acad Sci. 2016;113:6454-9.
15. The stigma of nonreplication -
The stigma of the replicator
Dirnagl U (2018) Can (Non)-Replication be a Sin?
https://dirnagl.com/2018/05/16/can-non-replication-be-a-sin/
16. p = 0.049 (p< α = 0.05)
Assume that the experimental result is correct, i.e.
measured difference equals (unknown) treatment effect.
Repeat experiment under identical conditions (i.e. 'strict
replication').
What is the probability to reproduce the significant
findings?
50 %!
How likely is strict replication ?
17. Nonreproducibility as an indicator of
cutting edge research?
Dirnagl (2017) How likely are your hypotheses, really?
https://dirnagl.com/2017/04/13/how-original-are-your-scientific-hypotheses-really/
18. Confirmation – weeding out the false
positives of exploration
Jonathan
Kimmelman
PLoS Biol. (2014) 12:e1001863.
20. Resolving the tension:
Discovery & Replication
Suggested reading:
Wagenmakers EJ, Dutilh G, Sarafoglou A.
Perspect Psychol Sci. 2018 Jul;13(4):418-427
Chang and Eng Bunker circa 1865. Foto Hulton/Getty
21. No scientific progress without
nonreproducibility
To boldly go where no man…
Exploration at low base rate
Innovation
‚Paradigm shift‘
Incompetence
Bad designs
Tacit knowledge (bad reporting)
Low validity (bias)
Misconduct
The Good The Bad
Essential non-reproducibility
(Kuhn)
Detrimental nonreproducibility
(Popper)
The first to stress the importance of reproducibility in science was the Irish chemist Robert Boyle, in England in the 17th century. Boyle's air pump was designed to generate and study vacuum, which at the time was a very controversial concept. Indeed, distinguished philosophers such as René Descartes and Thomas Hobbes denied the very possibility of vacuum existence. Historians of science e.g. Steven Shapin and Simon Schaffer, in their 1985 book Leviathan and the Air-Pump, describe the debate between Boyle and Hobbes, ostensibly over the nature of vacuum, as fundamentally an argument about how useful knowledge should be gained. Boyle, a pioneer of the experimental method, maintained that the foundations of knowledge should be constituted by experimentally produced facts, which can be made believable to a scientific community by their reproducibility. By repeating the same experiment over and over again, Boyle argued, the certainty of fact will emerge.
https://scientistseessquirrel.wordpress.com/2015/02/27/reproducibility-your-methods-section-and-400-years-of-angst/
The famous physicist Robert Boyle grappled with this question in the middle of the 1600s, and his answer had three elements (Shapin 1984). First, Boyle gave exhaustive detail of equipment, material, and procedures, so that readers could (at least in principle) reproduce his experiments. Second, he argued for “communal witnessing”: if results were to have authority, experiments should be witnessed – so Boyle conducted many of his key experiments in public, and published the names and qualifications of witnessing scientists along with his results. Third, Boyle described in exhaustive detail not just his methods, but his experiments’ circumstances and settings, his false starts and failures, and much else. For example, to accompany his reports of experiments using his famous vacuum pump, he provided an illustration (above) of the pump. Not, importantly, of a vacuum pump, but of the vacuum pump he used, complete with irregularities, dents, and dings. The point of all this description was to make readers feel as if they had been there – to recruit readers as “virtual witnesses”
Kant was perhaps the first to realize that the objectivity of scientific statements is closely connected with the construction of theories — with the use of hypotheses and universal statements. Only when certain events recur in accordance with rules or regularities, as is the case with repeatable experiments, can our observations be tested — in principle — by anyone. We do not take even our own observations quite seriously, or accept them as scientific observations, until we have repeated and tested them. Only by such repetitions can we convince ourselves that we are not dealing with a mere isolated ‘coincidence’, but with events which, on account of their regularity and reproducibility, are in principle inter-subjectively testable.
Every experimental physicist knows those surprising and inexplicable apparent ‘effects’ which in his laboratory can perhaps even be reproduced for some time, but which finally disappear without trace. Of course, no physicist would say in such a case that he had made a scientific discovery (though he might try to rearrange his experiments so as to make the effect reproducible). Indeed the scientifically significant physical effect may be defined as that which can be regularly reproduced by anyone who carries out the appropriate experiment in the way prescribed. No serious physicist would offer for publication, as a scientific discovery, any such ‘occult effect,’ as I propose to call it — one for whose reproduction he could give no instructions. The ‘discovery’ would be only too soon rejected as chimerical, simply because attempts to test it would lead to negative results. (It follows that any controversy over the question whether events which are in principle unrepeatable and unique ever do occur cannot be decided by science: it would be a metaphysical controversy.)
– Karl Popper (1959/2002), The Logic of Scientific Discovery, pp. 23-24.
An experimental system owes its temporal coherence to its reproduction, and its development depends on whether one manages to produce differences without destroying its reproductive coherence. Together, these two factors make up its differential reproduction.
The construction process is dominated by a kind of probing movement which with regard to the scientific object can be described as a “jeu des possibles”24 or a “‘game’ of difference.”25 I would like to suggest that it is precisely the way in which it is “falling prey to its own work” that makes the scientific enterprise similar in a certain sense to what Derrida called “the enterprise of deconstruction.”26 To play this game productively requires “Erfahrenheit”27 on the part of the experimenter, something that can perhaps best be paraphrased using the paradoxical expression ‘acquired intuition’.28 We can conclude from what has just been said that one never knows exactly where an experimental system will lead. As soon as one knows exactly what it produces it is no longer a research system. An experimental system in which a scientific object gradually takes on contours in the sense that certain signals can be handled in a reproducible way, has to simultaneously open windows in which new signals are visible. Once it is stabilised in one respect, it can and must be destabilised in another in order to arrive at new ‘results’.29 Stabilisation and destabilisation are interdependent. In order to remain productive, an experimental set-up has to be sufficiently open to produce unforeseeable signals and to let new technologies, instruments, and model substances seep in.
• Recent hand-wringing over failed replications in social psychology is largely
pointless, because unsuccessful experiments have no meaningful scientific
value.
• Because experiments can be undermined by a vast number of practical mistakes,
the likeliest explanation for any failed replication will always be that the replicator
bungled something along the way. Unless direct replications are conducted
by flawless experimenters, nothing interesting can be learned from them.
• Three standard rejoinders to this critique are considered and rejected. Despite
claims to the contrary, failed replications do not provide meaningful
information if they closely follow original methodology; they do not necessarily
identify effects that may be too small or flimsy to be worth studying; and they
cannot contribute to a cumulative understanding of scientific phenomena.
• Replication efforts appear to reflect strong prior expectations that published
findings are not reliable, and as such, do not constitute scientific output.
• The field of social psychology can be improved, but not by the publication of
negative findings. Experimenters should be encouraged to restrict their
“degrees of freedom,” for example, by specifying designs in advance.
• Whether they mean to or not, authors and editors of failed replications are
publicly impugning the scientific integrity of their colleagues. Targets of failed
replications are justifiably upset, particularly given the inadequate basis for
replicators’ extraordinary claims.
to be in the pillory - am pranger stehen
power irrelevant, as experiment reproduced under identical conditions
The Amazing American Story of the Original Siamese Twins
Few newcomers to the U.S. have crossed more daunting barriers than Chang and Eng Bunker