1. SE Myths on Replication
Natalia Juristo
Universidad Politecnica de Madrid
Spain
www.grise.upm.es/miembros/natalia/

2. Myths

In SE we have a mythical view about
replication in other sciences

Specially regarding replication in the hard
sciences, Physics as paradigm

3. Replication is Not a Myth



Replication does exist in other disciplines
To see replication and its absence in
practice Nature Magazine tracked the fate
of 19 papers in issue 6893 to check
whether the results had been reproduced
2 years later
In a large majority of cases they had.
Papers on fields like Neurology,
Paleontology, etc.

4. Myth 1
Others are able to give ALL the information
to replicators

5. What Is All the information?

Myht


Information that other researchers (physicist
as paradigm) interchange for replication is
complete and perfect
Reality

No, it is not. They face, as we do, two
problems
Tacit knowledge
 Unknown relevant variables


6. Inspiring quotes

“There is always something unspoken even in the middle
of the exact sciences”
“The Logic of Tacit Inference”
Physicist-Chemist Michael Polanyi

“It’s very difficult to make a carbon copy. You can make
a near one, but if it turns out that what is critical is the
way he glued his transducers, and he forgets to tell you
that the technicians always puts a copy of Physical
Review on top of them for weight, well, it could make all
the difference”
A physicist quoted in
“Changing Order”
Harry Collins

7. Inspiring quotes

“If I am trying to explain how to build the TEA laser I will not tell you that the
inductance of the top is important, because I assume you already know...
…Finally, you do not know what you need to know and I do not know what I
know...
…In fact, the principle of TEA lasers, scientists did not know necessarily
that the inductance of the top was important”
A physicist quoted in
“Tacit knowledge: you don't know how much you know”
New Scientist Magazine N. 2762
Harry Collins
2010

8. Inspiring quotes

“It is often hard to tell whether an inability to
replicate a result is due to a group’s failing or a
flaw in the original paper.
The reason is often the countless tiny details of
experimental method that are omitted from the
paper but can influence results”
Gillian Murphy
Cell Biologist
University of Cambridge
Quoted in “The Trouble with Replication”
Nature 2006

9. Myth 2
Independent replications means
0 interaction

10. Independence = No Interaction

Myth



Other researchers (physicist as paradigm) just
interchange materials and never interact that is why it
is an independent replication
Independent replication means do not interact with
the previous experimenters at all except
interchanging the protocol
Reality

They interact, talk, discuss why my replication does
not get the same results than yours, they visit each
other to see how you run your replication,…

11. They Do Interact

“A paper can never be a foolproof recipe for a
replication of its results because this sort of
information can never be entirely captured in a
scientific paper… Researchers compensate by
exchanging tips by e-mail and at conferences.
Because this social interactions are not
recorded anywhere, it is hard to consult or build
on them”
“The Trouble with Replication”
Jim Giles
Nature 2006

12. Inspiring quotes

“When I started in Science I was told that I
should be able to repeat an experiment by
reading the paper, but that is almost never the
case”
Michael Ronemus
Molecular and Cell Biologist
Cold Spring Harbor Laboratory
Editor of the Journal Cold Spring Harbor Protocols

Cold Spring Harbor Protocols, PLoS ONE and Nature
Protocols include discussion facilities that allow
researchers to talk to the original authors and others
users of the protocol. These could prove ideal places to
resolve problems that crop up during attempts to
reproduce a previous result

13. New ways of interchanging the
critical Info

PLoS ONE takes the comment model further
than anyone else with various functions for
promoting discussion

“The aim is to recreate the kind of discussion
that takes place in front of conference posters”
Surridge
PLoS ONE Manager

14. They Even Visit Each Other
“The story will be a familiar one to many biologist. After
publication, rival labs fell over themselves to reproduce
the results. Many contacted Verfaillie with requests for
the cells and the reagents used to create them, or to ask
for more details of the experimental protocol involved.
Several high-profile groups sent researchers to
Minnesota to learn how to extracted and culture the
cells…the procedure takes up to six weeks to master…”
“The Trouble with Replication”
Jim Giles
Nature 2006

15. …Cold fusion (NYT May 1989)

Nathan Lewis (Caltech) “Pons would never
answer any of our questions. So we asked Los
Alamos National Lab to put our questions to him
instead, since they were in touch with him”

Edward Redish (Univ of Maryland) “We have
invited Fleischmann to participate in the
Baltimore sessions“

“Many speakers at the meeting reported failure
in their effort to elicit information or comments
from Pons”

16. Myth 3
Other sciences get equal results

17. Hard Sciences Get Same Results

Myth



If results are truth (not flaw) they get the same
results
Variability in Physics experiments results is
very low
Reality

Variation of experimental results is also very
common in Physics

18. “Those of us […] know intuitively that there is something ‘softer’ and less cumulative
about our research than about those of physical sciences. […] distinguished
researchers have cited the pervasive presence of interactions or historical influences
as reasons not to expect a cumulative […] science. Still others have cited the low
quality of data […] as a barrier.”
“What is surprising is that the research results in the physical sciences are not
markedly more consistent […] data do suggest that results from replicated
experiments do not always tend to be consistent”
“The data […] are striking. When all studies are included in the quantitative reviews,
the average Birge ratio is over 2.00, which is 100% larger than expected when
studies yield consistent results. Moreover, 6 of the 13 reviews (46.2%) show
statistically significant disagreement among studies.”
L. Hedges
How hard is hard science, how soft is soft science? The empirical cumulativeness of
research
American Psychologist 1987

19. Myth 4
They know the relevant variables to control
&
Know how to measures them

20. 
Myth


Mature experimental sciences are able to
perfectly describe experiment variables, know
how to perfectly measure response variables,
etc.
Reality

Even in Physics they can discuss about
interpretation of results or adequate ways of
measuring response variables

21. The Cold Fusion Case


After several trials of reproducing Pons&Fleischmann experiment and fail the
assertion was that cold fusion observations were based on experimental errors
What type of errors can be done in a Physics experiment?

Using more sensitive equipment. Failing to install a stirring device in the test cell,
temperature differences in the cell led to false estimates of its overall heat. This may have
suggested that its cell was producing fusion energy


Regarding helium traces in P&F experiment: helium is a trace component of air, and the
amount of helium in the cell corresponded to what normally enters from the atmosphere.



Results can be interpreted differently
Regarding the burst of heat that was observed: Someone turned the current off for a while.
When that happens hydrogen naturally bubbles out of the palladium cathode, and creates a
hazard of fire or explosion. It is a simple chemical reaction that has nothing to do with fusion.


So measuring response variables are sometimes not direct or well-defined
Again results interpretation is not clear
The experiment was flawed because of the system used to measure heat. There were error
on temperature measurement. Meyerhof of Stanford University said: “Some scientist put a
thermometer at one place and not another”
About Jones replication of P&F

had used relatively crude neutron-detecting equipment, and had measured only a very small excess of
neutrons over what could be expected from natural sources without any fusion

22. Myth 5
Learning vs. verifying

23. Maturity & Changes



Depending on the maturity of the experiment, the level of
similarity required can vary
When an experimental study is young there are many
aspects that are unknown. Any change in replication
might cause a variation in the results of the experiment
impossible to be interpreted
In immature experimental studies begin replication as
similar as possible can know the conditions to be
controlled

24. Problems with Identical
Replications

SE has tried to make identical repetition of
experiments, but no exact replications have yet
been achieved

The complexity of the software development
setting prevents the many experimental
conditions from being reproduced identically

25. Learning vs. Checking


A replication designed to verify results should be exact,
whereas a replication for the purposes of knowledge
discovery and learning should be different
Both are necessary. At the beginning of experimental
research, equality is not even an option



The critical variables to be controlled are now known!
Later on, both knowledge discovery and testing can be
more systematic
In the early stages, failure to get the expected results
should not be construed as falsification, but as a step
towards the discovery of some new factor

26. Evidence Structure

Result


Constructs




A is more effective than B
A applied as follows: xxx
B applied…
Effectiveness measured …
Population



For inexperienced subjects
For small programs
For these types of faults: omission, …

27. SE Myths on Replication
Natalia Juristo
Universidad Politecnica de Madrid
Spain
www.grise.upm.es/miembros/natalia/