Your SlideShare is downloading. ×
0
Myths on Replication (LASER School Talk 2010)
Myths on Replication (LASER School Talk 2010)
Myths on Replication (LASER School Talk 2010)
Myths on Replication (LASER School Talk 2010)
Myths on Replication (LASER School Talk 2010)
Myths on Replication (LASER School Talk 2010)
Myths on Replication (LASER School Talk 2010)
Myths on Replication (LASER School Talk 2010)
Myths on Replication (LASER School Talk 2010)
Myths on Replication (LASER School Talk 2010)
Myths on Replication (LASER School Talk 2010)
Myths on Replication (LASER School Talk 2010)
Myths on Replication (LASER School Talk 2010)
Myths on Replication (LASER School Talk 2010)
Myths on Replication (LASER School Talk 2010)
Myths on Replication (LASER School Talk 2010)
Myths on Replication (LASER School Talk 2010)
Myths on Replication (LASER School Talk 2010)
Myths on Replication (LASER School Talk 2010)
Myths on Replication (LASER School Talk 2010)
Myths on Replication (LASER School Talk 2010)
Myths on Replication (LASER School Talk 2010)
Myths on Replication (LASER School Talk 2010)
Myths on Replication (LASER School Talk 2010)
Myths on Replication (LASER School Talk 2010)
Myths on Replication (LASER School Talk 2010)
Myths on Replication (LASER School Talk 2010)
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Myths on Replication (LASER School Talk 2010)

74

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
74
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. SE Myths on Replication Natalia Juristo Universidad Politecnica de Madrid Spain www.grise.upm.es/miembros/natalia/
  • 2. Myths  In SE we have a mythical view about replication in other sciences  Specially regarding replication in the hard sciences, Physics as paradigm
  • 3. Replication is Not a Myth    Replication does exist in other disciplines To see replication and its absence in practice Nature Magazine tracked the fate of 19 papers in issue 6893 to check whether the results had been reproduced 2 years later In a large majority of cases they had. Papers on fields like Neurology, Paleontology, etc.
  • 4. Myth 1 Others are able to give ALL the information to replicators
  • 5. What Is All the information?  Myht   Information that other researchers (physicist as paradigm) interchange for replication is complete and perfect Reality  No, it is not. They face, as we do, two problems Tacit knowledge  Unknown relevant variables 
  • 6. Inspiring quotes  “There is always something unspoken even in the middle of the exact sciences” “The Logic of Tacit Inference” Physicist-Chemist Michael Polanyi  “It’s very difficult to make a carbon copy. You can make a near one, but if it turns out that what is critical is the way he glued his transducers, and he forgets to tell you that the technicians always puts a copy of Physical Review on top of them for weight, well, it could make all the difference” A physicist quoted in “Changing Order” Harry Collins
  • 7. Inspiring quotes  “If I am trying to explain how to build the TEA laser I will not tell you that the inductance of the top is important, because I assume you already know... …Finally, you do not know what you need to know and I do not know what I know... …In fact, the principle of TEA lasers, scientists did not know necessarily that the inductance of the top was important” A physicist quoted in “Tacit knowledge: you don't know how much you know” New Scientist Magazine N. 2762 Harry Collins 2010
  • 8. Inspiring quotes  “It is often hard to tell whether an inability to replicate a result is due to a group’s failing or a flaw in the original paper. The reason is often the countless tiny details of experimental method that are omitted from the paper but can influence results” Gillian Murphy Cell Biologist University of Cambridge Quoted in “The Trouble with Replication” Nature 2006
  • 9. Myth 2 Independent replications means 0 interaction
  • 10. Independence = No Interaction  Myth    Other researchers (physicist as paradigm) just interchange materials and never interact that is why it is an independent replication Independent replication means do not interact with the previous experimenters at all except interchanging the protocol Reality  They interact, talk, discuss why my replication does not get the same results than yours, they visit each other to see how you run your replication,…
  • 11. They Do Interact  “A paper can never be a foolproof recipe for a replication of its results because this sort of information can never be entirely captured in a scientific paper… Researchers compensate by exchanging tips by e-mail and at conferences. Because this social interactions are not recorded anywhere, it is hard to consult or build on them” “The Trouble with Replication” Jim Giles Nature 2006
  • 12. Inspiring quotes  “When I started in Science I was told that I should be able to repeat an experiment by reading the paper, but that is almost never the case” Michael Ronemus Molecular and Cell Biologist Cold Spring Harbor Laboratory Editor of the Journal Cold Spring Harbor Protocols  Cold Spring Harbor Protocols, PLoS ONE and Nature Protocols include discussion facilities that allow researchers to talk to the original authors and others users of the protocol. These could prove ideal places to resolve problems that crop up during attempts to reproduce a previous result
  • 13. New ways of interchanging the critical Info  PLoS ONE takes the comment model further than anyone else with various functions for promoting discussion  “The aim is to recreate the kind of discussion that takes place in front of conference posters” Surridge PLoS ONE Manager
  • 14. They Even Visit Each Other “The story will be a familiar one to many biologist. After publication, rival labs fell over themselves to reproduce the results. Many contacted Verfaillie with requests for the cells and the reagents used to create them, or to ask for more details of the experimental protocol involved. Several high-profile groups sent researchers to Minnesota to learn how to extracted and culture the cells…the procedure takes up to six weeks to master…” “The Trouble with Replication” Jim Giles Nature 2006
  • 15. …Cold fusion (NYT May 1989)  Nathan Lewis (Caltech) “Pons would never answer any of our questions. So we asked Los Alamos National Lab to put our questions to him instead, since they were in touch with him”  Edward Redish (Univ of Maryland) “We have invited Fleischmann to participate in the Baltimore sessions“  “Many speakers at the meeting reported failure in their effort to elicit information or comments from Pons”
  • 16. Myth 3 Other sciences get equal results
  • 17. Hard Sciences Get Same Results  Myth    If results are truth (not flaw) they get the same results Variability in Physics experiments results is very low Reality  Variation of experimental results is also very common in Physics
  • 18. “Those of us […] know intuitively that there is something ‘softer’ and less cumulative about our research than about those of physical sciences. […] distinguished researchers have cited the pervasive presence of interactions or historical influences as reasons not to expect a cumulative […] science. Still others have cited the low quality of data […] as a barrier.” “What is surprising is that the research results in the physical sciences are not markedly more consistent […] data do suggest that results from replicated experiments do not always tend to be consistent” “The data […] are striking. When all studies are included in the quantitative reviews, the average Birge ratio is over 2.00, which is 100% larger than expected when studies yield consistent results. Moreover, 6 of the 13 reviews (46.2%) show statistically significant disagreement among studies.” L. Hedges How hard is hard science, how soft is soft science? The empirical cumulativeness of research American Psychologist 1987
  • 19. Myth 4 They know the relevant variables to control & Know how to measures them
  • 20.  Myth   Mature experimental sciences are able to perfectly describe experiment variables, know how to perfectly measure response variables, etc. Reality  Even in Physics they can discuss about interpretation of results or adequate ways of measuring response variables
  • 21. The Cold Fusion Case   After several trials of reproducing Pons&Fleischmann experiment and fail the assertion was that cold fusion observations were based on experimental errors What type of errors can be done in a Physics experiment?  Using more sensitive equipment. Failing to install a stirring device in the test cell, temperature differences in the cell led to false estimates of its overall heat. This may have suggested that its cell was producing fusion energy   Regarding helium traces in P&F experiment: helium is a trace component of air, and the amount of helium in the cell corresponded to what normally enters from the atmosphere.    Results can be interpreted differently Regarding the burst of heat that was observed: Someone turned the current off for a while. When that happens hydrogen naturally bubbles out of the palladium cathode, and creates a hazard of fire or explosion. It is a simple chemical reaction that has nothing to do with fusion.   So measuring response variables are sometimes not direct or well-defined Again results interpretation is not clear The experiment was flawed because of the system used to measure heat. There were error on temperature measurement. Meyerhof of Stanford University said: “Some scientist put a thermometer at one place and not another” About Jones replication of P&F  had used relatively crude neutron-detecting equipment, and had measured only a very small excess of neutrons over what could be expected from natural sources without any fusion
  • 22. Myth 5 Learning vs. verifying
  • 23. Maturity & Changes    Depending on the maturity of the experiment, the level of similarity required can vary When an experimental study is young there are many aspects that are unknown. Any change in replication might cause a variation in the results of the experiment impossible to be interpreted In immature experimental studies begin replication as similar as possible can know the conditions to be controlled
  • 24. Problems with Identical Replications  SE has tried to make identical repetition of experiments, but no exact replications have yet been achieved  The complexity of the software development setting prevents the many experimental conditions from being reproduced identically
  • 25. Learning vs. Checking   A replication designed to verify results should be exact, whereas a replication for the purposes of knowledge discovery and learning should be different Both are necessary. At the beginning of experimental research, equality is not even an option    The critical variables to be controlled are now known! Later on, both knowledge discovery and testing can be more systematic In the early stages, failure to get the expected results should not be construed as falsification, but as a step towards the discovery of some new factor
  • 26. Evidence Structure  Result   Constructs     A is more effective than B A applied as follows: xxx B applied… Effectiveness measured … Population    For inexperienced subjects For small programs For these types of faults: omission, …
  • 27. SE Myths on Replication Natalia Juristo Universidad Politecnica de Madrid Spain www.grise.upm.es/miembros/natalia/

×