This document discusses the importance of reproducibility and repeatability in scientific research. It defines reproducibility as getting similar results under changed conditions, while repeatability refers to getting similar results under the same conditions using the same methods. The document outlines factors that affect reproducibility and repeatability and gives examples of challenges with reproducibility in fields like medicine, biology and computer science. It stresses that reproducibility is a core principle of the scientific method and is important for validating results and enabling other researchers to build upon methods and findings.

1. Presented
by
Pramod Kumar
(Environmental Monitoring Lab)

2. Sciences and affecting factors.
New Idea, R and R.
Difference of two test results under ideal conditions should not more than 5%.
Introduction

3. History
 Aristotle: Father of Reproducibility.
 First importance of reproducibility in science by Irish
chemist Robert Boyle.
 Christiaan Huygens built his own air pump in Amsterdam,
which was a replicate of Boyle’s air pump.
Steven Shapin and Simon Schaffer (1985), book "Leviathan
and the Air-pumps” describe debate between Boyle and
Philosophers Hobbes.
Boyle’s air pump ,17th Century
www.wikipedia.org/reproducibility

4. “Closeness of agreement between results of successive
measurements of same measurand carried out under same
conditions of measurement”.
Criteria for repeatability
Same measurement procedure.
Same observer.
Short time.
Same location.
Same measuring instrument used under same
condition.
Measurement
average
Repeatability
Repeatability
http://www.raytheon.com

5. Problems associated with repeatability
Unaffordable cost of experimental procedure
Genetic differences in experimental model
Variation in experimental condition
More variable involve, more error
Restriction of using same instrument
 Biological process provide additional source of variability

6. Repeatability assessment result
Repeatability experiment of SIGMOD, 2008

7. “Scientific process relies on Reproducibility of analysis”
Aristotle,322 BC
“Reproducibility, getting bitwise identical results from run to run”
Demmel,2013
Closeness of agreement between results of measurements of same measurand
carried out under changed conditions of measurement.
Reproducibility

8.  Method of measurement
 Principle of measurement
 Observer
 Measuring instrument
 Reference standard
 Location
 Conditions of use
 Time
Criteria for Reproducibility
Appraiser 1 Measurement
Appraiser 2
Appraiser n
Reproducibility
Distribution of Appraiser Averages
http://www.raytheon.com

9. Reproducible research ?
Two basic reasons :
 First is to show evidence of correctness of your results.
 Second reason to aspire to reproducibility is to enable others to make use of our
methods and results.
Improve gage capability ?
 Repeatability: Gage maintain and Gage redesign,
 Reproducibility: Operator training ,More clearly define measurement scale
available to operator.
Stodden et al.2013

10. Repeatability V/s reproducibility
PublicationPublication
Provenance
Attribution Credit
Provenance
Attribution Credit
Context
Investigation
Study
Experiment
Context
Investigation
Study
Experiment
http://www.slideshare.net/sjcockell/reproducibility

11. http://forum.thefreedictionary.com
Reproducibility is one of main principle of the Scientific method.
Significance of repeatability and reproducibility in science
the Scientific method

12. Computer Science
Reproducible computational research is to expose more of search work
flow to our audience.
It was considered a challenge in translation from theoretical CS to
implementation of actual algorithms.
Toxicological Science
To check toxicity of a chemical waste by biological assays of
bioluminescent bacteria.
Impact assessment of a toxicant through models.

13. Medical Science
Reproducibility research is key to discoveries of new medicine .
A phase first discovery will be following by phase second reproductions
as a drug develops to commercial production.
In recent decades phase second has fallen from 28% to 18%.
A study in 2012 , found that 65% medical studies were inconsistent when
re-test ,and only 6% were completely reproducible.

14. Biological Science
 Synthesis of research results is of fundamental importance in seeking to
develop ecological and construct theories
 It is critical that published studies contain sufficient detail to allow their
methods to be replicated and their results compared.
 In response to this need, growing attention is being paid to publication and
presentation of analytical results in Ecology.

15. Digital Science :Astronomy
 In Astronomy, from observing proposals to final publications, data and
software.
 Current challenges in Astronomy is e cient exploitation of huge volumeffi
of data.
 Which is generated by experiments/ observations, computed by means of
numerical simulations.

16. Bayer:* Bayer halted in 2011 nearly two third of its early drug target project
because in house experiments failed to match claims made in literature.
Pfizer:** Pfizer wrote off $ 750 million in 2012 after results of a study with
dimebon for Alzheimer, published originally in journal lancet, could not
reproduced.
Solution:
Establish a common quality standard for biomedical research in drug
development.
WHO and ASQ published quality guidelines for biomedical research.
Pharmaceutical industry should promote these guidelines to improve R and D
productivity.
Source: *WSJ, December 2, 2011; **Fierce Biotech, January 17,2012
Impact on Industrial productivity

18. Repeatability of capillary absorption tests

19.  R and R provide validity of your experiment.
 Enhances quality in work.
 Recent years, increase in attempts to replicate the results of studies in
fields from Chemistry to Medicine .
 Discussion about repeatability and benefits to be gained from repeating
studies, with out necessarily expecting to replicate original results.
 Its have limitation in field-oriented disciplines e.g. Ecology, Archeology,
Psychology,and Social Science.

20. Clear experimental goal.
Use proper experimental methodology.
Standard statistical analysis
Enhance collaboration and open communication
Changes to regulations of funding bodies.
 Random audits .
 Give researchers a greater incentive.
Reduce unrepresentative workloads.
Measuring right object.
Future perspective

21. In 2012, a study by Begley and Ellis was published in Nature that reviewed a
decade of research. That study found that 47 out of 53 medical research paper
focused on cancer research were irreproducible.
Case study