Reproducible Research Sergey Fomel The University of Texas at Austin

Outline Personal experience Why do reproducible research? How to do reproducible research? Reproducible computational experiments

Personal experience

Why do reproducible research?

How to do reproducible research?

Reproducible computational experiments

Personal Experience Jon Claerbout and RR at Stanford Madagascar open-source software CiSE special issue (Jan-Feb 2009) David Donoho et al Randall LeVeque Roger Ping and Sandrah Eckel Victoria Stodden

Jon Claerbout and RR at Stanford

Madagascar open-source software

CiSE special issue (Jan-Feb 2009)

David Donoho et al

Randall LeVeque

Roger Ping and Sandrah Eckel

Victoria Stodden

Jon Claerbout’s Story 1987: Sunview experience Interactive programs are slavery 1992: LaTeX + cake Rebuilding books by a single command

1987: Sunview experience

Interactive programs are slavery

1992: LaTeX + cake

Rebuilding books by a single command

Reproducible Research at SEP Stanford Exploration Project Founded in 1973 2 Ph.D. students per year Reproducible research From CD-ROMs to WWW From cake to GNU make 2001 CiSE paper The principal beneficiary is the author

Stanford Exploration Project

Founded in 1973

2 Ph.D. students per year

Reproducible research

From CD-ROMs to WWW

From cake to GNU make

2001 CiSE paper

The principal beneficiary is the author

The Madagascar Project Multidimensional data analysis Started in 2006 Open community Open source (GPL) Three levels Building blocks in C Recipes in Python/SCons Papers in LaTeX + SCons http:// ahay.org

Multidimensional data analysis

Started in 2006

Open community

Open source (GPL)

Three levels

Building blocks in C

Recipes in Python/SCons

Papers in LaTeX + SCons

Personal Experience Jon Claerbout and RR at Stanford Madagascar open-source software CiSE special issue (Jan-Feb 2009) David Donoho et al Randall LeVeque Roger Ping and Sandrah Eckel Victoria Stodden

Jon Claerbout and RR at Stanford

Madagascar open-source software

CiSE special issue (Jan-Feb 2009)

David Donoho et al

Randall LeVeque

Roger Ping and Sandrah Eckel

Victoria Stodden

CiSE Reproducible Research David Donoho, Arian Maleki, Inam Rahman, Morteza Shahram, Victoria Stodden 15 years of reproducible research in computational harmonic analysis MATLAB WaveLab: 690 citations “ Striving for reproducibility imposes a discipline that leads to better work. ”

David Donoho, Arian Maleki, Inam Rahman, Morteza Shahram, Victoria Stodden

15 years of reproducible research in computational harmonic analysis

MATLAB

WaveLab: 690 citations

“ Striving for reproducibility imposes a discipline that leads to better work. ”

CiSE Reproducible Research Randall J. LeVeque Python tools for reproducible research on hyperbolic problems Fortran + Python Clawpack: 7,000 registered users “ Scientific and mathematical journals are filled with pretty pictures of computational experiments that the reader has no hope of repeating .”

Randall J. LeVeque

Python tools for reproducible research on hyperbolic problems

Fortran + Python

Clawpack: 7,000 registered users

“ Scientific and mathematical journals are filled with pretty pictures of computational experiments that the reader has no hope of repeating .”

CiSE Reproducible Research Roger D. Peng and Sandrah P. Eckel Distributed reproducible research using cached computations R language Cacher package “ We propose that a modular research approach lends itself more naturally to reproducible results .”

Roger D. Peng and Sandrah P. Eckel

Distributed reproducible research using cached computations

R language

Cacher package

“ We propose that a modular research approach lends itself more naturally to reproducible results .”

CiSE Reproducible Research Victoria Stodden The legal framework for reproducible research in the sciences Licensing and copyright ORL (Open Research License) “ We need a license designed with the needs of computational researchers in mind .”

Victoria Stodden

The legal framework for reproducible research in the sciences

Licensing and copyright

ORL (Open Research License)

“ We need a license designed with the needs of computational researchers in mind .”

Why Reproducible? Science is the systematic enterprise of gathering knowledge about the universe and organizing and condensing that knowledge into testable laws and theories. The success and credibility of science are anchored in the willingness of scientists to independent testing and replication by other scientists. This requires the complete and open exchange of data, procedures and materials .

Science is the systematic enterprise of gathering knowledge about the universe and organizing and condensing that knowledge into testable laws and theories. The success and credibility of science are anchored in the willingness of scientists to independent testing and replication by other scientists. This requires the complete and

open exchange of data,

procedures and materials .

Open-Source Software “ Abandoning the habit of secrecy in favor of process transparency and peer review was the crucial step by which alchemy became chemistry. In the same way, it is beginning to appear that open-source development may signal the long-awaited maturation of software development as a discipline.” Eric S. Raymond

“ Abandoning the habit of secrecy in favor of process transparency and peer review was the crucial step by which alchemy became chemistry. In the same way, it is beginning to appear that open-source development may signal

the long-awaited maturation of

software development as a

discipline.” Eric S. Raymond

How to Do Reproducible Computational Experiments? Code attached to published results Continuous maintenance Previous results used for testing Test-driven development Lessons from open-source Intellectual property Community http:// ahay.org

Code attached to published results

Continuous maintenance

Previous results used for testing

Test-driven development

Lessons from open-source

Intellectual property

Community